On-Call Rookies, Meet Your Seven Roommates: A Field Ranking of the Daemons That Will Ruin Your Weekend

Nobody tells you this when you sign the offer letter, but your first junior sysadmin job comes with roommates. They live in your servers, they never sleep, and at least one of them will call you at 2:14 a.m. on a Sunday. Welcome to on-call. Here's the ranking of the seven daemons most likely to ruin your weekend, based on hard-won scar tissue from rotations I still don't talk about at parties.

7. sshd — The Polite One

Ranking lowest because it almost never pages you directly — it just enables every other disaster by letting humans into places they shouldn't be. The classic rookie move: fat-fingering a firewall rule while SSH'd in and locking yourself out. Everyone does it once. Triage tip: before touching firewall rules on a remote box, keep a second session open and test with iptables -L from a console connection. Your future self, standing in a parking lot at an office in Ohio at midnight, will thank you.

6. ntpd — The Silent Saboteur

It never alerts. It just quietly drifts, and then your TLS certificates "expire," your database replication loses its mind, and your logs read like abstract art. Triage tip: if three unrelated things broke at once, check the clock first. It's always the clock.

5. httpd / nginx — The Diva

The web server is the daemon everyone blames and the daemon that's usually innocent. It'll happily page you with a 502 when the actual culprit is an application container that OOM-killed itself. Triage tip: check upstreams before you check the web server. The diva is just delivering the bad news, badly.

4. systemd — The Manager Who Rearranged Your Desk

It's not going to crash. It's going to change how restarts work, silently, six months ago, and you're going to find out during an incident. Triage tip: learn systemctl list-jobs and journalctl -u <service> before you need them, not during the pager fire. Twenty minutes in a lab VM saves you a very long night.

3. rsyslog — The Hoarder

You will not appreciate rsyslog until the day it saves you, and you will not fear it until the day its disk fills up. A full /var/log takes down databases, mail queues, and anything else that needs to write — and the alert will say something completely unhelpful like "disk I/O wait elevated." Triage tip: configure log rotation and disk usage limits in week one. journalctl --disk-usage is your friend. Ignoring it makes rsyslog your enemy.

2. crond — The Midnight Prankster

Cron gets the silver medal because its failures are always time-delayed. A cron job that's been running fine for two years picks the night of your cousin's wedding in Texas to hang forever, stacking up instances like planes over O'Hare. Triage tip: every job should lock itself (flock is one line) and log its own completion. If you inherit a crontab with no comments, budget a full afternoon to figure out what these things do before they introduce themselves at 3 a.m.

1. named — The Undisputed Champion of Weekend Ruin

And at number one: DNS. Not because BIND crashes often — it doesn't — but because when DNS breaks, everything breaks, everywhere, at once, and half your monitoring depends on it so you won't even get paged properly. You'll find out because a coworker texted "is the site down for you?" Triage tip: memorize dig +trace, keep a secondary resolver configured, and never, ever make DNS changes on a Friday. The old-timers call it the Friday DNS Rule. The old-timers have bodies buried everywhere.

Your Survival Kit

Three habits that'll carry you through your first rotation: document every fix like you're writing it for the next rookie (you are), keep a runbook for the top three daemons above, and never trust a monitoring system that depends on the thing it's monitoring. The daemons are always going to be there. The goal isn't to beat them — it's to know them well enough that when the phone rings, you sigh, open the laptop, and think: yeah, I bet it's cron again.

That, rookie, is when you've made it.