Daemons Discuss All articles
War Stories

Meet the Unsung Heroes Eating Your RAM at 3 A.M.: An Oral History of the Daemons That Never Sleep

Daemons Discuss
Meet the Unsung Heroes Eating Your RAM at 3 A.M.: An Oral History of the Daemons That Never Sleep

Photo: server room dark blue lights racks, via img.freepik.com

Nobody thanks a daemon. There's no award ceremony for cron, no lifetime achievement plaque for sshd. These background processes just... run, quietly forking away in the dark while the humans sleep, occasionally waking one of us up at 3 a.m. to remind us who's really in charge.

We asked the Daemons Discuss community for their best — or worst — stories about the seven background processes that quietly run modern life. What follows is part technical explainer, part group therapy session.

cron: The Original Passive-Aggressive Coworker

cron is the grandfather of daemons, faithfully waking up every minute since roughly 1975 to check whether you've left it any homework. It's simple, it's reliable, and it absolutely will not warn you when your crontab has a typo.

The war story: "We had a backup job scheduled at 2:30 a.m. that ran fine for three years," recalls a sysadmin we'll call Dave, because that's his name. "Then daylight saving time hit. Cron doesn't care about DST. The backup ran twice, filled the disk, and the database fell over. I learned more about inode exhaustion that night than I ever wanted to."

The lesson: Time-based scheduling is a lie we tell ourselves. If your job can't survive being run twice, or being skipped entirely, you don't have a backup — you have a hope.

sshd: The Front Door You Keep Forgetting to Lock

sshd is how every sysadmin enters the building. It's also how every botnet on the planet would like to enter the building. The daemon itself is rock solid; the humans configuring it are the problem.

The war story: A junior dev once "temporarily" set PermitRootLogin yes with a password of password123 on a client-facing box. It took the internet approximately 40 minutes to find it. "The box was mining crypto for three weeks before anyone noticed," Dave's colleague admits. "The CPU load average was the only tell, and honestly, we just assumed it was Java."

The lesson: Keys, not passwords. Fail2ban, not vibes. And audit your firewall rules more often than you audit your feelings.

nginx: The Bouncer at the Door of the Internet

nginx serves somewhere north of a third of all websites, quietly proxying traffic and judging your config files. Its syntax is famously unforgiving — one missing semicolon and the whole thing refuses to start, helpfully pointing you at line 1 when the problem is on line 247.

The war story: "Someone pushed an nginx config to production with a limit_req value of 1 request per second instead of 1000," one reader wrote in. "We rate-limited our own checkout page on Black Friday. Our biggest sales day of the year, and nginx was politely telling customers to slow down like a concerned barista."

The lesson: Staging environments exist for a reason. So does nginx -t. Use both, or the bouncer will bounce you.

systemd: The One Who Ate the Neighborhood

Love it or loathe it, systemd is the init system on most modern Linux distros, and it does everything from logging (journald) to network config (networkd) to time synchronization (timedated). Critics call it scope creep; fans call it integration. Everyone calls it at some point, usually while cursing at journalctl flags.

The war story: "A systemd unit file had Restart=always and the app crashed instantly on boot," a reader shared. "It restarted in a tight loop so fast it pegged the CPU and made the box unreachable. The fix was booting into single-user mode, which I hadn't done since my Slackware days in college. It was like running into your ex at a gas station."

The lesson: Understand what your service manager does when things fail. Restart=always without a rate limit is not resilience — it's a denial-of-service attack against yourself.

sshd's Cousin, ssh-agent: The Key Ring That Vanishes

Fine, we're cheating — but ssh-agent deserves a mention for the sheer number of deploy pipelines it has silently broken. When the agent dies, your automation dies with it, and everything looks exactly like a credentials problem.

The lesson: Half of all "the deploy is broken" tickets are actually "the agent is gone" tickets.

named (BIND): The Phonebook Nobody Reads Until It's Wrong

DNS is the daemon everyone forgets until it fails, at which point it's the only daemon that matters. BIND has been running the internet's phonebook since the 1980s, and a misconfigured zone file can take down a company faster than any ransomware crew.

The war story: "We typo'd a TTL from 3600 to 36000," one admin confessed. "Ten hours. Every DNS mistake we made was cached for ten hours. We called it 'the weekend that wouldn't end' even though it was a Tuesday."

The lesson: Lower your TTLs before maintenance, not after. And never, ever edit zone files on a Friday.

dockerd: The New Kid With the Trust Fund

The container daemon is the newest entry here, and it has already accumulated decades' worth of chaos energy. Docker's daemon runs as root by default, holds your container state hostage, and occasionally just... stops, taking every running workload with it like a landlord changing the locks.

The war story: "dockerd hit a known bug and died, taking down 40 containers across three products," a reader reported. "We restarted it and everything came back except the one container nobody had documented. We found it running on a sticky note from 2021."

The lesson: Document your containers. Preferably somewhere that isn't a sticky note.

Raise a Glass to the Invisible

These daemons don't want your thanks. They want their config files respected, their logs rotated, and their memory limits set to something reasonable. The rest of us just work here — and occasionally get woken up at 3 a.m. to prove it.

Got a daemon horror story of your own? Bring it to the forums. The break room coffee is bad, but the stories are excellent.

All Articles

Related Articles

Twelve Years of Yelling: A Field Guide to the Init War Ceasefire

Twelve Years of Yelling: A Field Guide to the Init War Ceasefire