Cold Pizza, Dead Pagers, and the Undead: Three War Stories From the Zombie Process Trenches

Every data center has a ghost story. Ours involve zombie processes.

If you've ever run ps aux at 2 A.M. and watched a column full of [defunct] entries pile up like unclaimed luggage, you know the particular dread. Zombies aren't dangerous on their own — they're just dead processes waiting for their parent to reap them. But they're a symptom, and symptoms have a way of escalating into full-blown incidents. So we asked our readers, listeners, and one very tired contributor from a colo outside Phoenix to share their worst zombie-adjacent outages. What follows is part post-mortem, part group therapy.

The Cron Job That Refused to Retire

Our first story comes from a mid-size e-commerce shop in Ohio — the kind of place where one person is the entire infrastructure team, the on-call rotation, and the janitor of last resort.

At 1:47 A.M., a Saturday-night flash sale triggered a backup cron job that hung on an NFS mount. The job spawned, forked, timed out, and — thanks to a shell script wrapper with no error handling — kept respawning. Each dead child left a zombie behind because the wrapper wasn't calling wait(). By 3 A.M., the process table hit the pid_max ceiling and the box refused to fork anything new. SSH sessions died mid-command. Monitoring went dark. The pager went off with the least helpful alert in existence: "host unreachable."

The fix, eventually, was a reboot. The real fix was rewriting the wrapper, adding timeout, and — this contributor's words — "learning what waitpid does before my next birthday."

The lesson: process tables are a finite resource. If your automation spawns children, it had better be ready to bury them.

The Orphaned Daemon Nobody Owned

Story number two arrives from a healthcare SaaS provider in the Research Triangle area of North Carolina. Somewhere in a migration two years prior, a legacy log-shipping daemon had been orphaned — its parent shell long gone, its PPID reassigned to init, its config file pointing at a decommissioned syslog server.

For twenty-six months, this thing quietly retried connections, leaked file descriptors, and grew its memory footprint like a houseplant nobody watered. Nobody knew it existed. It wasn't in the runbooks, the config management system, or anyone's head. Then a kernel upgrade rebooted the host, the daemon failed to come back (no systemd unit, no init script, nothing), and suddenly the compliance team noticed logs had stopped shipping. In 2021. Two years of audit trail: gone.

The post-mortem ran to eleven pages. The executive summary was one line: "We did not know what was running on our own servers."

The lesson: run ps aux like you mean it. Inventory is a practice, not a document.

The Dawn Reap

Our final tale is the feel-good one, relatively speaking. A streaming media startup in Austin had a load balancer that would slowly accumulate zombie processes from a misbehaving health-check script. Every few days, connections would start failing, the on-call engineer would restart the service, and everyone would move on. This went on for four months.

Four months of 2 A.M. wake-ups. Four months of cold pizza eaten in front of a KVM console. The breakthrough came when a new hire — a former kernel contributor, naturally — glanced at the process list and said, "Who's the parent here? Nobody's reaping these kids."

One line of code. One wait() call. One very humble engineering lead.

The lesson: sometimes the bug isn't the thing that's screaming. It's the thing that quietly refuses to clean up after itself.

What It All Means

Zombie processes are the perfect metaphor for technical debt: individually harmless, collectively catastrophic, and always a sign that someone upstream isn't doing their job. The pagers, the cold pizza, the dawn fix — that's the human cost. The wait() call is the engineering cost.

So tonight, before you close your laptop: check your process table. Your future 2 A.M. self will thank you.