Six Figures to Whisper at Strangers' Servers: A Week on the Road With a Production Exorcist
The Call Comes in at 4:47 PM on a Friday
Of course it does. Nobody calls Marisol Vega on a Tuesday morning when things are calm. They call her when the on-call rotation has collapsed, the Slack #incidents channel has more messages than a Black Friday sale thread, and somebody's VP has started using the phrase "material impact" in emails.
Vega is a production consultant — but not the kind who shows up with a 90-slide deck about digital transformation. Her niche is narrower and, frankly, weirder: she gets hired to figure out why a daemon is doing whatever cursed thing it's doing, when nobody left at the company remembers writing it, deploying it, or in one memorable case, naming it.
She charges rates that would make your HR department faint, and after a week riding shotgun with her, I understand why. What follows is a field journal from five days of engagement.
Day One: The Fintech Firm and the Scheduler That Wouldn't Die
The client is a payments company in Chicago with a job scheduler that has been firing duplicate batches every third Thursday of the month for two years. Nobody can reproduce it. Their monitoring swears nothing is wrong. Their database disagrees, loudly.
Vega's first move isn't technical. She buys the ops team pizza and asks a question that sounds like a joke: "Who here has been here longest?"
The answer — a sysadmin named Doug who'd been around since the office had a foosball table — cracked the case in eleven minutes. A former engineer had built a workaround around a leap-year edge case, documented exactly none of it, and left for a startup in Austin. The scheduler had been faithfully executing a bug ever since, like a clock that chimes wrong every hour but only on certain floors.
"It's never the code," Vega told me afterward, nursing a coffee at O'Hare. "It's the archaeology. I'm basically Indiana Jones, but the temple is a Jenkins instance and the golden idol is a shell script from 2016."
Day Two and Three: The Batch Daemon Eating CPU Since 2019
The next engagement was a Series B startup in Denver whose overnight batch daemon had been consuming a slow, steady, horrifying amount of CPU since roughly the second Obama-to-Trump transition. Their cloud bill had grown like a kudzu plant. Leadership suspected a memory leak. Vega suspected something dumber, because it's usually something dumber.
It was dumber. A retry loop with no backoff and no cap had been quietly reprocessing the same failed job — a job that had failed once, in 2019, due to a now-defunct third-party API — roughly every ninety seconds for six years. The daemon wasn't broken. It was loyal. It was holding a vigil for a job that died before some of the company's employees were hired.
The fix took forty minutes. The paperwork explaining to the CFO why they'd been paying for a digital candlelight vigil took considerably longer.
The Method: How You Whisper at a Runaway Process
Watching Vega work, a pattern emerges. Call it her playbook, call it a checklist, call it what it is — the accumulated scar tissue of a decade of engagements:
- Find the oldest person in the building and buy them lunch. Institutional memory lives in people, not wikis. The wiki is always wrong.
- Read the logs like a diary, not a dashboard. Dashboards tell you what's happening. Logs tell you what happened, which is usually the actual crime scene.
- Assume the daemon is behaving rationally. Runaway processes almost never malfunction. They're doing exactly what someone told them to do. The horror is figuring out who told them, and why.
- Never delete on day one. The urge to just kill the process and walk away is strong. It's also how you get invited back — or sued.
- Write the postmortem like it'll be read by a stranger at 3 A.M. Because it will be. Possibly by her, on the next engagement.
That last point matters more than it sounds. Half her billings, she admits, are really fees for writing down what a company already knew but never recorded.
Why Does This Profession Even Exist?
Here's the uncomfortable answer: because every company's infrastructure is a museum of decisions made by people who no longer work there. The daemon graveyard isn't a bug in the industry — it's a feature of how careers work. People change jobs at a pace that would make a 1950s company man weep, and every departure orphans a process.
Vega's job exists in the gap between "the person who wrote it" and "the person who's paged about it." That gap is widening. Every layoff cycle, every acquihire, every quiet departure of the one engineer who understood the billing pipeline — it all feeds her pipeline of clients.
She's ambivalent about it. "Some weeks I feel like a doctor," she said as we waited for a delayed flight back through Denver. "Some weeks I feel like a mortician. Mostly I feel like the only adult in a room full of processes that were raised by wolves."
The Takeaway for the Rest of Us
You may never hire a Vega. But her week offers a lesson for anyone who owns a server: the most dangerous code in your stack isn't the code you're worried about. It's the code nobody's worried about — the daemon that's been running quietly in the corner since before the last three reorgs, faithfully executing a job nobody remembers approving.
Go check on it. It's been waiting.
And if you find something horrifying, well — Vega's calendar opens up in Q3.
