Daemons Discuss All articles
War Stories

Boot Script Archaeology: Three Sysadmins Dug Into Their New Employer's rc.local and Uncovered Decades of Panic

Daemons Discuss
Boot Script Archaeology: Three Sysadmins Dug Into Their New Employer's rc.local and Uncovered Decades of Panic

Photo: Frank H. Nowell, Public domain, via Wikimedia Commons

Nobody onboards you to the boot scripts. The HR lady walks you through the 401(k), the engineering lead shows you the CI pipeline, and then on day nine, some file you've never heard of fails during a routine reboot and suddenly you're staring at the corporate unconscious.

That's the thesis of this piece: a startup's init scripts are a fossil record of its worst nights. Every line was written by a person who was tired, scared, or about to get on a plane. We asked three sysadmins to show us the artifacts.

Exhibit A: The Sediment Layers

Marcus, a sysadmin in Denver, joined a Series B logistics startup last spring. His rc.local read like geological strata.

"Bottom layer, 2016: a comment that just says # DO NOT TOUCH — ask Dave. Dave, it turns out, left in 2018. That line has been load-bearing for six years."

Above it, a 2019 block that hard-codes an IP address. Above that, a 2021 addition that checks whether the hard-coded IP is reachable and, if not, quietly swaps in a different one. "Nobody ever fixed the root cause," Marcus says. "They just kept adding layers. It's like tree rings, except each ring is a fire drill."

When Marcus finally refactored the whole thing into systemd units, he says the diff review took eleven minutes and three people cried. (Two from relief. One from Dave, who apparently does not work there anymore but is still on the alumni Slack.)

Exhibit B: The Comments Are the Confession

Priya, in Austin, excavated a boot directory at a healthcare SaaS company and found that the comments told a more honest story than the code.

"The script itself was forty lines of curl calls and sleep 30 statements. But the comments? Comments like # this is fine (it is not fine) and # TODO before IPO (we did not go public) and my personal favorite, # if you're reading this at 3am, I'm sorry, it was 3am when I wrote it too."

Priya describes the experience as "reading a stranger's diary that happens to run as root." Her favorite artifact: a function named please_work() that has been called in production since 2017 and has, by her estimate, worked roughly 94% of the time.

Exhibit C: The Hopeful Startup vs. The Tired Company

Devon, a contractor in Chicago, has seen enough boot scripts to sort companies into species.

"The hopeful startup writes comments like # temporary until we build the proper orchestration layer in 2014. The tired company writes # works, don't ask in 2014 and 2024. You can tell how many rounds of funding a company burned through by counting how many times 'temporary' appears."

Devon's most prized find: a boot script at a Midwest insurance company containing a comment in Portuguese that, roughly translated, reads "I do not know why this is needed but removing it breaks the billing system." Nobody at the company speaks Portuguese. It has been there since at least 2013.

A Field Glossary: What Boot Script Comments Actually Mean

For the next time you inherit someone's rc.local, here's our peer-reviewed translation table:

What the Dig Actually Tells You

Here's the part we'll defend in the comments section: reading a company's boot scripts before you accept the offer is more informative than any culture deck. A tidy rc.local with dated commits and honest comments means someone here had time to care. A 600-line shell script with three different authors' fingerprints and a comment reading # seriously do not ask means you're joining an archaeology dig, not a team.

Neither is necessarily a dealbreaker. Some of the best engineering cultures we know grew out of the worst rc.local in Ohio. But you should know what you're signing up for — because unlike Dave, the script is staying.

Bring a trowel. And for the love of all that is holy, back it up before you touch anything.

All Articles

Related Articles

Asking America's Junior Devs to Explain Daemons Was the Best Decision We Made All Quarter

Asking America's Junior Devs to Explain Daemons Was the Best Decision We Made All Quarter

Fifty Lines of Log, Four Communities, One Mystery: We Put a $500 Bounty on Our Crash and the Internet Did Not Disappoint

Fifty Lines of Log, Four Communities, One Mystery: We Put a $500 Bounty on Our Crash and the Internet Did Not Disappoint

Root for the Machine: Our Month of Letting an LLM Agent Loose on the Night Shift

Root for the Machine: Our Month of Letting an LLM Agent Loose on the Night Shift