The call came on a Tuesday afternoon. The Head of Engineering at a 28-person SaaS startup had just taken a resignation from their senior DevOps engineer — the person who had been with the company since month four, who had built the CI/CD pipeline, designed the Kubernetes cluster, knew every Terraform module, and was the only person who understood why one particular cron job had to run before the other.
Two weeks' notice. That's what they had to extract three years of infrastructure knowledge before the lights went out.
We've been called into this situation more times than we can count. What follows is the playbook we actually run.
The first 48 hours: triage before anything else
The instinct is to start writing documentation immediately. Resist it. Documentation is slow and what gets written under time pressure is usually wrong. Before you write a single runbook, spend the first 48 hours on three things:
1. Map what would break first
Walk through every customer-facing workflow and identify the infrastructure dependency chain. What happens at midnight when the billing cron job runs? What happens when a deploy triggers in CI? What happens when someone needs to access a production database for a support escalation? You're not documenting these yet — you're just listing them so you know what knowledge you need to extract first.
2. Inventory all credentials and access
The most irreversible damage in a departure scenario isn't broken infrastructure — it's access that disappears with the engineer. Before anything else, audit and transfer ownership of:
- Every cloud account (AWS, GCP, Azure) and the IAM roles that were provisioned by or for this person
- DNS registrars and Cloudflare accounts, especially if the engineer managed any zones directly
- Container registry credentials and the CI service account tokens
- Any vendor accounts set up under a personal email (this happens more than it should)
- SSH keys and VPN certificates provisioned for this engineer that also have access to production
We've had clients discover, weeks after a departure, that a monitoring integration was authenticating with the engineer's personal API key. The integration kept working — until the key was rotated or the account was closed.
3. Identify the undocumented decisions
Ask the departing engineer one specific question: "What things work only because of something you did manually at some point?" The answer to that question is worth more than any runbook. Manual interventions — a DNS record adjusted at 2am, a database parameter tweaked to fix a slow query, a Kubernetes ConfigMap changed directly in prod — never make it into documentation. But they're the things that will bring down production when they're missing.
The knowledge extraction sprint (days 2–10)
Once you've done triage, you have roughly one week to do structured knowledge transfer with someone still in the building. There are two modes for this, and the best practice is both:
Recorded walkthroughs
Have the engineer walk through each major system while someone records. Not a tutorial — a live "this is what I'd do if X happened" walkthrough. The value is in the reasoning: why is this Terraform module structured this way? Why does this deployment have 3 replicas instead of 2? What are the first three things you check when the API is slow? The recordings don't need to be polished. They need to exist.
Shadow sessions on real work
If any infrastructure work comes up during the notice period — a deploy, an incident, an access request — have a second engineer shadow every step. There's no substitute for watching someone actually navigate the system under real conditions. What an engineer does automatically under pressure is exactly what documentation misses.
What to document first (in priority order)
- Deployment process — step by step, including anything that can't be automated
- Incident response — what to check when the service is down, where the dashboards are, how to rollback
- Access management — how to grant a new engineer production access, how to revoke it
- Scheduled jobs — every cron, every scheduled Lambda, every maintenance window and why it exists
- Terraform state — what state files exist, where they're stored, how to run a plan safely
Everything else — the nice-to-have architecture docs, the decision logs, the "how the system was designed" narrative — comes after. The job during the notice period is operational continuity, not institutional history.
The stabilization phase (weeks 2–8)
After the engineer is gone, you have a decision to make: run lean while you recruit, or bring in external coverage immediately. Most companies try to run lean, distributing infrastructure ownership among developers who have other jobs. This works until it doesn't — usually at 11pm when something breaks and nobody in the Slack channel has ever looked at the alerting stack.
The risk of running lean isn't the routine operational work. It's the incident that requires someone who knows the system. Developers who didn't build the infrastructure can follow a runbook for a known failure mode. They cannot diagnose an unknown one at 2am.
In the eight departures we've backstopped in the last two years, the pattern is consistent: companies that brought in external coverage within the first two weeks had zero production incidents during the transition. Companies that waited until the first incident to ask for help paid a much higher price — both financially and in team trust.
When to hire immediately vs. use this as an inflection point
A departure forces a conversation that's worth having: should you replace the engineer like-for-like, or is this an opportunity to restructure how you handle infrastructure?
Replace like-for-like when:
- You have a complex, specialized infrastructure that requires deep in-house knowledge to evolve (not just maintain)
- The departing engineer's role was mostly platform engineering — building things that multiply other engineers' productivity
- You're at a stage (30+ engineers) where the overhead of managing an external team costs more than an in-house hire
Consider the fractional model when:
- The departing engineer was mostly doing operational work — access management, deployments, incident response — rather than building platform capabilities
- Your infrastructure isn't changing rapidly enough to justify a full-time hire whose job would be mostly maintenance
- You want to use the transition window to actually improve the infrastructure, not just restore it — a hire will be in ramp-up for 3 months; an experienced team is productive in week one
- You're concerned about recreating the single-point-of-failure problem with the next hire
The single-point-of-failure concern is the one most teams don't acknowledge until they're living through it. One engineer knowing everything is a structural risk that doesn't get fixed by hiring one engineer. The knowledge concentration reforms around the new person, and you're back in the same position 18 months later.
What we do when we take over a transition
When we're brought in to backstop a departure, the first engagement is always a full infrastructure audit — not to judge the previous engineer's work, but to build our own map of what exists. We spend two days across:
- Cloud account audit: every resource, its cost, its purpose, who or what created it
- Access audit: every human and service account with production access, whether each one should still have it
- Pipeline audit: every CI job, what it deploys, what happens when it fails, whether it's tested
- Dependency audit: external services, third-party integrations, anything the application calls at runtime
The goal isn't to document everything — we'll build the runbooks as we work through the first month. The goal is to not be surprised at 2am. The surprises that hurt come from things nobody knew to tell you about. A systematic audit surfaces them before they become incidents.
"Within 48 hours of our engineer leaving, DevOps Team had mapped everything and already found two things even he hadn't documented. We never had a single production incident during the transition."
The real lesson: the problem started before the resignation
The departure is the acute event. The underlying condition is the single-point-of-failure architecture — a team where one person holds all the infrastructure context, makes all the infrastructure decisions, and is the only one paged when something breaks.
That condition exists in most startups, not because of bad planning but because infrastructure ownership naturally concentrates. Once it does, fixing it requires deliberate effort: documentation requirements, cross-training, knowledge-sharing practices, and ideally a team structure where at least two people are deeply familiar with every critical system.
The companies that handle this best don't wait for a departure to create that structure. The ones that learn the lesson most expensively are the ones who hire the replacement, watch the same knowledge concentration reform around the new person, and eventually have the same conversation again three years later.