When we audit a new client's on-call setup, we usually find the same thing: one or two engineers who haven't had an undisturbed night in months. They're not the on-call rotation — they are the on-call rotation. Their phones have PagerDuty installed. They've learned to wake up, check if the alert is real, and fall back asleep if it's not. They are very good at this, in the way people get good at things they do too often.
Most CTOs frame this as an operations problem. It's not. It's a retention problem with a ticking clock, and the engineers you'll lose first are the ones you can least afford to lose.
Why the math doesn't work below 4 engineers
A sustainable on-call rotation needs at least 4 people to avoid burning anyone out. With 4 engineers, each person is primary on-call roughly one week in four — which is hard but survivable. With 3 people, it's one week in three. With 2, it's every other week.
At "every other week," on-call stops being an occasional disruption and starts being a permanent state. Your engineer is either on-call right now, or they're resting up for the next rotation in 7 days. There's no week where they can make plans, sleep without their phone on the nightstand, or take a long weekend without arranging coverage.
If your entire DevOps or platform function is one or two people, this is where you are. They may not have said anything because they don't want to seem like they can't handle it. But check how long they've been doing it. Twelve months of this is the typical window before someone starts looking.
Alert fatigue is a design problem, not a discipline problem
The standard response to too many alerts is "we need to tune the alerts." The deeper problem is that alerts are accumulating because no one owns the alert backlog. An engineer sets up a CPU alert at 80% because that felt right at the time. Months later, a traffic spike causes it to fire three times on a Saturday. The engineer silences it for a week. Two months after that, nobody remembers why the threshold was 80%, whether it was ever actionable, or what "actionable" would even mean.
The result is an on-call experience that looks like this: PagerDuty wakes you at 2am. You log into Grafana. CPU is at 82%. You check the application metrics — no errors, response times normal, no user reports. You silence the alert and go back to sleep. Six weeks later this happens again. You've been conditioned to treat this alert as noise.
The problem is that one night, the CPU alert at 82% will mean something. And you'll sleep through it for the same reason you've slept through it a dozen times before.
Good alert hygiene has one rule: every alert that fires should require a human to take action. If you acknowledge and silence an alert more than twice without fixing the underlying cause, the alert is either misconfigured or pointing at something that needs to be fixed. Either way, the alert is not doing its job.
What a functional on-call structure looks like at startup scale
The goal isn't to have zero incidents — it's to have incidents that are handled quickly, at reasonable hours, by people who have the context to fix them. Here's the structure that works for most 10–50 engineer startups:
- Primary on-call: The person who gets paged first. Responsible for initial triage — is this real, does it need escalation, can it wait until morning. Not necessarily responsible for fixing it.
- Secondary on-call: Gets paged if primary doesn't acknowledge within 10 minutes, or if primary explicitly escalates. This is the person with deeper infrastructure context.
- Escalation path: Who does secondary call if the incident is beyond their ability to resolve? For most startups, this is the CTO or VP Engineering. That person should not be on the regular rotation, but they should be reachable.
- Business hours SLA vs. off-hours SLA: Not every alert warrants a 2am wake-up. Classify your alerts: P1 (wake up anyone, immediately), P2 (notify, fix by start of business), P3 (log it, fix in the next sprint). Most startups don't do this classification and treat everything as P1.
The postmortem habit that actually prevents recurrence
The most common postmortem in startup engineering is the one that never gets written. An incident happens, gets fixed, and everyone goes back to their normal work with a vague intention to "figure out how to prevent this." Three months later, the same thing happens.
A postmortem doesn't need to be a long document. It needs to answer three questions: what happened, why it happened, and what specific change will prevent it from happening again. That last part is the one that actually matters. "We'll monitor this more carefully" is not a specific change. "We're adding a synthetic canary that tests this endpoint every 5 minutes and alerts before users notice" is a specific change.
We recommend a lightweight incident log: a shared doc where every P1 and P2 incident gets a 5-line entry. Date, what happened, root cause, fix applied, follow-up action with an owner and a due date. The follow-up actions go into the backlog. The person who was on-call owns them. If the follow-up actions don't get done, the incident will recur — usually at 2am on a Friday.
The retention cost nobody puts on the spreadsheet
When a senior engineer leaves, the cost isn't just the recruiting fee and the 90-day ramp-up for their replacement. It's the knowledge that walks out with them: which services are fragile, which configs haven't been touched since 2022 because the person who wrote them left, which alerts are real and which are noise. That knowledge doesn't get written down because it accumulated over two years of incidents and doesn't feel like something you can document.
Engineers leave on-call-heavy jobs for predictable reasons. The exit interviews say "better opportunity" and "career growth." What they mean, usually, is: "I can't be available every other week indefinitely, and no one here seems to think this is a problem."
The fix is not paying them more, though fair compensation helps. The fix is a rotation large enough that no one carries disproportionate load, alerts that fire only when they should, and a postmortem process that actually closes the loop. These are engineering problems, not HR problems.
"Our best backend engineer gave notice on a Monday. We found out later that the weekend before, he'd been paged 11 times while at his sister's wedding. We could have fixed the rotation. We didn't, until it was too late."
When you can't build a rotation internally
Below a certain team size — usually fewer than 6 engineers — you simply cannot build a sustainable rotation from internal headcount without making on-call someone's entire job. For most startups at this stage, that's not a viable option.
The alternatives: managed on-call services that handle first-line triage and escalate to your team only for incidents that require product context; fractional DevOps coverage where the external team carries the infrastructure on-call rotation and your engineers carry product-level on-call; or accepting limited-hours coverage for non-critical services and being explicit with users about SLAs.
None of these are perfect. But "one engineer on call every other week indefinitely" is also not perfect — it just doesn't show up as a cost until the resignation letter arrives.