On-Call Without Burnout: How to Structure 24/7 Coverage That Engineers Don't Quit Over

A founder called us after losing their only infrastructure engineer in 18 months. The engineer was talented, experienced, and had built most of what was running in production. They left for a company offering the same compensation — because the new job had a proper on-call rotation of 12 people. At their current job, the rotation was 2 people: them and a backend engineer who knew enough to restart a container but not enough to diagnose why it crashed.

This story is not unusual. Badly structured on-call is one of the most reliable ways to lose your infrastructure engineers, and the costs don't show up on a dashboard. They show up in attrition, in 3am Slack messages that go unanswered, and in the gradual accumulation of alerting debt — where alerts get silenced because nobody has time to fix what's causing them.

Here's how we structure on-call across our engagements, and what we've seen separate sustainable rotations from the ones that grind people down.

The problem is almost never "we don't have enough people"

That's usually the first instinct — if we had more engineers on the rotation, each person would be on-call less often. True, but it misses the root cause. Most on-call burnout comes from three things that aren't headcount problems:

• Alert noise. Alerts that fire frequently and resolve on their own, or that require the same manual intervention every time. Engineers get paged at 2am for something that has happened 40 times this month and never resulted in a customer-visible incident. They stop taking alerts seriously — which means they also stop taking the real ones seriously until it's too late.
• No runbooks. The on-call engineer has to figure out what's wrong and how to fix it from scratch, every time. Even for incidents they've seen before. Knowledge lives in one person's head, and that person is not always the one who gets paged.
• No toil budget. Toil — the manual, repetitive work that keeps systems running but doesn't improve them — accumulates without a deliberate process to eliminate it. If on-call generates 10 hours of toil per week and nobody is allocated time to automate it, the toil only grows.

Fix these three things and most on-call rotation problems become manageable even with a small team.

Alert triage: what to page on, what to log, what to delete

The first thing we do when taking over a new on-call environment is audit every alert. We ask two questions for each one: "In the last 90 days, how many times did this alert fire?" and "In the last 90 days, how many times did it indicate something that required human action?"

The gap between those numbers is alert noise. In most production environments we've inherited, 40–60% of alerts are noise by this definition — they fire, they resolve, and the on-call engineer has been trained to either acknowledge and ignore or apply the same fix that doesn't solve the underlying problem.

We sort alerts into three categories:

• Page immediately (P1): Customer-facing service is degraded or down. Error rate exceeds SLO. Data pipeline failure with business impact. These get PagerDuty with phone call escalation, any hour of the day.
• Notify but don't wake (P2): Something is trending wrong but not yet customer-impacting. Disk filling slowly, latency spiking but within SLO, a non-critical job failing. These go to Slack. The on-call engineer sees them at the start of their shift; they don't get paged at 3am.
• Log only (P3): Informational. Useful in retrospect for debugging, but not worth alerting on at all. If they're creating noise, they get deleted outright.

After this triage, the number of pages per week typically drops by 50–70%. That alone changes the character of on-call — from a constant drip of interruptions to a manageable, exception-based responsibility.

Runbooks that actually get used

The test of a good runbook is: can someone who has never seen this particular failure before resolve it without escalating? Not without thinking — with thinking, but without needing to reach someone at 2am who has.

Most runbooks fail this test because they were written after the fact by the engineer who already knew what to do. They're either too high-level ("check the logs") or too specific to a single past incident ("restart the authentication service if the token cache is poisoned"). Neither helps the next person who hits a different variation of the same problem.

The runbooks we write follow a standard structure:

1. Alert name and what it means — what condition triggered this, in plain language
2. Who this affects — which users or systems are impacted, at what severity threshold
3. First 5 minutes — the three commands to run immediately to confirm the issue and get situational awareness
4. Decision tree — the 3–5 most common root causes and the specific remediation for each
5. Escalation — who to page if the decision tree doesn't resolve it, and what context to give them
6. Post-incident — link to the incident template, and a reminder to file a toil ticket if the resolution was manual

Runbooks are living documents. After every incident, the on-call engineer adds what they learned. Within 6 months of consistent use, a well-maintained runbook library means that 80%+ of incidents are resolved without escalation, by whoever happens to be on-call — regardless of their familiarity with that particular system.

Escalation policies that reduce, not expand, on-call burden

A common mistake is building an escalation policy that makes the most experienced engineer the first contact for everything. This is the implicit policy at most small engineering teams — if something is wrong, ping the person who built it. It doesn't scale, and it keeps that person permanently tethered to their phone.

The escalation structure we use:

• Tier 1 (first responder): Anyone on the rotation, following the runbook. Goal: restore service or confirm escalation is needed. Time budget: 15 minutes.
• Tier 2 (service owner): The engineer most familiar with the affected system. Paged only if Tier 1 can't resolve via runbook. They know the system; they're not responsible for first-line incident response.
• Tier 3 (incident commander): For major incidents — customer data at risk, extended outage, multi-service failure. This is the person who coordinates the response, communicates with stakeholders, and runs the postmortem. Not necessarily the most technical person in the room.

Most incidents never reach Tier 2. The goal of good runbooks and alert triage is that Tier 1 handles the majority of on-call work independently. Tier 2 becomes a fallback for genuine unknowns, not a crutch for underdocumented systems.

The toil budget: the thing almost nobody does

Google's SRE book introduced the concept of a toil budget — the idea that SREs should spend no more than 50% of their time on operational work (toil), with the other 50% on engineering work that reduces future toil. In practice, most teams have no explicit budget at all. Toil expands to fill available time, engineers get frustrated, and the ratio quietly becomes 80% toil and 20% everything else.

The version we implement is simpler than Google's model but has the same effect: every manual incident response generates a ticket. The ticket describes the manual step and asks "what would need to be true for this to not require human intervention?" We set aside a fixed block of engineering time each sprint — typically 20% — for working through the toil ticket backlog in order of alert frequency.

Within 3 months, this systematically eliminates the highest-volume alerts. The ones that fired 40 times a month and always needed the same restart? Automated. The cron job that failed silently and required manual re-runs? Fixed. The disk that always filled up and needed a manual cleanup? Lifecycle policy. The on-call engineer's job gets quieter, which makes it sustainable, which means it stays staffed.

Structuring the rotation itself

A few practical patterns that work better than the default "weekly rotation, whoever's up is up":

• Follow-the-sun for small teams. If you have engineers in two timezones, split the 24 hours between them rather than having one person cover nights. A US-East engineer covering 8am–8pm EST hands off to a EU engineer covering EU working hours. Nobody covers hours when they'd normally be asleep.
• Shadow rotations for onboarding. New engineers spend 2–4 weeks shadowing the on-call engineer before taking primary. They see real incidents, they write their first runbooks, and they build confidence before they're responsible for the page. Engineers who are thrown into on-call cold either struggle or escalate everything — neither outcome is good for the rotation.
• Compensation.** On-call should be compensated — either in additional pay for hours actually paged, or in time-off-in-lieu for weekend coverage. "It's just part of the job" is a statement that works until your best engineers do the math and decide the effective hourly rate for a weekend paged 4 times at 3am doesn't justify staying.
• Hard limits on consecutive nights. Nobody should be paged more than 3 nights in a row, regardless of rotation schedule. If the team is too small to avoid this, that's a staffing problem — but acknowledging it as a staffing problem is the first step toward fixing it.

When the team is too small to rotate

The hardest situation is a team of 2–4 engineers where a proper rotation is mathematically impossible without every engineer being on-call more than half the time. This is the most common situation we encounter at seed and early Series A companies, and there's no clean solution — only a set of tradeoffs.

Options in order of preference:

1. Outsource Tier 1. A fractional DevOps team provides 24/7 first-line coverage. Engineers are only paged for Tier 2 escalations — incidents that require deep system knowledge. This is how we operate across most of our engagements. The in-house engineers' nights are protected; they handle the hard problems, not the routine ones.
2. Invest in reliability over coverage. If you can't staff a sustainable rotation, the next best thing is to have fewer incidents. Every hour spent on reliability engineering — better health checks, graceful degradation, automated recovery — directly reduces the on-call burden. An application that recovers from its own failures doesn't need a human to restart it at 2am.
3. Be explicit about the unsustainability. If the team is genuinely too small to staff reasonable on-call, that information should be visible to leadership, not absorbed silently by the engineers carrying it. The cost of the current situation — burnout, attrition risk, quality degradation — needs to be visible before it can be addressed.

"Before you came in, I was paged an average of 11 times a week. Six weeks later it's down to 2, and most of those I can handle in 5 minutes with the runbook. I've slept through the night every night this week."

On-call is a solvable problem. Not by ignoring the human cost of it, and not by simply adding headcount — but by treating alert quality, documentation, and toil elimination as first-class engineering work. The teams that do this retain their infrastructure engineers. The ones that don't find out what it costs to replace them.