The On-Call Problem Most Startups Never Actually Solve

Two years ago we onboarded a client whose sole infrastructure engineer had quit six months earlier. The reason: 18 months of being the only person who could respond to production incidents. Every alert, every 3am page, every "are you around?" Slack message on a Saturday — all of it landed on one person, with no backup, no runbooks, and no escalation path. When they left, the company had zero infrastructure knowledge and zero on-call coverage overnight.

This is not an unusual story. Startups solve "we need someone on-call" by finding a person and putting them in PagerDuty, then calling it done. The problems with that approach only become visible at the worst possible time.

The three broken on-call models

1. The developer who gets paged for infrastructure they didn't build

Without a dedicated infrastructure engineer, on-call often rotates through the engineering team. A senior developer is on primary this week. At 2am, PagerDuty fires: database connection pool exhausted. The developer can see something is wrong. They can look at the error. What they can't do is safely modify connection pool settings in a production database cluster they've never touched.

The best outcome: they escalate to the CTO who also isn't at their best at 2am. The worst outcome: they try something, make it worse, and now two people are panicking instead of one. Either way, you've burned hours of engineering time and built zero institutional resilience — the same incident will happen again and play out the same way.

2. The one infrastructure engineer on permanent rotation

Having one person who actually understands the infrastructure and putting them on 24/7 on-call is the most common model at Series A companies — and the most reliable way to lose that person within 18 months. The math is simple: being the single point of failure for production is an exhausting position that compounds over time. The first month is manageable. The first year is a grind. The second year doesn't happen.

It also creates a knowledge silo that makes the eventual departure catastrophic. We've inherited three separate engagements in the past two years that started with "our infrastructure engineer left and nobody knows how anything works." The infrastructure was fine. The documentation was nonexistent. The runbooks were in the engineer's head.

3. "We'll deal with incidents as they come in"

Smaller teams sometimes skip formal on-call entirely. Alerts go to a Slack channel. Whoever sees it first handles it. This works until the first serious incident at 4am on a Sunday — when whoever "handles it" is asleep, and the first indication of a problem is a customer email at 9am asking why their data hasn't synced in eight hours.

This model also builds a hidden cultural problem: the engineers who check Slack most frequently — often the most conscientious ones — end up de facto on-call every day. They're providing coverage without recognition, without rotation, and without any of the structural support that makes on-call sustainable.

What good on-call actually requires

Good on-call is not a pager rotation. It's three things in combination, and the rotation is the least important of the three:

Someone who can resolve the incident, not just escalate it. The on-call engineer needs to understand the system well enough to diagnose and fix the most common failure modes without waking anyone else up. If the on-call rotation includes people who can't resolve infrastructure incidents — developers who work entirely on product, engineers new to the company — the rotation is theater, not coverage.

Runbooks for the incidents that actually happen. Not hypothetical incidents. The ones that actually fire. We audit on-call alert histories for every client: 80% of pages over a 3-month period are typically 4–6 repeating incident types. Those are the ones that need runbooks. A runbook doesn't need to be comprehensive — it needs to answer three questions: how do I assess severity? What are the first three steps? Who do I escalate to if those don't work?

An escalation path that isn't "hope someone is awake." Define your tiers explicitly and write them down: Tier 1 is the on-call primary. Tier 2 is the on-call secondary plus the engineering lead. Tier 3 is full incident command with the CTO and potentially the CEO. The threshold for Tier 3 is "production is down and we have no ETA on resolution" — not "something looks wrong." Without explicit thresholds, every incident either escalates too fast (burning leadership on manageable problems) or too slow (burning users on unacknowledged outages).

The rotation structure that works without burning people out

The minimum viable rotation for infrastructure on-call that's sustainable long-term: two engineers who can both resolve infrastructure incidents, on alternating weekly primary/backup rotations. Engineer A is primary this week and backup next week. Engineer B is backup this week and primary next week. Every page goes to primary; if primary doesn't acknowledge within 15 minutes, it escalates to backup.

Neither engineer is permanently the fallback. Neither accumulates all the context. When a new engineer joins who's capable of handling production incidents, they phase into the rotation gradually — shadow on-call first, then backup, then primary — before they're carrying actual load.

The critical constraint: both engineers on the rotation need to be capable of handling at least 80% of pages without escalating. If they can't, you have a knowledge distribution problem, not a scheduling problem. The solution is documentation and cross-training, not adding more people to the rotation who also can't resolve the incidents.

Alert fatigue is a design failure, not a staffing problem

The on-call rotation that nobody dreads isn't the one with the most engineers in it — it's the one where most nights, the pager doesn't go off. That requires deliberate alerting design.

We reviewed one client's PagerDuty history after they complained their on-call rotation was destroying morale. They'd received 47 pages in the previous month. We worked through each one:

• 14 were for a pod restart that always recovered within 30 seconds — not an alert, an event worth logging
• 11 were for CPU above 70% that never crossed 80% and never affected response times — wrong threshold
• 9 were for a certificate with 30 days remaining, paging daily — one alert was sufficient
• 8 were genuine incidents worth paging for
• 5 were test alerts from a monitoring configuration that had never been cleaned up

They hadn't solved their on-call problem by adding more engineers. They'd just distributed the noise more broadly. The fix was reducing 47 monthly pages to 8 — the ones that actually required action.

The rule we apply: if a page doesn't require a human to do something within 15 minutes to prevent a worse outcome, it shouldn't be a page. It should be a log entry, a dashboard metric, or a weekly digest. Violating this rule is how on-call rotations stop feeling like responsible engineering and start feeling like being a human monitoring dashboard.

When your team genuinely can't provide 24/7 coverage

At seed stage with 8 engineers, nobody is staffed to provide real infrastructure on-call around the clock. There are three honest options:

Design the infrastructure to fail gracefully. Self-healing deployments (Kubernetes restarts crashed pods), redundancy (multiple replicas so a single failure doesn't take down the service), and automatic scaling (so a traffic spike doesn't require manual intervention) all reduce the class of incidents that need a human response. You can't eliminate 3am pages through architecture alone, but you can make them rare.

Use managed services for the components most likely to generate incidents. Managed Kubernetes (EKS, GKE) instead of self-hosted. Managed databases instead of self-administered PostgreSQL clusters. Managed queues instead of self-hosted Redis. Each of these trades some control for someone else handling the 2am disk fill on the primary node.

Use an outsourced infrastructure team that includes 24/7 coverage as part of the engagement. For teams where infrastructure isn't yet big enough to justify two dedicated in-house engineers, a fractional team with rotation coverage eliminates the "single engineer who absorbs everything" problem without the cost of two full-time hires.

"We'd been paged 47 times in a month and assumed we needed more engineers on-call. We didn't. We needed to stop alerting on things that didn't require anyone to do anything."

On-call isn't a staffing problem. It's a design problem. The companies with healthy on-call rotations didn't find the right person to absorb the operational burden — they built infrastructure that doesn't generate 3am pages for normal operational events, and defined runbooks so whoever is paged can actually resolve the incident rather than wake up three more people to figure out what to do.