The CTO's message was blunt: "Our lead infrastructure engineer is about to quit. He's been paged 11 times a night on average for the last three weeks. We need to fix this or we lose him — and he's the only person who knows how the system works."
This is a pattern we see often at Series A–B companies. The infrastructure works well enough that nobody invested in observability. Then scale hits, and suddenly one person is the human equivalent of a load balancer — absorbing every alert, every 3am page, every customer complaint about slowness. The system doesn't fail all at once. The person does.
Six weeks later, the same engineer was sleeping through the night. Pages were down to 2–3 per week, all actionable. Here's exactly how we got there.
The starting point
When we audited their alerting and incident history, the picture was familiar:
- 147 distinct alert rules in DataDog, most of them set up by copying examples from the internet with thresholds that were never tuned
- Every alert paged the same person — no escalation, no rotation, no secondary
- No severity levels — a disk at 80% triggered the same page as a complete API outage
- No runbooks — resolving each incident required institutional knowledge that only lived in one engineer's head
- No postmortem process — incidents were fixed and forgotten, same issues recurring monthly
- 70% of pages were for conditions that self-resolved within 10 minutes without any action
That last point is the crux of alert fatigue. When most pages don't require action, engineers stop treating pages as urgent. Which means the ones that do require action get the same slow response as the noise. You end up with a system that's simultaneously over-alerting and under-responding.
Week 1–2: Stop the bleeding
Alert audit and immediate culling
Before building anything new, we spent the first week doing nothing but auditing existing alerts. For each of the 147 rules, we asked three questions: Has this fired in the last 30 days? When it fired, did it require human action? If it hadn't fired at all, would anyone have noticed?
The result: 89 alerts were immediately silenced. Not tuned — silenced. They were either pure noise (CPU spikes that lasted seconds), vanity metrics nobody acted on, or conditions already covered by higher-level alerts. Going from 147 to 58 alert rules in a week is aggressive, but it immediately cut overnight page volume by 60%.
Severity tiers
We implemented three tiers — no exceptions:
- P1 (immediate page, any time): Customer-impacting. API error rate >2% for 3+ minutes, checkout flow failures, authentication completely down. Page the on-call engineer immediately, escalate to CTO if no acknowledgment in 10 minutes.
- P2 (page during business hours only): Degraded performance or elevated error rate below the P1 threshold, non-critical service failures. Slack notification overnight, page at 9am if still active.
- P3 (Slack only): Capacity warnings, slow queries, non-critical resource utilization. Check during normal working hours.
This sounds obvious. Every engineering team knows they should do this. Almost none of them actually enforce it. The reason it doesn't happen is that defining P1 criteria requires someone to make hard decisions about what constitutes "customer-impacting" — and those decisions feel uncomfortable until you've lost sleep to a false alarm for the 40th time.
Week 3–4: Runbooks and rotation
Runbook-first culture
We picked the 15 most common alert types and wrote runbooks for each before we did anything else. Not wiki pages — actual numbered decision trees. Alert X fires → check metric Y → if above threshold, run command Z → if not resolved in 10 minutes, escalate.
The key constraint: if a runbook required anything not in the runbook, we added it to the runbook. The goal was for any engineer on the team — not just the infrastructure lead — to be able to follow it at 3am without Slacking anyone for help. We validated each runbook by having a backend engineer who had never touched infrastructure walk through it against a staging environment.
Runbooks that couldn't be made self-contained pointed to one of two things: either the alert was bad (alerting on a symptom too far upstream from the cause) or the system had a design problem worth fixing. We fixed several underlying issues this way.
Building the rotation
The engineer had been on-call 365 days a year with no backup. We built a 4-person rotation from engineers who each owned different parts of the stack. The catch: none of them had ever been on-call for the full system. So we ran a 3-week shadow period — the original engineer remained the primary responder, but each rotation member shadowed every real incident, working through the runbook themselves while the lead watched.
By the end of the shadow period, each rotation member had seen at least 8 real incidents. The runbooks had been battle-tested and updated. The rotation went live in week 5 with the original engineer as secondary escalation only.
Week 5–6: Fixing the root causes
SLOs over thresholds
We replaced most of the raw metric alerts with SLO-based alerting. Instead of "alert if error rate >1% for 5 minutes," we defined a 99.5% availability SLO for the API and used DataDog's error budget tracking to alert when the budget was burning faster than sustainable. This is a fundamentally different posture: you're alerted when reliability trajectory is problematic, not when a single metric spikes briefly.
The practical effect: eliminated another 12 alerts that were catching brief spikes that never meaningfully impacted uptime, while ensuring that sustained degradation — even subtle — got flagged before it burned through the error budget.
The postmortem habit
We implemented a lightweight postmortem template for every P1 and any P2 that recurred more than twice. Not a blame document — a five-field form: what happened, how we detected it, how we resolved it, why it happened, what prevents recurrence. Takes 20 minutes to fill out. Gets filed in Notion within 24 hours of resolution.
The postmortem discipline caught two systemic issues in the first month: a memory leak that was triggering weekend restarts, and a third-party API integration that was silently failing in a way that looked like high latency. Both had been causing alerts for months. Both were fixed permanently.
The results
- Overnight pages: 11/night → 0.4/night average (96% reduction)
- Alert rules: 147 → 58 (all actionable)
- Mean time to resolution: 47 minutes → 11 minutes (runbooks + rotation knowledge)
- Engineers on rotation: 1 → 4
- Recurring incidents: 8 in 3 months before → 1 in 3 months after (postmortems)
"I slept through the entire weekend for the first time in eight months. I didn't realize how much the constant interruption was affecting my judgment during the day until it stopped."
What we'd do differently
The runbook-first approach was the right call, but we underestimated how much institutional knowledge the lead engineer had that he didn't know he had. He would write a runbook step like "check the queue depth" without realizing that the reasoning behind that step — which queue, what threshold, and why — wasn't obvious to anyone else. A second pass through every runbook with a fresh reader is non-negotiable on future engagements.
The other thing: SLOs should be defined before runbooks, not after. Once you have SLOs, alert design becomes obvious. Without them, you're making up thresholds and hoping they're right.
Alert fatigue isn't a monitoring problem. It's a prioritization problem masquerading as a technical one. The tooling is largely irrelevant — DataDog, Prometheus, Grafana, Honeycomb, it doesn't matter. What matters is someone making deliberate decisions about what warrants waking a person up, writing it down, and distributing the burden across a team that's prepared to handle it.