The VP of Engineering's first message to us was blunt: "Our on-call rotation is destroying morale. Two engineers have said they'll quit if it doesn't get better. We get paged constantly and half the time there's nothing wrong."
The team — a 35-engineer SaaS company running on GKE — had built up monitoring organically over three years. Every time something broke, someone added an alert. Nobody ever removed one. The result: 80+ PagerDuty pages per week, roughly half of them either false positives or issues that self-resolved before the on-call engineer even opened their laptop. Mean time to recovery for real incidents was 47 minutes — not because the problems were hard, but because nobody could find the relevant context in the noise.
Six weeks later, they were averaging 11 pages per week, all of them actionable. MTTR was down to 9 minutes. Here's what we changed.
The diagnosis: what alert fatigue actually looks like
We started with a three-week audit of their PagerDuty incident history before touching anything. Across 240 incidents:
- 41% self-resolved — the alert fired, the on-call acknowledged it, and by the time they started investigating, the metric had returned to normal
- 18% were duplicates — the same underlying issue triggering multiple unrelated alerts (a single slow database query firing CPU, latency, and error rate alerts simultaneously)
- 22% had no runbook — the on-call engineer had to figure out from scratch what to do, adding 20–30 minutes to every investigation
- Only 19% were real, unique, actionable incidents
The monitoring stack wasn't broken — it was thorough in the wrong direction. It was measuring the right things but alerting on the wrong level of abstraction. Individual pod CPU spikes don't tell you whether users are experiencing a problem. A 5-minute CPU spike on one of 10 pods is rarely worth waking someone up at 3am.
The shift: from machine metrics to user impact
The core insight behind fixing alert fatigue is this: alerts should fire when users are experiencing degradation, not when infrastructure metrics cross a threshold. The former is what matters. The latter is just a proxy — and often a noisy, unreliable one.
We introduced SLOs (Service Level Objectives) as the primary alerting layer. For each of their customer-facing services, we defined:
- Availability SLO: 99.5% of requests succeed (non-5xx) over a 30-day window
- Latency SLO: 95th percentile response time ≤ 500ms over a 30-day window
Alerts fire when the error budget is burning too fast — not when a single metric spikes. A useful heuristic: if the SLO burn rate is high enough that you'll exhaust the month's error budget within 24 hours, page someone immediately. If it's only fast enough to exhaust it within a week, create a ticket for business hours. If it's within normal burn, do nothing.
This single change eliminated the "something looks weird but it's not actually affecting users" category of pages entirely.
What we built
Prometheus + Grafana + Loki (unified observability)
They had Datadog for metrics and a hodgepodge of CloudWatch logs and ad-hoc Elasticsearch. The cost was significant ($8K/month on Datadog alone) and the context-switching between tools was slowing down investigations. We migrated to a self-hosted Prometheus/Grafana/Loki stack on their GKE cluster:
- Prometheus for metrics, with kube-state-metrics, node-exporter, and custom application metrics instrumented via the Prometheus client libraries
- Grafana as the unified visualization layer — metrics, logs, and traces in a single interface
- Loki for logs, with Promtail on every node — structured JSON logs indexed by labels, queryable with LogQL
- Tempo for distributed traces, connected to Grafana so you can go from a slow request in the latency dashboard to the trace to the relevant logs in under 30 seconds
Total infrastructure cost: ~$1,200/month. Datadog bill: canceled. Net savings: $6,800/month just from the observability stack consolidation.
SLO-based alerting in Grafana
We defined SLOs for their six customer-facing services and implemented the multi-window, multi-burn-rate alerting pattern from Google's SRE Workbook. The alert rules live in a Git repository alongside the service code — infrastructure as code, same as everything else. Any engineer can see why an alert exists, what SLO it's protecting, and what a violation means for users.
Runbooks, finally
For every alert that remained after the cleanup (we went from 340 active alert rules to 87), we wrote a runbook. Each runbook is a Markdown file in Git, linked directly from the PagerDuty alert body. Structure:
- What fired: plain-language description of the alert condition
- What it means: the user impact if this is real
- Immediate checks: 3–5 kubectl/Grafana commands to confirm the issue and assess severity
- Common causes: the two or three things that have caused this alert before, with links to past incidents
- Resolution steps: ordered by likelihood
- Escalation: who to call if the common causes don't apply
Writing 87 runbooks sounds like a lot. We did it in one week by doing them in order of alert frequency — the 20 most-fired alerts account for 80% of on-call volume. We started there, and the team filled in the rest over the following month.
Alert routing cleanup
Not every alert needs to wake someone up at 3am. We restructured PagerDuty routing into three severity levels:
- P1 — Page immediately: User-facing SLO breach. Error budget burning at critical rate. Requires human response now.
- P2 — Slack only (business hours): SLO burn rate elevated but not critical. Something to look at today.
- P3 — Daily digest: Capacity warnings, infrastructure health signals, cost anomalies. Things that matter but never needed to wake anyone up.
Before this change, 90% of alerts were routed identically — all of them capable of generating a 3am page. Triaging by severity let the on-call schedule focus on what actually required immediate human response.
The migration: don't rip, layer
We didn't turn off the old monitoring the day we turned on the new. We ran both systems in parallel for two weeks, comparing alert volumes and verifying that the new SLO alerts would have caught every real incident the old system had caught. Only after two weeks of data did we route PagerDuty to the new alerting and archive the old rules.
This is the move most teams skip in their rush to "clean up" monitoring. Don't. You will miss something, and you'll find out the hard way.
The results
- Weekly pages: 80+ → 11 (86% reduction)
- False positive rate: ~60% → under 5%
- MTTR for real incidents: 47 minutes → 9 minutes
- On-call burnout: Two engineers who had threatened to quit are still on the team
- Observability cost: $8,000/month → $1,200/month
The MTTR improvement came almost entirely from runbooks. Having a document that tells you exactly what to check and in what order turns a 45-minute investigation into a 10-minute checklist. The knowledge was always in someone's head — we just extracted it and made it findable at 3am.
"I used to dread being on-call. Now it's genuinely fine. The pages that come through are real, the runbooks tell me what to do, and I actually sleep. I didn't think that was possible."
The underlying principle
Alert fatigue is a signal that your monitoring was built reactively — added alert by alert in response to incidents, never pruned. The solution isn't more tooling. It's a deliberate redesign around one question: is a user experiencing degradation right now? If yes, page. If not, don't.
Every additional alert that doesn't answer that question erodes the on-call engineer's trust in the system. Once engineers learn to ignore pages, you've lost the thing monitoring was built to do.