Your on-call engineer gets paged at 2:47am. CPU on the API pods is at 74%. They check the dashboards, don't see anything obviously wrong, wait five minutes, and the alert resolves on its own. This is the third time this week. They go back to sleep, annoyed but not alarmed, and log it as a false positive. Two days later, a customer reports that checkout was broken for 45 minutes on Tuesday night. The error was buried in a service the alert didn't cover.
This is not a staffing problem. Adding a second on-call engineer won't fix it. It's a design problem: the alert stack is optimizing for noise while the real signals are invisible.
We review on-call health as part of every engagement. Here's what broken looks like, and what well-designed looks like.
Why most alert stacks are wrong by design
Alerts in most companies accumulate the same way technical debt does — gradually, one reasonable decision at a time, until the aggregate is unmanageable. An engineer sets up a CPU alert when a server ran hot once. A developer adds a memory alert because something OOM-killed a pod last quarter. Someone installs a monitoring tool that comes with 60 default alerts and enables all of them. Nobody reviews the full list together. Nobody asks: "Would waking an engineer up for this at 3am be justified?"
The result is a high-volume, low-signal alert stream. Engineers learn to distrust it. They start sleeping through pages, triaging in the morning, or silencing alerts rather than fixing them. The system degrades further because now even the real alerts feel like noise.
We call this the alert-credibility collapse. Once it happens, the only fix is a full reset — not more alerts, not better dashboards, but a deliberate redesign from scratch.
The principle: alert on symptoms, not causes
CPU at 74% is a cause. It might indicate a problem. It might not. Whether it matters depends on whether it's affecting users — and a CPU metric alone can't tell you that.
A user-facing symptom is something like: error rate on the checkout endpoint exceeding 1% over a 5-minute window, or p95 response time on the API exceeding 2 seconds for 3 consecutive minutes. These are things users experience. CPU utilization is an internal system state that may or may not correlate with user experience, depending on the workload.
Every alert that wakes an engineer should be answerable with: "If this alert is firing, users are being harmed right now." If the honest answer is "maybe" or "it depends," the alert doesn't belong in PagerDuty. It belongs in a dashboard someone checks during business hours.
A practical alert taxonomy
We structure alerts in three tiers:
Tier 1 — Page immediately (24/7, wake-up-worthy)
Reserved for conditions where user impact is happening now and getting worse. Criteria: objectively measurable, user-facing, and not self-resolving without intervention.
- Error rate on critical paths (checkout, auth, payment) > 2% for 5 minutes
- Service completely unavailable — health check failing for 3+ consecutive checks
- Certificate expiry within 72 hours (enough lead time to fix before users are affected)
- Data pipeline SLA breach — nightly job failed or ran > 2x expected duration
Tier 2 — Notify during business hours (Slack, not PagerDuty)
Conditions that indicate degradation or drift that will become a Tier 1 problem if left unaddressed, but don't require immediate human response.
- p95 latency trending 50% above baseline for > 30 minutes
- Error rate elevated (0.5–2%) but stable — not growing
- Disk usage above 80% on a persistent volume
- CPU or memory requests exceeding 85% of limits — approaching OOM territory
- Dependency (third-party API, database) showing elevated error rates
Tier 3 — Dashboard only (no notification)
Metrics worth tracking over time but not worth anyone's attention until they cross a threshold. CPU utilization, memory usage, request rates, cache hit ratios. These live in dashboards. They inform quarterly planning. They do not page engineers.
The runbook requirement
Every Tier 1 alert must have a runbook — a short document that answers three questions: what does this alert mean, what are the most likely causes, and what do I do first. The runbook doesn't have to be exhaustive. It has to be enough that a tired engineer at 3am can orient themselves in under two minutes.
The discipline of writing a runbook before enabling an alert is powerful precisely because it forces you to answer: "What would I actually do if this fired?" If the answer is "I'd look around until I figured it out," the alert isn't ready to page anyone. Either the runbook needs to be written or the alert needs to be demoted to Tier 2.
We've seen teams cut their alert volume by 60% just by enforcing this rule — because half their alerts had no actionable response.
SLO-based alerting
The most mature approach — and the one we implement for clients with complex, multi-service architectures — is SLO-based alerting. Instead of alerting on individual metrics, you define a Service Level Objective (for example: 99.5% of checkout requests complete successfully within 3 seconds, measured monthly) and alert when the error budget is being consumed faster than expected.
The practical advantage: SLO burn rate alerts fire when something is genuinely threatening your availability target, not when a random metric crosses a threshold. A 5-minute spike in CPU that doesn't affect error rates doesn't touch the error budget — no alert. A sustained 2% error rate that burns through the weekly error budget in 2 hours — that fires immediately.
This requires investment in defining your SLOs, building the calculation pipeline, and educating the team on how to interpret burn rate alerts. But the payoff is an alert stack that only pages when users are actually at risk, which is the only criterion that matters.
The on-call rotation itself
Alert design is only half the problem. The rotation design matters too. The patterns that reliably burn out engineers:
- One person per rotation, indefinitely. A single person on-call every week has no recovery time. Cognitive load from interrupted sleep compounds over months.
- No escalation policy. If the primary on-call doesn't respond, who gets paged? "Everyone checks Slack" is not an escalation policy.
- No on-call handover. The outgoing engineer should brief the incoming one on anything that's in a degraded state, any alerts that have been noisy recently, and any changes planned for the upcoming week that carry deployment risk.
- No postmortem after significant incidents. If an incident doesn't produce a written postmortem with action items, the conditions that caused it remain in place. The same incident recurs. Trust in the system erodes further.
The rotation we recommend for teams under 20 engineers: minimum 3-person rotation (one week on, two weeks off), PagerDuty or equivalent with explicit escalation levels (primary → secondary → manager), and a 30-minute weekly on-call sync every Monday where the outgoing and incoming engineer walk through the previous week's alert log together.
"We had 200 alerts set up. We deleted 140 of them in the first session with DevOps Team. Our on-call engineer went from being paged 12 times a week to 2. He told me it was the first month in a year he'd felt like a human being."
Where to start
Pull your on-call log for the past 30 days. For every alert that fired, note: did a human have to do something in response, or did it resolve on its own? If more than 30% of alerts required no human action, your alert stack has an SNR problem. That's the threshold that separates an informative system from one that trains your team to ignore it.
Start there. Work backwards from the alerts that actually required action and ask what they have in common. Those are your real Tier 1 signals. Everything else can be demoted, tuned, or deleted.