From "Users Are Complaining" to 5-Second Incident Detection: Our Observability Playbook

The incident report said the outage lasted 47 minutes. Customer support told a different story: the first ticket had come in 68 minutes before the on-call engineer was paged. For 21 minutes, users were experiencing failures that the team had no visibility into.

This is the most common observability gap we see in fast-growing engineering teams: they have logs, they have uptime checks, maybe they have Datadog or New Relic on a few services — but when something breaks in a way the dashboard doesn't cover, the first signal is a support ticket.

Here's the stack we built for this client — a 60-engineer SaaS company running on Kubernetes — and what changed in the three months after we deployed it.

The starting point

When we started the engagement, their observability posture was:

• Application logs: unstructured text, printed to stdout, accessible only by kubectl or a log aggregator with no retention policy
• Uptime monitoring: Pingdom on the main app URL — checked every minute, meaning a full-failure outage could go undetected for up to 60 seconds
• Infrastructure metrics: basic CloudWatch dashboards for EC2 CPU and RDS connections — no application-level metrics, no per-service error rates
• Alerting: CloudWatch alarms emailing a shared inbox nobody watched outside business hours
• Incident response: ad hoc — whoever saw the Slack message first started investigating

The team knew this was insufficient. The 47-minute incident had been the third time in two months that customers found out about a degradation before the engineers did. The cost wasn't just engineering time — it was three NPS-destroying support conversations and one enterprise renewal discussion that got complicated.

What we built

Layer 1: Structured logs with Loki

The foundation of observability is structured logs you can query. We standardized all services on JSON log output with a consistent schema: timestamp, level, service, trace_id, user_id (when applicable), duration_ms, and message. Services that were already emitting structured JSON needed a label schema update. Services that were printing plaintext needed a one-day code change to use a structured logger library.

Logs are shipped via a Fluent Bit DaemonSet running on every Kubernetes node, routed to Grafana Loki with per-service labels. Retention is 30 days for application logs, 1 year for audit and security logs in separate streams. Total storage cost: $180/month on S3-backed Loki. They had been paying more than that for a log management SaaS that nobody was actively using.

Layer 2: Metrics with Prometheus and kube-state-metrics

Prometheus was already in the cluster in a basic configuration — scraping kube-state-metrics for node and pod health. We extended it in three directions:

• Application metrics: Every service got a /metrics endpoint via the Prometheus client library (Node.js, Python, or Go depending on the service). Each service exposes request rate, error rate, and latency — the RED method. This took an average of 4 hours per service.
• Database metrics: postgres_exporter on the primary and replica, giving us query latency percentiles, connection pool saturation, replication lag, and slow query counts — all queryable from Prometheus.
• External dependency health: Blackbox Exporter probing every third-party API dependency on a 30-second interval. When Stripe's API started timing out on a Tuesday afternoon, we saw it in Grafana before any transaction failed.

Layer 3: Distributed tracing with OpenTelemetry

Logs and metrics tell you that something is wrong. Traces tell you where. We instrumented the three highest-traffic services with OpenTelemetry, exporting to Tempo (Grafana's trace backend). The trace_id field in every log line means you can go from a log entry to the full request trace in one click inside Grafana.

Full OpenTelemetry instrumentation of every service would have been a multi-month project. Instead, we targeted the three services responsible for 80% of user-visible latency — the API gateway, the job queue processor, and the data export service. Those three alone surfaced two latency bugs the team didn't know existed: a database query pattern that was hitting N+1 on large accounts, and a synchronous third-party API call sitting in the request path that should have been async.

Layer 4: Dashboards that actually get used

Most Grafana setups we inherit have dashboards that haven't been opened in months. We built three and only three:

• Golden signals dashboard: For each service, latency (p50/p95/p99), error rate, request rate, and saturation on a single screen. This is what's on the TV in the engineering room and what engineers open first during an incident.
• Infrastructure health dashboard: Node CPU/memory/disk, pod restarts, pending pods, PVC usage, certificate expiry. Reviewed weekly by whoever is on ops rotation.
• Business metrics dashboard: Active users, job queue depth, API success rate by customer tier, data export latency. Shared with product and customer success — not just engineers.

Layer 5: Alerting with PagerDuty and escalation policies

This is where the 21-minute detection gap gets closed. We defined alert thresholds for two severity levels:

• P1 (page immediately): Error rate above 1% for 2 consecutive minutes on any customer-facing service; p99 latency above 5 seconds for 3 minutes; any pod restart loop; primary database replication lag above 30 seconds
• P2 (alert, no page): Error rate above 0.1%; p99 latency above 2 seconds; any node at >85% memory for 5 minutes; certificate expiring within 14 days

P1 alerts go to PagerDuty with a 5-minute acknowledgment SLA before escalating to a secondary on-call. P2 alerts go to a dedicated Slack channel that the on-call checks twice per shift. The alerting rule file lives in Git alongside the infrastructure code — every threshold change is a reviewed PR, not an undocumented click in a UI.

The results: three months later

• Mean time to detect (MTTD): 21 minutes → 4 seconds (the alert fires before the first user sees an error)
• Mean time to resolve (MTTR): 47 minutes → 11 minutes (traces make root cause identification immediate)
• Support tickets from infrastructure incidents: 3 per month → 0 (two incidents occurred; neither reached customers before resolution)
• Incidents found proactively (before any user impact): 4 in three months — memory leak, disk pressure on a worker node, certificate about to expire, slow query degrading one enterprise tenant

"The first time PagerDuty woke me up for something that hadn't hit a single user yet, I realized how much we'd been operating blind. We went from reactive to actually running the service."

What we'd do differently

OpenTelemetry adoption was the right long-term choice but the slowest part of the project. If we were doing it again, we'd instrument the top three services in week one and leave the rest for a later phase instead of trying to scope it comprehensively upfront. Partial tracing that's actually deployed is more valuable than a perfect tracing plan that ships six months later.

We'd also be more aggressive about retiring the old monitoring setup earlier. Running Pingdom alongside Blackbox Exporter created confusion about which alert was authoritative. Parallel systems are fine during cutover; dragging them both into steady state is just noise.

The tools, in one place

• Logs: Fluent Bit (collection) + Grafana Loki (storage + query)
• Metrics: Prometheus + kube-state-metrics + postgres_exporter + Blackbox Exporter
• Traces: OpenTelemetry SDK + Grafana Tempo
• Dashboards: Grafana (self-hosted, in the same cluster)
• Alerting: Prometheus Alertmanager → PagerDuty + Slack
• Alert rules: version-controlled in Git, applied via Helm

Every component is open-source and runs in the same Kubernetes cluster as the application. Total infrastructure overhead: two 2-vCPU nodes. Total monthly cost added to the AWS bill: $190. The 21-minute detection gap had been costing more than that in engineer time every single week.