A client came to us after a 4-hour outage. Their monitoring showed the service was up. Health checks were passing. The load balancer reported all backends healthy. But users couldn't connect.
The issue was a memory leak in a sidecar container slowly starving the main process. It took hours to find because nothing was measuring that container's memory. "Everything looked green," their CTO told us. "That was the problem."
Good observability isn't about having dashboards. It's about knowing when something is wrong — and having enough context to know why, without an hour of log archaeology first. Here's the exact stack we deploy to get there, and the reasoning behind each piece.
The three pillars
Observability in a Kubernetes environment means three things:
- Metrics — numbers over time. CPU, memory, request rate, error rate, latency percentiles. Metrics tell you what is happening.
- Logs — timestamped events. Application output, system events, audit trails. Logs tell you what happened.
- Traces — the path a request took through your system. Traces tell you why it happened — specifically, which service, which database call, which downstream dependency caused a latency spike or error.
Most teams have some version of logs. Most are missing useful metrics and almost everyone is missing traces. The fix is to implement all three systematically rather than reactively.
The metrics layer: kube-prometheus-stack
We deploy the kube-prometheus-stack Helm chart on every cluster. This single chart installs:
- Prometheus Operator — manages Prometheus instances as Kubernetes CRDs, so you define what to scrape in YAML rather than static config files
- Prometheus — the time-series database and scrape engine
- Alertmanager — receives alerts from Prometheus and routes them to PagerDuty, Slack, email
- Grafana — the visualization and dashboarding layer
- node-exporter — exports host-level metrics (CPU, memory, disk, network) from every node
- kube-state-metrics — exports Kubernetes object state: pod restarts, deployment replica counts, resource requests vs. limits
Why this stack over Datadog, New Relic, or other SaaS observability platforms? A few reasons:
- Cost. At scale, Datadog billing for metrics can exceed the cost of the infrastructure you're monitoring. kube-prometheus-stack is open source — you pay for storage, not per-metric or per-host.
- Data ownership. Your metrics stay in your cluster. No data leaving to a third-party SaaS, which matters for regulated industries.
- Kubernetes-native. The Prometheus Operator's ServiceMonitor and PodMonitor CRDs let you define scrape targets as part of your Helm charts — observability configuration lives next to the workload it monitors.
The first dashboards we configure after install are the USE method (Utilization, Saturation, Errors) for infrastructure resources and the RED method (Rate, Errors, Duration) for services. These two frameworks cover the vast majority of what you need to know about system health.
One thing to calibrate: Prometheus retention. The default is 15 days. We typically set this to 90 days for most clusters — enough history to understand seasonal patterns and compare post-deploy behavior against a pre-deploy baseline. For long-term trend data, we configure remote write to a cheaper storage backend (Thanos or Grafana Mimir).
The logging layer: Loki + Promtail
For log aggregation, we use Grafana Loki with Promtail as the collector. If you've used Elasticsearch, the operational model is different in a way that takes some adjustment: Loki doesn't index log content. It only indexes the labels you attach to log streams.
This is the right tradeoff for most Kubernetes environments:
- Elasticsearch indexing everything is powerful but expensive — the index itself can be larger than the logs it describes
- Loki's label-based model is cheap, operationally simple, and fast for the queries you actually run most often (give me all logs for this pod in the last hour)
- Native Grafana integration means one UI for metrics and logs — you can correlate a Grafana alert with the logs from the relevant pod without switching tools
The label strategy matters. We configure Promtail with four labels: namespace, pod, container, and environment. That's it. The most common mistake with Loki is adding too many labels — each unique label combination is a separate log stream, and high-cardinality labels (like pod IP addresses) create a combinatorial explosion that hurts performance.
Promtail runs as a DaemonSet, collecting logs from every container on every node by reading from the container log files on the node filesystem. Zero changes required to applications — it just works. If applications output JSON structured logs, Promtail can parse the JSON and expose fields as labels or for filtering. If they output unstructured text, it works too — just with fewer filtering options.
Retention configuration:
- Application logs: 90 days
- Audit logs (separate stream, separate storage): 1 year minimum, sometimes 3 years for regulated environments
The tracing layer: OpenTelemetry + Grafana Tempo
Tracing is the hardest of the three pillars to retrofit onto an existing application. Metrics and logs require no application changes — you instrument the infrastructure and collect what's already there. Distributed tracing requires your application to propagate trace context between services, which means either auto-instrumentation (available for many runtimes) or manual SDK integration.
Our approach:
- OpenTelemetry Collector as the vendor-neutral collection layer. This is the key decision — it decouples your application instrumentation from the backend. You can switch from Tempo to Jaeger to a SaaS backend without changing a line of application code.
- Grafana Tempo as the trace storage backend. Cheap (traces compress well, and Tempo is designed for cost-efficient storage), fast for the queries that matter, and native Grafana integration for the unified UI.
- Auto-instrumentation where possible. For Java services, the OpenTelemetry Java agent instruments most frameworks (Spring, Quarkus, gRPC) with zero code changes. For Node.js, the OTel SDK auto-instruments common libraries. For Python: same.
Even partial tracing pays dividends quickly. If you only instrument your API gateway and your database calls, you've covered the two most common sources of latency problems. You don't need to trace every internal function call on day one — start at the boundaries and work inward.
One specific configuration that catches most teams by surprise: sampling. Tracing 100% of requests at production scale generates enormous volumes of data. We configure tail-based sampling at the OTel Collector: sample 100% of errors and slow requests (p99 > 500ms), sample 1% of everything else. This keeps storage costs manageable while ensuring you always have trace data for the requests that matter.
Alerting: the part most teams get wrong
The most common alerting mistake is alerting on causes instead of symptoms. "CPU is above 70%" is not an alert — it's a metric. "Users are experiencing a 5% error rate" is an alert. The difference matters because the first fires constantly (every deploy, every batch job, every garbage collection pause) and conditions your team to ignore alerts. The second fires when something actually needs attention.
Our alerting philosophy: every alert must be actionable. Before creating any alert, we ask: "What would the on-call engineer do when this fires?" If the answer is "nothing until it gets worse" or "check if it resolves on its own," the alert is noise. Delete it.
The starter alert set we deploy on every cluster:
- Error rate above 1% for 5 minutes — service-level, evaluated per namespace. This is the most important alert.
- p99 latency above 500ms for 5 minutes — symptom-based; tune the threshold per service after observing baseline
- Pod crash looping — pod restarted more than 3 times in 15 minutes
- Node disk above 85% — 85% is the trigger to investigate, not the trigger to panic. At 95% you're in an emergency; 85% gives you time to act calmly.
- PersistentVolumeClaim above 80% full — Kubernetes doesn't automatically expand volumes; you need to know before they fill
- TLS certificate expiring within 14 days — cert-manager usually handles renewals, but this alert catches cert-manager failures before they become outages
- Deployment not progressing for 10 minutes — catches stuck rollouts before the CI/CD timeout does
Routing: Alertmanager sends critical alerts to PagerDuty (on-call rotation), warning-level alerts to Slack. We create the PagerDuty service and escalation policy as part of the setup. No more "Slack mentions" for production incidents — if someone needs to be woken up, PagerDuty wakes them up.
What the first two weeks look like post-deploy
Installing the stack is day one. Making it useful takes two weeks.
In the first week, we focus on signal quality: are the right things being measured? We look at every alert that fires, every metric that appears anomalous, and ask whether it reflects a real problem or a measurement artifact. False positives get tuned immediately — an alert that fires every day and gets ignored is worse than no alert at all.
In the second week, we build application-specific dashboards. The generic Kubernetes dashboards tell you about cluster health. The useful dashboards tell you about your business: request rate per API endpoint, database query latency per service, queue depth for each background worker. These require knowing the application, so we build them together with the engineering team.
By the end of week two, the team is using the stack. Not just "it exists and the dashboards are there" — actually opening Grafana to investigate slow queries, using Loki to search logs for a specific request, and trusting that an alert-free night means the system is actually healthy.
"Before this, we were flying blind and didn't know it. We had an outage last month — and we had a root cause in 8 minutes. Before the observability work, the same incident would have taken hours."
When to use a SaaS platform instead
The self-hosted Grafana stack is the right choice for most teams we work with. But there are cases where a SaaS platform (Datadog, New Relic, Grafana Cloud) makes more sense:
- Very small clusters with no dedicated ops capacity. Datadog requires zero operational overhead — it just runs. If nobody on the team will maintain Prometheus, a managed SaaS service avoids the operational burden at the cost of higher recurring spend.
- Teams that need APM without code changes. Datadog's agent-based APM instruments many runtimes at the host level without SDK integration. If instrumentation-free tracing is the priority and cost isn't, this is a real advantage.
- Compliance requirements that mandate a specific vendor. Some enterprise customers require specific monitoring vendors in their vendor agreements.
For everyone else: the self-hosted stack is operationally manageable, significantly cheaper at scale, and gives you complete control over your data and retention policies.
The missing piece: SLOs
Metrics, logs, and traces give you the data. Service Level Objectives give you the framework for deciding what to do with it. An SLO is a target: "99.9% of requests will return a response within 200ms." An error budget is the acceptable breach of that target over a rolling window.
SLOs are the next step after the observability stack is stable. We typically introduce them at the 60-day mark — after the team has seen enough baseline behavior to set meaningful targets. The combination of SLOs with the Prometheus/Grafana stack is what takes a team from "we have monitoring" to "we have a principled way to make reliability decisions."
But that's a separate article.