Every new client we onboard has at least one of these: CloudWatch dashboards nobody looks at, application logs that disappear when pods restart, or no alerting at all — just customers reporting outages in the support queue. Usually two of the three. Sometimes all three at once.
This isn't negligence. When you're moving fast and building product, observability is the thing that feels optional until it isn't. The first time you're debugging a production incident with no logs and no metrics is the moment you understand why it needed to happen sooner.
Here's the stack we deploy at the start of every engagement — what it is, why each piece is there, and how we get to something genuinely useful within the first two weeks.
What observability actually means (and what it doesn't)
Observability has become a marketing term that now means everything and nothing. For our purposes it means one thing: can your team diagnose an unknown production failure without adding new instrumentation to the running system?
The "unknown" is the key word. Monitoring tells you when predefined thresholds are breached. Observability lets you ask arbitrary questions about what your system was doing when something went wrong — and get answers from data that was already being collected. If you can only diagnose failures you anticipated, you're monitoring. If you can diagnose failures you didn't anticipate, you're observing.
The practical implication: a CloudWatch dashboard with five metrics and an alert on CPU is monitoring. A system where you can correlate a spike in error rate with a specific deployment, trace it to a single slow database query, and see the logs from every pod that processed affected requests — that's observability.
The three pillars we build first
Every observability build starts with the same three layers, in order of operational priority:
1. Metrics (Prometheus + Grafana)
Metrics are the fastest signal. A request rate dropping to zero, error rate crossing 1%, or p99 latency doubling — these show up in metrics seconds after they start happening. That's why metrics come first.
Our baseline Prometheus deployment scrapes:
- kube-state-metrics — Kubernetes object state (pod restarts, deployment conditions, node pressure)
- node-exporter — host-level CPU, memory, disk I/O, network
- cAdvisor — container-level resource usage (already built into kubelet)
- Application metrics — RED metrics (Rate, Errors, Duration) from each service, exposed on
/metrics - External services — RDS via CloudWatch exporter, Redis via redis-exporter, PostgreSQL via postgres-exporter
Grafana sits on top of Prometheus as the query and dashboard layer. We start with the Kubernetes dashboards from the community (cluster-level, namespace-level, pod-level) and the dashboards from kube-prometheus-stack. They're not perfect for every team, but they're a working baseline in under an hour — and they reveal every resource pressure issue that was invisible before.
The application RED metrics dashboard is different. We build that one specifically for each client, because the most important metric for a payment service is different from the most important metric for a video transcoding queue. The questions are always the same though: how many requests per second, what's the error rate, what does the latency distribution look like, and is any of that changing compared to yesterday?
2. Logs (Loki + Promtail, or OpenSearch for high-volume)
Logs answer the "what was happening at exactly T-minus-3-minutes" question that metrics can't. Metrics tell you something went wrong. Logs tell you what it was.
For most clients (under ~50 GB/day log volume), we deploy Grafana Loki with Promtail as the log shipper. The setup ships all container logs to Loki, where they're indexed by label (namespace, pod, container, app) and stored compressed. Loki's query language (LogQL) makes it easy to grep across all pods for a given service at a specific time window — something that's painful with CloudWatch Logs Insights and borderline impossible with raw pod logs.
Two things we always configure that most teams skip:
- Structured logging enforcement — JSON log output from every service. Free-text logs are grep-able; structured logs are queryable. If an application is logging
user 8472 checkout failed, we work with the engineering team to get it logging{"event":"checkout_failed","user_id":8472,"reason":"payment_gateway_timeout","duration_ms":4823}. The difference in debuggability during incidents is enormous. - Retention and cardinality guards — Loki is cheap to run but expensive when cardinality explodes. We set log retention to 30 days for application logs, 90 days for audit logs, and we add label cardinality limits to Promtail configs so a misbehaving service can't create millions of unique label combinations that kill the indexer.
For clients with very high log volume (data pipelines, high-traffic APIs), we use OpenSearch instead of Loki — it handles cardinality better at scale, and the query experience for security/compliance log analysis is stronger.
3. Distributed tracing (OpenTelemetry + Tempo, or Jaeger)
Tracing comes third because it requires application code changes and takes longer to roll out. But it's the layer that makes the other two navigable for complex systems.
Without tracing, debugging a slow API call in a microservices system looks like this: an alert fires on high p99 latency in the API service. You check the API service logs — no obvious errors. You check the database metrics — CPU is normal. You check the cache hit rate — looks fine. You check the message queue depth — also fine. After 20 minutes of cross-referencing four different dashboards, you find the issue: a third-party payment provider added a new validation step that takes 800ms, and it's called synchronously on every checkout request.
With tracing, the same investigation takes 2 minutes: open the slow trace in Grafana, expand the waterfall, see that the payment provider call is taking 800ms while everything else is under 20ms. Done.
We instrument services with OpenTelemetry — the vendor-neutral standard — and store traces in Grafana Tempo, which integrates with Loki and Prometheus so you can navigate from a metric spike to the traces that were happening at that time to the logs from those traces. The three pillars become one unified investigation workflow instead of three separate tabs.
The dashboards we build in week one
A Grafana instance with 40 dashboards is as useless as one with zero. The goal is a small number of dashboards that the team actually opens during incidents. We build three mandatory ones before anything else:
The SLO dashboard
One dashboard per service showing: availability (successful requests / total requests), p50/p95/p99 latency, error rate, and a traffic volume graph. No more than 6 panels. This is the first screen anyone opens when something might be wrong. It should answer "is this service healthy?" in under 10 seconds.
The deployment impact dashboard
Every deploy should answer the question: did this make anything worse? We build a dashboard that overlays deployment markers on top of error rate and latency graphs for the previous 48 hours. You deploy, wait 5 minutes, look at this dashboard. If the lines are flat, you're done. If a line jumps at the deployment marker, you know what caused it.
The infrastructure health dashboard
Node CPU, memory, and disk pressure across the cluster. Pod restart counts by namespace. PVC usage trends. Database connection pool saturation. This is the screen on-call checks first at 3am — it either rules out infrastructure as the cause immediately or points directly at the problem.
Alerts that actually wake someone up
Most teams we inherit have either no alerts or too many. No alerts means you find out about outages from customers. Too many alerts means PagerDuty fires 30 times a week for things that resolve themselves, the on-call engineer mutes everything, and you find out about outages from customers anyway.
The alerts we configure for every deployment, in order of severity:
P1: Wake someone up at any hour
- Service availability below 99% for 5 consecutive minutes
- All pods for a deployment in CrashLoopBackOff
- Primary database unreachable
- Certificate expiry within 7 days (this kills more production services than people expect)
P2: Alert during business hours
- Error rate above 1% sustained for 10 minutes
- p95 latency more than 3× the 7-day baseline
- Pod restart count above 5 in 1 hour for any service
- Node memory pressure (above 85% for 15 minutes)
- PVC usage above 80% on any persistent volume
P3: Slack notification only
- Deployment succeeded / failed
- HorizontalPodAutoscaler scaled up (capacity planning signal)
- Certificate expiry within 30 days
- Scheduled job failed
The critical rule we enforce: P1 alerts must be actionable at 3am by whoever is on-call. If an alert fires and the on-call engineer's response is "I'll look at it in the morning," it's not a P1. Move it down or fix the underlying condition. An on-call rotation where engineers dread PagerDuty is an on-call rotation that will stop working soon.
What we skip at early stage
Not everything belongs in a seed-stage or early Series A observability stack. Things we deliberately defer:
- Full distributed tracing before the services are stable — trace instrumentation adds a maintenance burden to application code. We defer it until the service architecture has settled and the team has bandwidth to maintain the instrumentation.
- Real user monitoring (RUM) — frontend performance data is valuable but it's a second conversation. Get the backend instrumented first.
- Synthetic monitoring — scripted user journey checks every 5 minutes are useful but expensive to maintain as features change. Start with endpoint health checks, graduate to synthetic tests when the product is stable enough that maintaining the scripts doesn't become a full-time job.
- On-call rotation tooling (PagerDuty, OpsGenie) — if you have 3 engineers total, a Slack alert is fine. Rotation tools pay off when you have enough engineers to actually rotate, and you need escalation policies and on-call scheduling that Slack can't handle.
When to use a commercial APM instead
Datadog, New Relic, Honeycomb, and similar tools are excellent products. We use them on engagements where the client is already paying for them or where the operational complexity of self-hosting matters more than the cost difference.
The self-hosted stack (Prometheus + Grafana + Loki + Tempo) costs roughly $300–800/month in compute and storage for a medium-complexity application. Datadog at equivalent coverage for 20 hosts with APM, logs, and custom metrics costs $4,000–8,000/month. The commercial APM is easier to set up and easier to hand off to an ops-light team. The self-hosted stack is cheaper at scale and doesn't have per-host pricing that punishes you for running many small services.
Our recommendation: if the engineering team is going to own and operate the observability stack long-term and has the bandwidth to run it, self-hosted. If the team is small, non-DevOps-native, or you're in a cost-is-secondary phase, commercial APM saves more in engineering time than it costs in vendor spend.
"We had CloudWatch. We thought that was observability. The first real incident after the new stack went in, we found the root cause in 6 minutes. Before, that same class of incident took us 4 hours and a war room."
The shift that actually matters
The technical stack is the easy part. The harder shift is cultural: getting engineering teams to treat observability as a first-class output of the development process, not something DevOps installs on top of the application afterward.
Every new feature should ship with its metrics. Every error case should log a structured event. Every external dependency should be wrapped with a span. These aren't DevOps requirements — they're engineering quality requirements, in the same category as tests and documentation.
Teams that get there stop having 4-hour war rooms. They also stop having the 11pm Slack message that starts with "is anyone seeing something weird in prod?" Because the answer to that question is no longer "let me check with the person who was on-call last time this happened." It's a 30-second Grafana query away.