What to Monitor Before Something Breaks: Production Observability for Growing Startups

There's a version of production monitoring that looks fine when nothing is going wrong and fails you completely when something is. It's the version where dashboards exist, some alerts are configured, and then a customer emails to report that their account has been broken for two hours — and you go look at the dashboards and everything appears normal.

The gap between "we have monitoring" and "we know what's happening in production" is larger than most teams realize until they're debugging a production incident with no useful data. This is how to close it — in priority order, because almost everyone tries to instrument too much too fast and ends up with dashboards nobody trusts and alerts nobody responds to.

Start with the user, not the server

The most common monitoring mistake is instrumenting infrastructure first and application behavior second. CPU utilization is a metric. Five percent of API requests returning 500 errors is an incident. Both matter — but only one of them tells you something is actually broken for users.

Before adding any infrastructure dashboards, answer one question honestly: how would I know, within 5 minutes, if users couldn't complete a core action in my product right now? If the answer involves manually checking logs, running queries, or waiting for a support report, you don't have sufficient observability. That's the gap to close first.

The four signals that matter most, in priority order:

• Error rate — percentage of requests failing. HTTP 5xx responses, unhandled exceptions, downstream service failures that surface as errors to users. Alert here first.
• Latency — P95 and P99 response times on your core endpoints. The average is misleading; P99 is what your slowest users experience on every request. A P99 above your SLA is a user experience problem even if the average looks healthy.
• Throughput — requests per second on critical endpoints. A drop in throughput when traffic hasn't decreased is a signal that something is failing silently — requests are timing out or being dropped before they log an error.
• Availability — synthetic monitoring that hits your health endpoint every 30 seconds from outside your cluster. This is the simplest possible check and catches outages within 2 minutes. Every team should have this before anything else.

The four observability layers — and which one teams underinvest in

Good production observability has four distinct layers. Most teams have some of layer one, miss layer two entirely, and haven't thought about layers three and four.

Layer 1: Infrastructure metrics

Is the platform healthy? CPU, memory, disk, and network on nodes and pods. In a Kubernetes environment, kube-state-metrics and node-exporter provide the signals that matter out of the box. What to alert on: node NotReady, disk above 85%, sustained memory above 90% for 10 minutes, pod OOM kills. What not to alert on: every brief CPU spike, warning events from the Kubernetes event log, pods that restart and recover immediately.

Layer 2: Application metrics (where most teams underinvest)

Are my services healthy? This means the RED metrics — Rate (requests per second), Errors (failed requests), Duration (latency) — instrumented at the application level. Not inferred from infrastructure metrics. Not scraped from logs after the fact. Emitted directly from the application code using a Prometheus client library.

This is the layer most teams skip or implement incompletely. Server CPU being normal doesn't mean the API is handling requests correctly. The database being up doesn't mean queries are returning in time. You need the application to tell you directly that it's healthy — infrastructure metrics can't tell you that.

# Python (FastAPI/Flask) — minimal Prometheus instrumentation
from prometheus_client import Counter, Histogram

request_count = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)
request_latency = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency',
    ['method', 'endpoint']
)

Add these to every service that handles user traffic. The data they emit is what makes your alerting on error rate and latency possible. Without application-level metrics, your observability is looking at the building's power consumption to determine if the employees are productive.

Layer 3: Distributed traces

Why did this specific request take 3 seconds? Traces answer the question that metrics can only point toward. A high-latency alert tells you something is slow; a trace tells you it was the payment service's third-party API call that took 2.8 seconds, while your own code took 200ms.

OpenTelemetry is the standard — instrument once, export to any backend (Jaeger, Grafana Tempo, or a managed APM). The critical decision is sampling rate: trace 100% of requests in development, 10–20% in production. Traces are expensive to store; sampling enough to diagnose patterns while not paying for every request is the right balance for most teams at this scale.

Layer 4: Structured logs

What happened, in detail? Logs answer questions that metrics and traces don't — what was the exact payload when this error occurred, which user ID was affected, what was the full stack trace. The important word is "structured": JSON logs with consistent field names (user_id, trace_id, status, duration_ms) you can query by field, not grep through by hand.

Correlating logs to traces via a shared trace_id field is the pattern that makes debugging fast: from the alert, to the trace that shows which service was slow, to the logs from that service during that request window.

The signal vs. noise problem

A monitoring system where every alert gets acknowledged and acted on is a monitoring system that works. A monitoring system where engineers route pages to a "check later" queue because most of them turn out to be nothing is a monitoring system that has failed — it's just going to take an incident to make that visible.

The rule: an alert should require a human to do something within 15 minutes to prevent a worse outcome. If that isn't true, it shouldn't be an alert. It should be a log entry, a dashboard panel, or a weekly summary email.

Common alerts that should be logs instead:

• A pod restarted and recovered in under 30 seconds — log it, don't page for it
• CPU at 75% for 2 minutes — log it; alert at sustained 85%+ with latency impact
• A single 500 error from one request — log it; alert at 1% error rate sustained over 2 minutes
• Certificate expiry in 30 days — one weekly digest email, not a daily page

The cost of bad alert hygiene isn't just annoyed engineers — it's slower response when something actually matters. Teams conditioned to treat pages as noise take longer to respond to the genuine incidents.

The tool recommendation that fits most teams

For a team of 5–50 engineers, this stack handles 95% of production observability needs without significant operational overhead:

Prometheus + Grafana for metrics. Prometheus scrapes metrics endpoints from your services and infrastructure. Grafana visualizes them and manages alert rules. AlertManager routes firing alerts to PagerDuty, Slack, or email. If you'd rather not run these yourself, Grafana Cloud has a generous free tier and a managed Prometheus endpoint you can push metrics to.

Grafana Loki for logs. Cheaper to run than Elasticsearch, integrates natively with Grafana so your metrics dashboard and log search are in the same UI. The LogQL query language is similar enough to PromQL that engineers familiar with one can pick up the other in an afternoon.

Grafana Tempo or Jaeger for traces. Trace storage is cheap because you sample — 10–20% of requests gives you representative data for debugging without the storage cost of 100% trace retention.

If you'd rather buy than build: Datadog covers all four layers in a managed service. Be deliberate about what you enable — log ingestion at full volume for a busy service can add $8,000–15,000/month on top of the per-host charge. Set a spending alert on your Datadog account the same week you set it up, and configure log sampling before you're surprised by an invoice.

The 3am test

Every observability setup we build for a client ends with the same question: if the product had a critical failure at 3am — users couldn't log in, a checkout was broken, data wasn't syncing — would the on-call engineer know within 5 minutes without a customer reporting it?

If the answer is no, that's the alert to write before anything else. Not the infrastructure dashboards. Not the distributed tracing. The alert that tells you the product is broken for users.

For most applications, that alert looks like:

alert: CoreEndpointHighErrorRate
expr: |
  rate(http_requests_total{status=~"5..",endpoint=~"/api/v1/(login|checkout|dashboard)"}[2m])
  /
  rate(http_requests_total{endpoint=~"/api/v1/(login|checkout|dashboard)"}[2m])
  > 0.01
for: 2m
labels:
  severity: critical
annotations:
  summary: "Error rate above 1% on core user endpoints for 2 minutes"

It isn't clever. It doesn't require distributed tracing or sophisticated anomaly detection. It's the thing that tells you the product is broken before your users tell you — which is the entire job of observability.

Don't build too much too fast

The failure mode on the other end from "no monitoring" is "monitoring no one uses." Teams that instrument everything, add every possible alert, and build 40 Grafana dashboards end up with engineers who don't know which dashboard to open during an incident and who ignore the constant alert noise until something is obviously on fire.

The right amount of observability is: you know about problems before your users report them, and you can identify the source of an incident within 15 minutes using the data you have. An error rate alert on your core endpoints, a latency alert at your SLA threshold, infrastructure alerts on disk and memory, and structured logs you can search by trace ID covers most of that for most teams. Build from there based on the gaps you discover in actual incidents — not based on what's theoretically possible to instrument.