The Production Readiness Review We Run on Every New Client

A SaaS company came to us 18 months after their initial launch. They had 3,000 customers, a stable product, and a belief that their infrastructure was solid — they'd been running in production without a major incident. The week after we onboarded, they ran a Black Friday-style promotion that brought 8x normal traffic. Three things broke at once: the database connection pool maxed out, an autoscaling policy that had never been tested didn't trigger, and an S3-dependent feature silently failed because nobody had set up alerting for that path.

They hadn't been "in production." They'd been lucky. The promotion was the first time the system had faced real stress, and the gaps that had always been there became visible simultaneously.

A production readiness review isn't a one-time checklist you run before launch. It's a recurring audit of whether your system will hold up under conditions you haven't seen yet. Here's exactly what we check, what we find, and how we prioritize the results.

What a production readiness review is not

It is not a code review. It is not a security audit (though it overlaps with one). It is not a certification you receive and forget. It's a structured answer to one question: if something unexpected happens — 10x traffic, a node failure, a bad deploy, a third-party API going down — does your team know what breaks, how to detect it, and how to recover?

The output is a prioritized list of findings, not a report that sits in a drive. Findings go into the sprint. Critical ones get fixed this week. High-severity ones get a remediation date. Medium ones go into the backlog with an owner.

The six categories we audit

1. Deploy and rollback

Most teams can deploy. Fewer can roll back quickly under pressure. The questions we ask:

• How long does a deploy take from merge to production? (More than 15 minutes is a flag.)
• When did you last test a rollback? Not "we have rollback capability" — when did you actually run it in staging?
• If a deploy causes errors within 5 minutes, what's the fastest path to recovery — re-deploy the previous version, or a one-command rollback?
• Are database migrations backward-compatible? Would the previous version of the app work on the current schema?

The most common finding here: rollbacks have never been tested. The team knows how to deploy forward; they've never verified the path backward works under real conditions. When they need it is not the time to find out.

2. Observability

Can your team diagnose an incident without SSH access to production boxes? The bar is higher than it sounds. We look for:

• Structured logs — JSON-formatted, with request IDs that span across service calls. If your logs are unstructured strings, correlating an incident across three services takes hours instead of minutes.
• RED metrics per service — Request rate, Error rate, Duration (p50, p95, p99). Not just "is the service up," but "is it working correctly."
• Distributed tracing — For multi-service architectures, you need to be able to follow a single request across service boundaries. Without this, a slow request that touches four services produces four sets of logs with no obvious connection between them.
• Alert coverage — Alerts that fire on symptoms the user experiences, not just infrastructure metrics. CPU at 80% is not an alert. Error rate above 1% for 5 minutes is.

In practice, we find that about half of new clients have monitoring but not observability. They have dashboards that show infrastructure health (CPU, memory, disk). They don't have dashboards that answer "is the checkout flow working right now?"

3. Database resilience

The database is where production disasters are most consequential and most expensive to recover from. We check:

• Automated backups with tested restores. Backups that have never been restored are a false sense of security. The most common critical finding across our last 12 engagements: 9 out of 12 clients had automated backups they had never tested restoring. One had backups running for 8 months to a misconfigured destination — zero usable backups existed.
• Connection pool configuration. Most ORM defaults create a new connection per query. Under 5x normal traffic, this hits database connection limits and produces errors that look like application failures. We've seen this exact scenario bring down services that had no other issues.
• Read replica for analytical queries. Background jobs, dashboard queries, and analytics that run against the primary database compete with transaction traffic. A 30-second analytical query can degrade API response times by 2–3x.
• Failover time. For managed databases (RDS, Cloud SQL), how long does a failover take, and does the application reconnect automatically? We run a simulated failover in staging on every engagement — the results are often surprising.

4. On-call and incident response

Having PagerDuty is not the same as having incident response. We assess:

• Who gets paged? Is there a defined primary + backup? Or is the answer "whoever sees the Slack message first"?
• What's the escalation path? If the primary on-call can't resolve an incident in 20 minutes, who do they call? Is that person reachable at 2am?
• Do runbooks exist and are they current? A runbook written 14 months ago for an architecture that's since changed is worse than no runbook — it sends the on-call engineer down the wrong path under pressure.
• What's the postmortem process? Teams that learn from incidents systematically have dramatically fewer repeat incidents. Teams that "move on" after fixing the immediate problem see the same failure modes recur every 6–12 months.

We look at the last 6 months of incidents and ask: what percentage had a runbook? What percentage had a postmortem? What percentage recurred? The answers tell you more about operational maturity than any tool audit.

5. Secrets and access

This category overlaps with security compliance but the focus is operational risk, not audit readiness. We check:

• Are there secrets in code repositories, environment files, or Slack? (A Gitleaks scan on the main branch is usually part of this step.)
• Do former employees, contractors, or vendors still have production access? Access is almost never revoked on departure without a deliberate offboarding process — and most teams don't have one.
• Are service accounts scoped to least privilege? A CI pipeline that has AdministratorAccess because someone wanted to "avoid permission errors" is a single compromised token away from a catastrophic breach.
• Is there a secrets rotation policy? Credentials that haven't been rotated in 18+ months are typically credentials that will never be rotated — until something forces it.

6. Capacity and load

The last category is the one most teams have never tested at all:

• What's the breaking point? If you don't know how much traffic your system handles before degrading, you find out the hard way during a traffic spike or viral moment.
• Is autoscaling configured and tested? "We have horizontal pod autoscaling" is not the same as "we've verified that HPA triggers in time to handle a 5x traffic ramp." HPA has scale-up latency — the 60-second default metric evaluation window plus pod startup time means you can have 90 seconds of degraded service before new pods absorb load.
• Are there cost controls? A traffic spike shouldn't become a surprise bill. CloudWatch billing alarms, AWS Budget alerts, or equivalent controls catch runaway costs before they compound.
• Can the database handle 3x load? The API layer scales horizontally with new pods. The database doesn't. Connection pool limits, lock contention, and index degradation under high write load are the failure modes that kill systems that "scale fine" under normal traffic.

What the findings typically look like

Across our last 12 engagements, the average production readiness review found:

• 3 critical findings — issues that would cause data loss, extended downtime, or complete inability to recover in an incident
• 8 high-severity findings — gaps that meaningfully increase incident risk or recovery time
• 12 medium findings — missing monitoring, inefficient configurations, documentation gaps

The most common critical finding by far: no tested backup restore process (9 out of 12 clients). The second most common: no defined escalation path for on-call (production incidents with no clear human responsible after the first responder). Third: service accounts with excessive permissions that create catastrophic-failure blast radius.

"We thought we were in good shape. The review found that our database backups hadn't worked in four months due to a misconfigured IAM role. Nobody noticed because nobody had ever tried to restore one. That finding alone justified the entire engagement."

What the review doesn't tell you

A production readiness review doesn't certify that your system is ready for production. It tells you what risks you're carrying and which ones you're accepting knowingly versus unknowingly. The goal is the latter: knowing exactly where the gaps are so you can make informed decisions about which to fix this sprint and which to live with for now.

Some findings are worth fixing immediately (untested backup restores, active credentials held by former employees). Others are worth scheduling (autoscaling testing, runbook updates). A few are genuinely acceptable risks given your stage and traffic profile. The review gives you the information to make those calls — it doesn't make them for you.