We Inherited a Broken Production Environment. Here's What We Found.

A Series A e-commerce company brought us in after their CTO quit. The outgoing hire had managed all of the infrastructure alone for two years. There was no documentation, no runbooks, and a handover that consisted of one Notion page with root credentials and a note that said "it works, mostly." Their new VP Engineering had been in the role for two weeks and needed to understand what she'd inherited before she could start improving it.

What we found over the following two weeks is more common than most engineering leaders want to believe. We're publishing an honest account of it — anonymized — because the pattern repeats across engagements, and the first step to fixing it is knowing what "broken" actually looks like up close.

The audit process

We run every new engagement through the same structured review: cloud accounts, deployment process, secrets management, monitoring, database hygiene, access control, and dependency inventory. We don't just read the docs — we trace the path of a production deploy from git commit to live traffic, watching every manual step and every assumption.

This one took ten days. Most audits take five. The extra time was because nothing was where we expected it to be.

What we found

Deployments: manual, undocumented, load-bearing

The deploy process was an SSH session into a single EC2 instance followed by running a shell script stored in the engineer's home directory — not in the repo. The script pulled from main, built the Docker image locally on the instance, stopped the running container, started the new one, and ran database migrations inline. The whole thing took 40 minutes and required the engineer to watch it run.

There was no rollback. If the new container failed to start, the fix was to re-SSH and run a different script (also not in the repo) that restored the previous Docker image from a tarball — if the tarball still existed. Two of the last five deploys had required manual database intervention to undo a failed migration. Neither was documented.

This was the entire deploy process for a company doing $800K/month in revenue.

Secrets: six different places, one of them Slack

We found production credentials in:

• A .env file in the application repo, committed to git — database password, Stripe keys, third-party API tokens
• A pinned message in the #engineering Slack channel ("backup of prod env vars just in case")
• A Notion page titled "Important: do not share" — no access controls, visible to the entire company
• Hardcoded in one service's Dockerfile as ENV directives
• In AWS SSM Parameter Store — but using the free tier, which stores values in plaintext, not encrypted
• In an exported CSV in a Google Drive folder called "creds"

Two former contractors still had active credentials. One had left the company eight months earlier. Their Stripe API key had been used to issue refunds as recently as six weeks ago — a transaction nobody on the current team could explain.

Database: one instance, no verified backups

The production RDS instance had automated snapshots enabled. That was the good news. The bad news: the last restore test was never recorded, which almost certainly means it was never done. We ran one. The restore completed, but the resulting database was missing three tables that existed in the production schema — the backup was from before a migration that had altered the schema structure.

There was no staging database. Developers tested against production with a set of "test accounts" created years ago. Those accounts had made real purchases.

Monitoring: three tools, none of them actionable

The team had CloudWatch, a New Relic trial that had lapsed six months ago but was still receiving some data, and a self-hosted Uptime Robot checking the homepage every 5 minutes. CloudWatch had 47 configured alarms — none of them had been tuned since the initial setup and all of them were alerting to an email address that bounced. The last alert that reached an engineer was from Uptime Robot, nine months ago, when the homepage went down for 23 minutes at 11pm because a deploy script had exited mid-run.

The engineering team had no idea what their p95 API response time was. They had never defined what "degraded" looked like before a full outage. Their first signal of problems was typically a customer email.

Access control: the root key did everything

The AWS account had one IAM user: the former CTO's personal account, with full AdministratorAccess. That key was the one in the Notion doc, the Slack message, and the .env file. There was no MFA on the root account. There were no IAM roles — the application accessed AWS services using the same root key hardcoded in the environment.

GitHub was better — at least it had two-factor authentication enabled. But there were no branch protection rules on main. Any engineer could push directly to production.

What we built over the following eight weeks

We've written elsewhere about the specifics of CI/CD pipelines and SOC 2 controls — this isn't the place to repeat that detail. The short version:

• Weeks 1–2: Rotate every credential, revoke all former contractor access, create scoped IAM roles, enable AWS root account MFA, lock main branch
• Weeks 3–4: Migrate secrets into AWS Secrets Manager with application-level fetching at runtime; add Gitleaks to CI to prevent future commits of secrets
• Weeks 4–5: Build a real staging environment — separate RDS instance, production data sanitized and restored weekly, identical application configuration
• Weeks 5–6: Replace the SSH deploy with a GitHub Actions pipeline: build → test → push image to ECR → ECS rolling deploy with health checks → automatic rollback on failure
• Weeks 6–7: Observability from scratch — Datadog agent on all services, structured JSON logging, p50/p95/p99 latency dashboards, error rate alerts with PagerDuty escalation
• Week 8: Backup verification automation — weekly RDS restore to a test instance, schema comparison against production, alert if the restore doesn't match

The outcome

Eight weeks after we started, the VP Engineering ran her first production deploy herself from the GitHub UI. It took 9 minutes. She could see the deploy progress in real time, watch the health checks pass, and confirm in Datadog that error rates hadn't changed. When I asked what it felt like, she said it was the first time she'd felt in control of the infrastructure since joining.

"The audit report alone was worth it — just having someone write down what was actually there. The rebuild was what let me sleep at night. We had no idea how exposed we were."

What this means for your team

None of what we found at this company was caused by negligence or incompetence. It was caused by one engineer moving fast alone for two years without a review process, a handover process, or a second pair of eyes. This pattern shows up in roughly 70% of the Series A companies we audit. The debt accumulates invisibly because the system works — until it doesn't.

The questions worth asking now, before a CTO departure forces the answer:

• Could a new engineer reproduce your deploy process from documentation alone?
• When did you last restore a production database backup and verify its contents?
• How many places do your production credentials live?
• If your on-call engineer is unreachable, does anyone else know how to run a deploy?

If any of these questions make you uncomfortable, you already know what the audit will find.