We've done infrastructure audits for startups across FinTech, HealthTech, SaaS, and logistics. Different stacks, different team sizes, different cloud providers. The surface details vary. The underlying gaps are almost always the same five things.
This isn't a criticism of the teams we audit. These gaps are the natural result of engineers focused on shipping product — which is exactly what they should have been doing at the time. But at some point, usually when something breaks badly or an enterprise deal requires security documentation, the gaps become expensive. Here's what we find, how we find it, and what actually fixes it.
1. Backups that have never been tested
Almost every startup has backups configured. Almost none have tested a restore. The distinction matters more than anything else on this list.
In the last six months, we've found:
- A startup whose nightly RDS snapshots had been failing silently for 11 weeks — the CloudWatch alarm was misconfigured and routing to a dead email address
- A team whose PostgreSQL backups were being written to an S3 bucket in the same region as the primary database, making them useless in a regional outage
- A startup that had tested a restore exactly once, 18 months prior, and assumed it still worked
The fix isn't "configure backups better." It's a scheduled restore test. Every two weeks, we add a job that restores a recent backup to a temporary test instance, runs a database health check against it, and terminates the instance. The job takes 20 minutes and runs automatically. If it fails, an alert fires before you discover the gap at 2am during an actual incident.
What we add to every engagement: cross-region backup replication, a restore verification cron job, and an alert on backup job failure — not just on the absence of a backup.
2. Secrets in places they shouldn't be
We run git log --all --full-history -- '*.env' and a Gitleaks scan on every audit. It finds something almost every time. The most common locations where secrets are living when they shouldn't be:
- Committed
.envfiles in git history (even if removed from HEAD, they're still in the log) - Hardcoded in Dockerfiles — usually a database URL or third-party API key added "just for testing"
- In CI/CD environment variables that are visible to all project members with Developer access, not masked or restricted to protected branches
- In a shared Notion doc, Slack message, or team password manager that former employees still have access to
When we find a secret in git history, we can't just delete the file — the credential is compromised and needs to be rotated. That's the expensive part. The remediation on one recent audit involved rotating 23 separate credentials across 8 services, which took most of a day to coordinate without breaking anything in production.
After cleanup: AWS Secrets Manager or HashiCorp Vault as the single source of truth, Gitleaks added to the pre-commit hook and CI pipeline, and existing credentials rotated. The rotation is painful once. The automation prevents it from happening again.
3. No cost alerts — and a bill that's been growing for months
Cloud cost surprises are almost always visible in hindsight. The spike happened weeks ago; nobody noticed because there were no alerts, or because "DevOps handles the AWS bill" and nobody owns it clearly.
In a recent audit of a 25-engineer SaaS company, we found that their monthly AWS spend had grown from $8,000 to $31,000 over 14 months — a trend perfectly visible in Cost Explorer that nobody had looked at. The causes were mundane: a data processing job added in Q3 that ran 24/7 when it only needed to run for 4 hours daily, three NAT Gateways processing traffic that could have used VPC endpoints, and a staging environment with production-sized EC2 instances that ran on weekends when no developer was working.
None of this required a major re-architecture. It required someone to look at the bill.
What we add: AWS Budgets or GCP Budget Alerts with anomaly detection thresholds at 10% and 20% variance from the 30-day average. Tag enforcement for every resource so cost is attributable to a team or service. A monthly cost review in the shared calendar. Cost is not an infrastructure problem — it's a visibility problem.
4. On-call that's really just the CTO's phone
The on-call rotation at most early-stage startups is one of: the CTO, the most senior engineer, or whoever is unlucky enough to be paged. There's no runbook. The alert thresholds are either too sensitive (constant noise) or too loose (alerts fire after customers are already complaining). And there's no escalation path, so every alert goes to the same person whether it's a deployment blip or a data loss event.
We've onboarded to clients where the CTO had been woken up an average of 3.5 times per week for 6 months. They weren't sleeping. Their family had opinions about it. And the alerts that were waking them up were mostly recoverable — pod restarts, temporary queue depth spikes, certificate renewals — not the actual incidents that needed a CTO's attention.
The on-call fix has three parts:
- Alert triage: Categorize every existing alert as P1 (wake someone up), P2 (notify during business hours), or noise (fix the alert threshold). Most alerting configurations we inherit have 80% noise.
- Runbooks: For every P1 alert, write a runbook. Not a 20-page document — a checklist in Notion or Confluence with the three commands to run, the likely causes in priority order, and who to call if the runbook doesn't resolve it.
- Rotation and coverage: A single engineer as the permanent on-call is not a rotation — it's a sacrifice. Proper coverage means at least two people in the rotation with a defined escalation path. For clients who don't have the headcount for this, we provide the after-hours coverage as part of the engagement.
5. Infrastructure that only one person understands
This is the most invisible gap and the most dangerous. In every audit, we ask: "If the person who set up your Kubernetes cluster left tomorrow, how long would it take a new engineer to understand it well enough to make a safe change?" The answer is usually "weeks" or "we'd have to figure it out as we go."
The knowledge concentration problem takes two forms:
- Undocumented manual changes: Infrastructure that was set up by hand, with the decisions and the reasoning living only in one person's memory. When that person isn't available, the next engineer makes changes without understanding the constraints — and breaks something that wasn't obviously connected.
- Implicit infrastructure: Terraform or Helm charts that exist, but nobody on the team can explain why certain values were chosen, what the dependency order is, or what will break if a particular change is made.
The fix isn't a 200-page architecture document. It's a 2-page decision log for each major system — what it is, why it was built this way, what to check before changing it, and who to call if it breaks. A runbook is better than no runbook even if it's incomplete. A Terraform comment is better than none. The goal is the bus factor: the infrastructure should be understandable by someone who wasn't there when it was built.
Why the same five things?
These gaps appear so consistently because they're all low-urgency, high-importance work — the category that always loses to shipping features. Untested backups don't cause pain until they do. Secrets in git don't become a breach until they're discovered. Nobody notices the cost trend until it's embarrassing. On-call is painful but manageable until it isn't. And infrastructure documentation is everyone's lowest priority until the engineer who knew everything leaves.
We find these same five things in companies run by excellent engineers because excellent engineers prioritize correctly for their stage. The audit is useful not because the team didn't know — they usually did — but because it creates the forcing function to fix what's been on the backlog for six months.
"We knew the backups were probably fine. The audit confirmed they weren't. Worth every minute."