The AWS invoice arrived every first of the month. January: $61K. February: $74K. March: $82K. Nobody on the team could explain the trend — the product wasn't growing that fast, no new features required significantly more infrastructure, and yet the bill kept climbing. The CTO called us in to find out where the money was going.
Eight weeks later, the bill was $31K. Here's every cut we made, how much each one saved, and the three systemic fixes that will keep the bill flat as they grow.
The audit
Before touching anything, we spent a week in AWS Cost Explorer and Trusted Advisor building a complete picture. We tagged every resource by team, environment, and service, then mapped each line item to a business justification. Several line items had no justification at all.
The breakdown, before any changes:
- EC2 / EKS worker nodes: $34K/month (41%)
- RDS: $18K/month (22%)
- Data transfer: $12K/month (15%)
- ElastiCache: $7K/month (9%)
- S3 + CloudFront: $5K/month (6%)
- Everything else: $6K/month (7%)
Three problems stood out immediately: the EC2 fleet was dramatically over-provisioned, there was no Savings Plan coverage despite stable baseline usage, and $12K/month in data transfer costs was almost entirely unnecessary.
Cut 1: Right-size the EC2 fleet — $14K/month saved
We pulled two weeks of CloudWatch utilization metrics for every instance type in the EKS node groups. The results were hard to argue with: average CPU utilization was 18%, average memory was 31%. The team had provisioned for peak-of-peak load and never revisited it.
We ran the workloads through AWS Compute Optimizer and identified the actual instance profiles needed. Then we did three things:
- Switched the baseline node group from
m5.2xlarge(8 vCPU, 32GB) tom7g.xlarge(4 vCPU, 16GB, Graviton3) — 40% cheaper per core, better performance-per-dollar - Set proper Kubernetes resource requests and limits on every workload — without these, the cluster scheduler doesn't know where to pack pods efficiently
- Enabled cluster autoscaler with Karpenter to provision nodes exactly sized to the pending workload instead of running a fixed fleet
We cut 40% of the EC2 bill without any performance regression. The team's p99 API latency actually improved slightly — Graviton3 handles their Go workloads faster per dollar than the previous Intel instances.
Cut 2: Savings Plans — $9K/month saved
The company had been running on-demand for three years. Their baseline EC2 usage was almost perfectly stable — the same 60% of their fleet ran 24/7 regardless of traffic. They were leaving an enormous discount on the table.
We purchased 1-year Compute Savings Plans (not instance-specific — Compute Plans cover any instance type, which matters because we were changing instance families). Coverage: 65% of baseline usage. The discount: 37% off on-demand rates for covered usage. We deliberately left 35% on-demand to retain flexibility for Karpenter to provision burst capacity.
One rule we follow: never buy more than 12 months of Savings Plan coverage at once. Architecture changes, acquisitions, and product pivots happen. 12 months is aggressive enough to capture meaningful savings; 3-year plans lock you into today's assumptions for too long.
Cut 3: Eliminate cross-AZ data transfer — $9K/month saved
This was the most surprising finding and the quickest fix. AWS charges $0.01/GB for data that crosses availability zones — in both directions. The application was making inter-service calls that traversed AZs on every request because the Kubernetes services had no AZ affinity configuration.
The fix was two changes in the Helm charts:
- Enable
topologySpreadConstraintsto spread pods evenly across AZs rather than packing them into one - Add
service.kubernetes.io/topology-aware-routing: Autoannotations so kube-proxy routes intra-AZ traffic locally when possible
We also found that the application was fetching a 50MB configuration blob from S3 on every cold start — called from pods across all three AZs, multiple times per minute during peak traffic. We cached it in ElastiCache with a 5-minute TTL. The S3 request volume dropped 94% and the cross-AZ transfer that was happening when an AZ-local pod called an AZ-distant ElastiCache endpoint disappeared.
Cut 4: Consolidate and right-size RDS — $8K/month saved
The account had six RDS instances. Two were clearly active — their production database and the read replica. The other four required investigation:
- One was a staging database running
db.r6g.2xlarge— the same size as production. Staging doesn't need $3K/month of database. - One was a reporting replica that hadn't received a connection in 47 days (CloudWatch
DatabaseConnectionsmetric confirmed it). - One was an old analytics database from a migration 18 months ago that the team thought had been decommissioned.
- One was a development database that each developer was connecting to individually — replaced by local Docker Compose Postgres.
We terminated three instances (after taking final snapshots), downsized staging to a db.t4g.medium, and optimized the production read replica's instance class based on actual query patterns. Total RDS savings: $8K/month.
Cut 5: ElastiCache and S3 tuning — $3K/month saved
The ElastiCache cluster was a three-node Redis setup running cache.r6g.large across three AZs for a cache that was 8% full at peak. We consolidated to two nodes with automatic failover and moved to cache.t4g.medium. For a non-critical cache (the application degrades gracefully on cache miss), this was a no-brainer.
On S3: we enabled Intelligent-Tiering on buckets older than 90 days and deleted 2.3TB of orphaned multipart uploads that had accumulated over years and were being charged at standard storage rates. One-time cleanup, $200/month ongoing savings — not large, but completely free money.
The results
- Monthly bill: $82K → $31K (62% reduction)
- Annualized savings: $612K
- EC2 CPU utilization: 18% → 52% (same workload, smaller fleet)
- p99 API latency: unchanged or slightly improved
- Incidents caused by cost changes: 0
"We knew we were overpaying but had no idea it was this much. The Savings Plan alone covered our DevOps Team retainer for the year. Everything else was pure savings."
The three systemic fixes that keep it this way
Point-in-time cost optimization is a one-time win. Without guardrails, bills creep back up within 6 months. We put three things in place before we closed the engagement:
- Cost allocation tags enforced at the IAM policy level. New resources without required tags are blocked by Service Control Policy. No tag, no resource.
- Weekly cost anomaly alerts via AWS Cost Anomaly Detection. If any service spends 20% more than the same week prior, the engineering channel gets an automatic Slack alert. Problems get caught within a week, not at month-end billing review.
- Quarterly right-sizing review. Part of our fractional retainer: every 90 days we pull Compute Optimizer recommendations and check whether the Savings Plan coverage level still matches actual usage. Cloud bills grow whenever nobody's watching. Watching is the job.
Key takeaways
Cloud bills grow by default. Every architectural decision, every new environment, every forgotten staging instance adds to the monthly total — and none of it reverses automatically. The companies with controlled cloud spend aren't the ones with the most sophisticated infrastructure; they're the ones with someone whose job it is to look at the bill every week and ask "why?"
Most of what we found here is universal: over-provisioned compute, no Savings Plans, cross-AZ transfer nobody noticed, and resources that outlived their purpose. We see this pattern across nearly every account we audit for the first time. The optimization window is almost always 40–65% — rarely less, occasionally more.