Last quarter we audited the GKE bill for a B2B SaaS company with 18 engineers. Their monthly Kubernetes spend was $22,400. After two weeks of right-sizing and configuration changes — no architectural rewrites, no migrations — it was $13,800. They were running a cluster that cost $8,600/month more than necessary, and nobody had noticed because the bill had crept up gradually over 18 months.
This is not unusual. In every infrastructure audit we run, Kubernetes cost waste is in the top three findings. Here's exactly where it comes from and how to fix it.
Finding #1: Resource requests set to production peak, not production average
Kubernetes schedules pods based on their resource requests, not their actual usage. If a pod requests 2 CPU cores but only uses 0.3 on average, 1.7 cores of node capacity sit idle — but you still pay for the node.
In this audit, we ran kubectl top pods --all-namespaces against two weeks of Datadog metrics and found:
- API pods: requesting 1,000m CPU, averaging 120m actual usage
- Worker pods: requesting 2,000m CPU, averaging 340m actual usage
- Background job pods: requesting 512Mi memory, averaging 80Mi actual usage
The team had set resource requests during a load test peak and never revisited them. The fix: set requests at roughly the 95th percentile of actual usage, not the maximum. For these pods that meant dropping from 1,000m to 200m CPU requests on the API tier alone — enough to shrink the node pool by two n2-standard-4 nodes, saving $840/month on that tier.
Tooling tip: Vertical Pod Autoscaler (VPA) in recommendation mode generates suggested request values based on actual historical usage without changing anything. Run it for two weeks and read the kubectl describe vpa output before touching any values manually.
Finding #2: Three clusters where one would do
This client had separate GKE clusters for development, staging, and production — each with its own control plane, node pools, and minimum node count. The development cluster was running 24/7 even though it was only used during business hours in one timezone.
Kubernetes namespaces handle multi-environment isolation for development and staging. You need separate clusters when you have true blast-radius requirements (production vs. everything else), compliance mandates, or drastically different traffic profiles. You do not need three clusters because "we've always done it this way."
We consolidated dev and staging into a single cluster with namespace isolation. Development got a namespace-scoped RBAC role that prevented access to staging. Resource quotas per namespace kept dev workloads from crowding out staging. The dev cluster was decommissioned. Savings: $2,100/month.
Finding #3: The dev cluster ran nights and weekends
Even before consolidation, we set up node pool autoscaling on the development cluster with a scheduled scale-down at 8 PM and scale-up at 8 AM local time. On weekends, the cluster scaled to zero non-essential nodes. This alone saved $610/month — and it took 20 minutes to configure. It's almost always the fastest win in any Kubernetes cost audit.
GKE, EKS, and AKS all support scheduled node pool scaling. There is no reason a development cluster should run at full capacity on a Sunday morning.
Finding #4: Spot instances not used for tolerant workloads
Batch jobs, background workers, asynchronous processors, and CI/CD runners are all good candidates for spot (preemptible) instances — they can be interrupted and rescheduled without user impact. Spot instances typically cost 60–80% less than on-demand.
This client was running all workloads on standard on-demand nodes. We created a separate spot node pool and added tolerations to:
- The email/notification worker (handles retries natively — preemption is fine)
- The report generation job (runs in off-peak hours, restarts cleanly)
- GitLab CI runners (a job that gets interrupted just restarts)
User-facing API and web workloads stayed on on-demand. Spot saved an additional $1,900/month. The total preemption rate over 30 days: 3 interruptions, all handled transparently.
Finding #5: Forgotten load balancers
Every Kubernetes Service of type LoadBalancer provisions a cloud load balancer. At $0.025/hour on GCP, each one costs ~$18/month minimum. This client had 14 load balancers. Four of them had no active traffic — they were left over from decommissioned services, test environments, and a demo environment that was spun up for a conference six months ago.
Deleting four idle load balancers saved $72/month — not huge, but it's the kind of waste that compounds. We also identified three services that could share a single load balancer using Kubernetes Ingress instead of individual services. Net result: 7 load balancers instead of 14, saving $126/month.
Finding #6: No cost attribution by team or service
The biggest meta-problem: nobody knew which workloads were responsible for which costs. Without namespace-level cost attribution, there's no feedback loop. Teams can't be accountable for costs they can't see.
We deployed Kubecost in read-only mode and set up a monthly cost report per namespace sent to each team lead. Within three months, two teams had independently right-sized their own workloads without being asked — because they could see their numbers.
The total
- Right-sized resource requests across 3 tiers: −$840/month
- Cluster consolidation (dev + staging): −$2,100/month
- Scheduled scale-down for dev: −$610/month
- Spot instances for tolerant workloads: −$1,900/month
- Orphaned load balancer cleanup: −$126/month
- Miscellaneous (oversized PVCs, unused IPs): −$1,024/month
Total monthly savings: $6,600. Annual: $79,200. The two-week engagement cost less than one month of the waste it eliminated.
"We'd been watching the bill go up every month and assuming it was growth. It wasn't growth — we were just paying for things we'd forgotten about."
Where to start
If you run Kubernetes and haven't done a cost audit in the last 6 months, start here: run kubectl top nodes and compare actual node utilization against what you're paying for. If your nodes are consistently below 40% CPU utilization, you have a right-sizing problem. That single check takes 30 seconds and will tell you whether the rest of this audit is worth running.
The underlying issue is almost never recklessness — it's that infrastructure costs accumulate invisibly while the team's attention is elsewhere. Cost visibility tooling and a quarterly review process are what prevent the next 18 months of drift.