Why Your Kubernetes Pods Are Costing You Twice What They Should

We run cost audits on Kubernetes clusters as part of every new engagement. The number that surprises clients most isn't their total bill — it's how much of it is pure waste. The average overspend we find is 45–65% of actual compute spend. Not because the team made bad decisions, but because the defaults are wrong and nobody went back to fix them.

Here's what that looks like in practice and how to fix it — without rewriting your application or changing your cloud provider.

The root cause: resource requests are guesses that never get corrected

Kubernetes schedules pods based on resources.requests — how much CPU and memory a pod declares it needs. Nodes only accept pods their available capacity can satisfy. The problem is that developers set requests once, at the time they write the Deployment manifest, and then never revisit them.

The typical pattern we see:

• Developer sets cpu: "500m" and memory: "512Mi" because it seemed reasonable
• The pod actually averages 40m CPU and 180Mi memory at peak
• Kubernetes schedules the pod as if it needs 500m CPU, reserving that capacity on a node
• The node is "full" at 60% actual utilization — the rest is reserved but unused
• The Cluster Autoscaler adds another node because existing nodes appear full
• You pay for 60% more compute than your workload requires

Multiply this across 40 pods and three environments, and you understand where the bill comes from.

What a real audit reveals

In a recent engagement — a Series B SaaS company running 12 services on GKE — here's what we found before touching anything:

• Average CPU utilization across the cluster: 22% — with requests set such that the scheduler considered nodes 78% full
• Three services with memory limits set to 4Gi — all three averaged under 400Mi actual usage
• Staging environment running 24/7 at the same node size as production, with one-tenth the traffic
• No Horizontal Pod Autoscaler (HPA) on any service — every service ran at fixed replicas regardless of traffic
• Persistent volumes (PVCs) for four deleted services still attached and billing at $0.10/GB/month — 600GB total, unused
• One NAT Gateway per availability zone instead of one shared — a $90/month difference for a cluster this size

Total identified waste before any optimization: $4,200/month on a $7,800/month total compute bill. 54%.

The fix, in order of impact

1. Right-size resource requests with VPA in recommendation mode

The Vertical Pod Autoscaler (VPA) has a updateMode: "Off" setting that collects CPU and memory usage data over time and generates recommendations — without automatically changing anything. Run it for 7 days across all your workloads, then review the recommendations and update your Deployment manifests accordingly.

This alone typically reduces your request totals by 40–60%, which means more pods fit on fewer nodes. We do this before touching cluster size so we're measuring real utilization, not theoretical.

The one trap to avoid: don't use VPA in auto-update mode on production stateful workloads — it will evict pods to apply new limits, which is disruptive. Recommendation mode, then manual manifest updates, then rollout.

2. Add HPA to traffic-sensitive services

Fixed replica counts mean you're provisioning for peak traffic at all times. For a typical B2B SaaS product, peak load is roughly 3–5x baseline. If you're running 6 replicas to handle peak, you're running 6 replicas at 2am on a Saturday.

Horizontal Pod Autoscaler scales your replica count based on CPU or custom metrics (request rate, queue depth, etc.). The configuration is typically 10 lines of YAML. A simple CPU-based HPA on your API service — minReplicas: 2, maxReplicas: 10, targetCPUUtilizationPercentage: 60 — often reduces your average pod count by 40% while maintaining the same peak capacity.

3. Scale down non-production environments overnight

Staging and development environments don't need to run at 3am. A scheduled CronJob that patches your Deployments to replicas: 0 at 20:00 and back to normal at 07:00 on weekdays cuts non-production compute costs by roughly 55%. Add a cluster autoscaler that terminates nodes when they're empty, and those savings flow through to actual node costs, not just idle pod allocations.

We use a simple script in a Kubernetes CronJob that runs kubectl scale against a configurable list of namespaces. Three hours to implement, saves $800–2,000/month depending on your staging environment size.

4. Spot/preemptible instances for stateless workloads

Spot instances (AWS) and preemptible VMs (GCP) are 60–80% cheaper than on-demand. They can be reclaimed by the cloud provider with 2 minutes' notice, which makes them unsuitable for stateful workloads (databases, anything with local state). But for stateless services that restart cleanly — your API, your workers, your background jobs — spot is safe when combined with a pod disruption budget and multiple availability zones.

The pattern we use: a mixed node pool with 20% on-demand instances as a baseline (never preempted) and 80% spot. The Cluster Autoscaler and node affinity rules prefer spot but fall back to on-demand when spot capacity isn't available. In practice, spot reclamation events are rare enough that this configuration is invisible to users.

5. Audit and delete orphaned resources

Every cluster accumulates orphaned resources — PersistentVolumeClaims from deleted services, LoadBalancer Services that were replaced but not cleaned up, snapshots from manual debugging sessions. These are small individually but add up fast.

The command you probably haven't run recently:

kubectl get pvc --all-namespaces | grep -v Bound
kubectl get svc --all-namespaces | grep LoadBalancer

Compare the load balancers against services you know are active. Any LoadBalancer service without active traffic is costing you $15–20/month in cloud load balancer fees — plus the hourly charge per rule.

The results

For the GKE client mentioned above, the full optimization took six weeks — VPA analysis (week 1), manifest updates and rollouts (weeks 2–3), HPA implementation (week 3), staging scale-down automation (week 4), spot node pool migration (weeks 5–6), orphan cleanup (ongoing).

Before: $7,800/month. After: $3,100/month. $56,400 saved in the first year — without changing a line of application code, without migrating cloud providers, and without reducing the capacity available to the engineering team.

"We knew we were probably overpaying but assumed optimization was a big project. Six weeks later, we cut the bill in half and nothing broke. I wish we'd done it a year earlier."

Where to start

If you want to do this yourself, start with kubectl top nodes and kubectl top pods --all-namespaces. If actual utilization is below 30% of requested resources, you have significant headroom to reclaim. Install VPA in recommendation mode and let it run for a week. The data will tell you exactly where the waste is.

The patterns above apply regardless of whether you're on EKS, GKE, or AKS. The tools and specific commands vary slightly, but the underlying waste profile is identical — because the defaults in Kubernetes are identical.