A client called us after an incident that took down their API for 22 minutes on a Tuesday afternoon. They had Kubernetes. They had rolling updates configured. They believed they had zero-downtime deployments. What they actually had was a deploy that gracefully replaced pods while their database migration ran — and the old pods, still serving traffic during the rollout, were now talking to a schema that had been partially altered. The result was 500 errors for about a third of requests while both versions ran side-by-side.
Rolling updates are the default. They are not always the right tool. Here's a clear breakdown of what each strategy actually does, when it fails, and how to choose.
The three strategies
Rolling update
Kubernetes replaces old pods with new ones incrementally — a few at a time, waiting for new pods to pass health checks before removing more old ones. At any moment during the rollout, both old and new pod versions are running and serving traffic.
Cost: No extra infrastructure. No additional complexity. It's the Kubernetes default and it works for the majority of deploys.
When it fails: Whenever the new and old versions of your application are not compatible with each other. The most common case is a database schema change that the old code doesn't understand — a renamed column, a dropped nullable constraint, a new required field. During the rollout window, old pods hit a new schema and produce errors. The window is usually 2–5 minutes, but for high-traffic APIs, that's thousands of failed requests.
Use it when: Your deploy is purely application code — no schema changes, no API contract changes, no config changes that old pods can't tolerate. This covers maybe 70% of typical production deploys.
Blue-green deployment
You run two identical environments — blue (current production) and green (new version). You deploy the new version to green, run tests, and then switch all traffic from blue to green at once. Blue stays up for a few minutes as your instant rollback target, then gets decommissioned.
The traffic switch is typically a load balancer or DNS weight change. In Kubernetes, this is usually done by updating the Service selector to point at the new Deployment, or by using a tool like Argo Rollouts or Flux with a traffic-split mechanism.
Cost: 2x infrastructure during the transition (5–15 minutes). For large clusters, that's meaningful — $50–200 in extra compute depending on pod counts and instance types. Also more operational complexity: you need to manage two environments, and your database migration strategy has to account for both versions being potentially live.
When it excels: Major version changes, API contract changes, deployments where you need smoke testing on the new version before it receives real traffic. The ability to roll back by flipping a switch — not by re-deploying the previous version — is genuinely valuable when something goes wrong in the first five minutes after a release.
Use it when: The risk of the deploy justifies the cost. Major releases, schema-heavy changes, deployments that have historically needed rapid rollback.
Canary deployment
You send a small percentage of traffic — typically 1–10% — to the new version while the old version handles the rest. You watch error rates, latency, and business metrics. If everything looks good after a defined period (or a defined request count), you gradually increase traffic to the new version until it's handling 100%.
In Kubernetes, this is most cleanly implemented with Argo Rollouts or Flagger, which handle the traffic weighting and the automated analysis rules. The analysis step is what makes canaries powerful: you can define "roll forward only if p99 latency stays under 500ms and error rate stays under 0.1% for 10 minutes." The rollout either promotes automatically or pauses for human review.
Cost: Requires a service mesh or a traffic-weighting-capable ingress. Higher operational complexity. The analysis configuration is non-trivial to tune — too sensitive and you get false positives that halt good deploys; too lenient and the canary doesn't actually protect you.
When it excels: High-traffic services where even a 5-minute outage during a blue-green switch is unacceptable. Features you want to validate under real traffic before full rollout. Changes to performance-sensitive code paths where synthetic tests don't fully capture production behavior.
Use it when: You have the monitoring infrastructure to actually analyze the canary (you can't canary effectively if you don't have p99 latency and error rate metrics per version), and the traffic volume to get meaningful signal from 5% of traffic within a reasonable time window.
The database migration problem
Every discussion of zero-downtime deployments eventually hits the same wall: the database. Application code is stateless and easy to roll back. Database schemas are stateful and much harder.
The pattern that actually works for zero-downtime schema changes is called expand-contract (or sometimes the three-phase migration):
- Expand: Deploy a schema change that is backward-compatible — add the new column as nullable, add the new table, add the new index. The old code still works on the new schema. No downtime risk.
- Migrate: Deploy the new application code that writes to both old and new schema simultaneously. Backfill existing data to the new column/table. Both versions of the app work correctly during this phase.
- Contract: Once you're confident the old schema path is no longer needed, deploy a cleanup migration that removes the old column or enforces the constraint you couldn't add earlier.
This turns one risky deploy into three low-risk deploys. The third phase is often skipped under time pressure, which leads to databases accumulating dead columns and tables. We've audited production databases with 30–40% of their columns in this "zombie" state — added years ago as expand migrations, the contract step never came.
The constraint: this only works if your application code is written to tolerate partial migration states. That requires discipline at the application layer — not just DevOps tooling.
What we actually deploy per client
Across our current client engagements, the split looks roughly like this:
- ~65% rolling updates — code-only changes, feature additions, dependency updates, config changes where both versions are tolerant
- ~25% blue-green — major releases, schema-heavy deploys, post-incident changes where confidence is low and fast rollback matters
- ~10% canary — high-traffic, latency-sensitive services at Series B+ companies with mature observability stacks
The canary percentage is low not because it isn't valuable but because it requires investment to do correctly. A canary without good metrics is just a slow rolling update that makes you feel safer without actually giving you signal.
The tooling question
For most teams, the right starting point is Kubernetes rolling updates with a defined rollback procedure and discipline around schema migrations. That gets you most of the safety benefit with minimal complexity.
If you want blue-green or canary, Argo Rollouts is the most mature option in the Kubernetes ecosystem — it handles both strategies with a clean CRD, integrates with most ingress controllers, and has good analysis template support for canary metrics. We've deployed it across a dozen clients and it's the tool we reach for first when a team is ready to go beyond rolling updates.
What we don't recommend: rolling your own blue-green with manual DNS flips or traffic weight changes. It works until someone forgets a step at 11pm, at which point "manual" becomes "incident."
"We'd been doing 'zero-downtime deploys' for two years. After the schema incident, we finally understood we'd just been lucky. The expand-contract pattern felt like overhead until we ran our first migration with it — took the same time and we slept better."
The right question to ask
Before choosing a strategy, ask: "If this deploy fails in the first 5 minutes, how do we recover?" If the answer is "re-deploy the previous version and wait 8 minutes," rolling updates are fine. If the answer needs to be "flip a switch," you need blue-green. If the answer needs to be "we find out before it affects more than 5% of users," you need canary.
The deploy strategy is downstream of the failure tolerance. Get clarity on that first, then pick the tooling.