A charter dispatcher faced with a late departure does not guess. They work through a short list: crew, aircraft, slot, fuel, paperwork. Clusters running Kubernetes stall in comparable ways, and the practice some teams call KubernetesOps is largely a catalogue of these causes with their fixes. This article follows the diagnostic order described in the Ops Atlas entry, translating each symptom into an operations parallel familiar to anyone in the business-aviation world.
Symptom one: the drain never finishes
Before a worker machine is upgraded it is drained, which means its workloads are asked to leave politely through the eviction interface. If the drain hangs, the first suspect is a protection rule known as a disruption budget. The Atlas lists a specific mistake: a budget that permits zero interruptions. With that setting, not a single copy may ever be removed, so the machine can never be emptied. It is the equivalent of a dispatch rule that forbids any aircraft from leaving the gate while another is still being loaded.
The fix is to review every budget before the upgrade window. Where a service has two replicas and the budget demands both be up at all times, allow one to move. Scan the namespace for budgets in this state as a pre-flight check, and agree the new numbers with the service owner.
Symptom two: the drain returns errors
Read the error code. A refusal that arrives as a “too many requests” answer, status 429, simply means the budget would be violated right now, so the drain tool should retry. A server error, status 500, deserves more suspicion: the Atlas says it can indicate that two budgets select the same pod, which confuses the arithmetic. Clean up the overlap, and the retry succeeds.
A related setting governs pods that are already unhealthy, and it deserves a look in its own right. The Atlas recommends choosing the option that lets broken pods be evicted regardless of the budget, because otherwise a pod that is already failing can keep a node hostage. The default is more cautious, which is understandable but unhelpful in an upgrade.
Symptom three: the node empties but workloads fail elsewhere
Sometimes the drain succeeds and the damage appears afterwards. A common cause is a health probe that asks too much. The Atlas insists that the “is this process alive” probe must be answered locally. If it calls out to a database or another service, then when that dependency blinks, every copy of the application looks dead and gets restarted, and the restarts feed the load that caused the failure. A dispatcher would recognise the pattern: cancelling every flight because one airport reported a ground stop makes the disruption larger.
The solution is to keep the liveness check shallow and use a separate readiness check for dependencies. That way, a struggling downstream service reduces traffic to a copy rather than killing it.
Symptom four: new workloads cannot be admitted
Another stall appears when the control side calls out to admission webhooks, small services that approve or reject changes. By default, the Atlas notes, a webhook that cannot be reached causes the request to fail. If such a webhook lives on the very nodes being upgraded, or guards the system namespace and the networking component, the cluster can deadlock waiting for approval from a service that needs the cluster to run. The advice is to keep webhooks away from those critical namespaces and to review their failure policy.
Think of a ramp supervisor whose signature is needed to release the aircraft that carries the supervisor. The rule is sound until the one aircraft that holds the signature is in the hangar.
Symptom five: nothing is wrong, but there is nowhere to put the pods
Draining a machine means its work lands on the others, so they need headroom. Here the Atlas’s contract idea matters: placement is decided by declared requests. If requests have crept upward and little room remains, evicted pods sit unschedulable. Teams often find the opposite problem too: requests are large, actual usage small, and capacity is quietly wasted. The Atlas’s own arithmetic arrives at 35 percent actual use against what was asked for, a gap anyone in charter will find familiar, since empty seats on a repositioning flight tell the same tale; a primer on the mechanics of empty-leg jet trips shows how idle capacity can be put to work. In clusters the way to do that is rightsizing, starting with a recommender tool running in observation mode.
Symptom six: the upgrade went through, but something is subtly broken
Post-upgrade failures often trace back to an announced removal that nobody acted on. The Atlas cites a March 2023 incident at a well-known discussion site: partway through a version change, a role label disappeared from the control-plane machines, and the network components that picked their peers by that label lost contact across nodes. Restoring from an etcd backup consumed some five hours, and the company’s own review admitted that the restore steps were not up to date. The label’s removal had been announced in advance. The remedy is dull and effective: read the release notes for deletions, search your manifests for them and test in a copy first.
A dispatcher’s checklist for upgrade day
- Confirm the current and target versions, and that the target is exactly one minor step away.
- List disruption budgets and fix any that allow zero interruptions.
- Check the headroom: will the remaining machines hold the work of the drained one?
- Review webhooks that could block the system namespace.
- Take a snapshot of the cluster’s datastore and confirm a restore path exists.
- Run the upgrade in the documented order, draining before each node.
- Record how long each stage took, for the next window.
If the upgrade still stalls, resist forcing pods off the machine. The Atlas treats force-deletion as a last resort that bypasses the very budgets meant to keep services up, and says it should be agreed with the owner first.
Writing the delay report
Every aviation delay ends in a short report: what happened, the root cause, the fix and who owns it. Do the same after a stalled upgrade. Two paragraphs are plenty, provided they name the specific budget, probe, webhook or capacity gap involved. Attach the timeline, note which step took longest and add the new check to the pre-flight list. Months later, when someone new runs the next window, that page is worth more than any amount of tribal memory.
Prevention beats diagnosis
Dispatchers get better at fixing delays by analysing patterns, and so should cluster teams. Keep a log of each stall and its cause. After three or four upgrades you will see the same two or three culprits. Move those checks into a pre-flight routine, and the diagnostic list shrinks. For a related look at planning around spare capacity, there is also a guide to empty-leg flights for private travellers, a good reminder that unused capacity is information, not just waste.

