Saturday, 09:41. The first charter quotes of the day had just gone out when the booking screen on Meridian Air Charter’s site stopped responding — a failure the DevOps diagnosis that followed would trace back months, to decisions nobody remembered making. Customers saw a spinner. Sales saw nothing at all. The engineer on duty restarted the application server, which brought the site back for eleven minutes before it froze again. By the time the system stabilised for good, a peak-season Saturday had been badly dented — and nobody could yet say why. What follows is the diagnosis this fictional operator worked through, because the pattern behind it is anything but fictional, and the fixes turned out to be textbook DevOps work.
Symptom or Disease?
The obvious read was technical: the server misbehaved, restart it. But a symptom that returns after every restart is the system telling you something structural is wrong. When the team reconvened on Monday, the question on the whiteboard was deliberately blunt: why did a routine change take down the whole booking flow, and why did recovering from it take most of the day? Those are not engineering questions. They are delivery-culture questions, which is to say DevOps questions, and the answers were uncomfortable for everyone in the room, starting with the person who had signed off the release. That discomfort turned out to be useful. Incidents that end in a restarted server teach nothing; incidents that end in an honest whiteboard teach the quarter.
The Diagnosis in Three Questions
Condensed for the leadership meeting, the inquiry came down to three questions worth stealing. How big was the change? If the honest answer is measured in weeks of work rather than days, blast radius was never under control. Had the undo been performed by a human being this year? A runbook nobody has executed is a document, not a capability. And who was authorised to act at the moment of failure? If the answer is a meeting that was not in session, the outage lasted exactly as long as the queue. Meridian’s answers were six weeks, never, and nobody — which is why the fixes that followed addressed all three rather than the server that happened to be holding the spinner.
Root Cause One: Changes That Had Grown Too Big
The frozen release bundled six weeks of work — a new quote engine, a redesigned payment step, a customer-portal refresh — because the team worked on long-lived branches and merged everything at once. Nobody could say which of the three contributed the failure. Diagnosing a big change is like troubleshooting an aircraft that has three simultaneous faults: every signal is contaminated. The remedy was structural, not heroic — trunk-based development, with small changes merged into the mainline constantly, each one verifiable on its own. Within a month the team had a new rule of thumb: if a release note needs three sentences, split it.
Root Cause Two: A Rollback That Existed Only on Paper
Recovery was slow for a reason the team found almost embarrassing: nobody had ever actually performed the documented rollback. The runbook referenced a database state that no longer matched production, and the person who knew the workaround was on leave. Delivery research has quantified this exact gap: one team, chasing a payment error with an undo they had never drilled, spent 190 minutes on the recovery; a practised crew does the same job in twenty. Meridian’s morning sat firmly on the wrong side of that comparison. The fix was to drill — undo a real deployment, against a clock, monthly — and to write a rule beside the drill: no release may contain an irreversible migration unless the undo path has been tested against it.
Root Cause Three: A Queue Where a Decision Should Have Been
The freeze had been spotted within minutes by someone with a fix half-ready. It sat for two hours waiting for an approval, because the previous month’s minor scare had added a signature requirement. This is the reflex the delivery research keeps warning about: after every incident, organisations add approvals, and heavyweight change boards measurably correlate with worse performance, not better. Speed of recovery is a permission problem more often than a technical one. Meridian replaced its signature queue with a rule — the on-call engineer may act first, and every emergency action pages a second person automatically — which kept oversight while removing the queue.
The DevOps Treatment Plan
The fixes followed directly from the diagnosis, and none required new tooling budget.
- Small, frequent changes from short-lived branches, so any failure has a short suspect list.
- Feature flags on the quote engine and payment step, letting the team disable a misbehaving path without redeploying anything.
- A monthly rollback drill with the clock visible, plus the irreversible-migration rule.
- Deployment identifiers with named owners, so the 2 a.m. responder always knows what changed and who can help.
- Blameless reviews within five working days, asking what made the failure easy rather than who caused it.
Note what is absent from the list: no new platform, no reorganisation, no hiring plan. Every item is a change to how existing people touch existing systems — which is precisely why all five were in place within a month, and why the quarter’s only significant spend was the pizza at the drills. That fact was noted pointedly in the post-incident review, since the previous year’s modernisation proposal had been declined as too expensive. The most effective changes cost agreement, not capital.
What DevOps Improved, Honestly
The quarter that followed was not a straight line. Automated checks raised the team’s own standards before they raised its speed, and there was a stretch where output dipped — the J-curve that delivery research predicts, where early gains give way to a plateau that feels like regression before the climb past the old baseline. Meridian had agreed in advance to tolerate a defined dip, which is the only reason the programme survived its sixth week. By quarter’s end, the wait from “ready” to “live” had shortened from weeks to days, restores were being measured in minutes, and the Saturday crew had stopped treating releases as events. The quote engine shipped behind a flag in week nine, took forty percent of traffic on day one, and took the rest without ceremony. The Saturday crew’s favourite detail was smaller still: the rollback drill now had a trophy, a small plastic propeller taped to whoever led the fastest undo. Trophies appear in no framework. Retention of the lesson does, and the trophy was doing that job.
The One-Sentence Diagnosis
If the whole incident must compress into a single line for the next leadership meeting, it is this: the platform did not fail because the code was bad; it failed because the changes were big, the undo was theoretical and the permission to act was slow. Organisations that recognise all three causes — and there is a full reference treatment of them in xOps School’s Ops Atlas write-up on DevOps — usually find that the software was never the deepest problem. Meridian’s team kept the whiteboard question on the wall for a year as a reminder. By the following peak season the booking screen was, in the ops manager’s words, aggressively uninteresting — the highest compliment this trade has to offer. For a wider look at how operators modernise around these practices, this overview of aviation, innovation and local business services makes a reasonable companion read.
