Nobody sets out to build a slow, flaky, expensive test suite. It assembles itself, one urgent deadline at a time, the way a hangar accumulates clutter: every item arrived for a reason, and together they cost you the room. The way back out is equally unromantic — a sequence of small, boring steps, each useful on its own. That is genuinely good news, because TestOps (running your automated checks like a service rather than a ritual) rewards teams that start clumsy and consistent far more than teams that start grand and stall.
What TestOps Means Before the Steps Begin
One definition, so the steps have something to aim at. Treat the suite as a service you operate. The engineers waiting on results are its customers, and what it sells them is a verdict they can act on without convening a committee. Everything below exists to make that verdict quicker, cheaper and more honest — in that order of difficulty. For the full treatment behind the practice (definitions, metrics, research), the Ops Atlas TestOps explainer is the companion piece to this walkthrough, and teams already running a suite will find kindred operations writing in the wider aviation and business gateway on this site.
Step 1: Measure before you touch anything
Grab two numbers this week and resist the urge to fix things yet. First, the wait: how many minutes pass, typically, before a change gets its answer? Most CI systems will export timestamps if you ask nicely, and a week of data beats a month of opinions. Second, the lies: of the failures your team saw last month, how many were real defects? Ask the engineers who lived through them; their recollection is sharper than the logs. Write both on something visible. Both numbers tend to be embarrassing, which is precisely why they motivate. Every later argument — buy, build, prune, parallelise — gets settled by these figures, and a baseline taken after changes begin is worthless.
Step 2: Give the suite a memory
Results that live only in the CI log scroll into oblivion. Set up a store where every attempt is recorded against the test’s name, the exact code version and which run it belonged to. It sounds administrative; it is actually the foundation, and one afternoon of plumbing here saves a quarter of reconstructive archaeology later. Once history is queryable, you can answer the question that opens every serious meeting afterwards: how often does this particular check disagree with itself? Guesswork stops being the office’s operating system from that day forward.
Step 3: Put a budget on every stage
Checks belong in different places, and each place gets a spending limit:
- Next to the code, on the laptop: seconds.
- The pull-request stage: a cap near ten minutes — breaches count as pipeline bugs, not weather.
- After merge: the complete run, split across machines so it lands in roughly half an hour.
- Overnight: the marathon scenarios nobody should wait on during working hours.
- After deployment: a short smoke pass confirming reality held.
Adopt one standing rule from the start as well: whenever a late stage catches something, a cheaper check for it gets built into an earlier stage, so the same flaw never travels that far twice.
Step 4: Handle flakes in the open
When the same check passes and fails on identical code, name the situation in the open. Confirm it, then move it out of the deciding path with three things attached — an owner, a ticket and a removal date. What you must not do is close your eyes and re-run: that habit manufactures green screens while burying the defects that genuinely come and go. Give brand-new checks a probation period, too — run each one many times before its verdict is allowed to block anybody, because a good share of unreliable tests are born unreliable rather than worn down. A bench with no expiry quietly turns into the place tests go to be forgotten, which is how suites rot from the middle.
Step 5: Agree a target and publish it
Here is the step that turns effort into a commitment. Propose a share of true verdicts over a rolling window — something like ninety-eight per cent across a month is a common starting point. Do the arithmetic out loud: on twelve hundred runs, that budget permits twenty-four false alarms. Publish the actual number weekly next to the target. And while you are in the workflow file anyway, check the trap that catches nearly everyone: the step that hands down the final call has to execute even when the jobs it evaluates fall over, or branch protection will keep approving work on yesterday’s evidence.
Step 6: Prune, then keep pruning
Once verdicts are believed, count stops being comforting. Look once a quarter at which checks have caught a real problem lately and which merely burn minutes, and delete with enthusiasm. Delivery research keeps confirming what operators learn the hard way: a compact set of checks that everyone trusts outperforms a sprawling one that everyone routes around. Deletion is not regression; it is the rent staying paid.
Keeping TestOps Alive After Step Six
The sequence loops, of course. New code arrives, new checks join, drift resumes — so the measurement from step one becomes a standing habit rather than a one-time audit, and the cycle repeats at a calmer pitch each quarter. Total tooling required for the first pass: a stopwatch and a shared document. Teams usually reach for proper dashboards around step five, once a target exists to plot against. Until then, resist the temptation to leapfrog: the order is not sacred, but the second step is the one people skip and later regret, because every other improvement gets argued from memory instead of records. Six steps, repeated quarterly, beat any grand rebuild you could schedule — and unlike a rebuild, they leave behind a team that knows how to run the place.
A last thought on why the boring version wins. There is a listing on this site — a reduced-price offer under the In the Big City heading — that has outlived several redesigns, and it makes an odd kind of sense as a mascot for this whole exercise: whatever a page promises, sooner or later reality checks it. Suites are the same. A verdict that cannot survive inspection is not a verdict; it is marketing. Run the steps, publish the numbers, and let the marketing and the truth converge — quietly, one quarter at a time.
