The Release That Almost Shipped: A DevSecOps Story

The Release That Almost Shipped: A DevSecOps Story

Meridian Charter’s software team is fictional, but every event in this story happened to real teams, on dates you can verify. Six engineers maintain the systems behind a small jet-charter operation: the scheduling board that pairs empty legs with passengers, the quoting engine behind charter pricing, the crew iPad app. Their pipeline ran on GitHub Actions, their secrets lived in repository settings, and their security plan fit in one sentence: “the cloud provider handles that”. This is the account of the fortnight in which that sentence stopped being acceptable, and DevSecOps — checks, traces and proof woven into delivery itself — stopped being a buzzword and became a checklist. The public record behind each incident is collected at the Ops Atlas’s DevSecOps entry.

Tuesday: the advisory nobody expected

The warning arrived dressed as a routine dependency badge. A dependency alert linked to the March 2025 compromise of tj-actions/changed-files, a GitHub Action so common that Meridian had three workflows using it. An attacker had reached the action’s code, and every secret that had flowed through it had to be considered exposed; CISA’s guidance was characteristically blunt — rotate everything. The team spent the evening doing arithmetic no spreadsheet enjoys: cloud keys, registry tokens, signing credentials, all of it potentially burned, none of it inventoried anywhere except in one engineer’s memory.

The rotation itself taught the second lesson of the week. Two credentials were easy: invalidate and reissue. Two more were tangled — the first instinct was to scrub the git history so the old values would stop appearing in search, which the team’s new advisor vetoed with a sentence worth framing: rotate first, rewrite history second. History rewriting without rotation just hides the same key in a quieter room. They rotated, then tidied, and the pattern stuck.

Thursday: the second shoe, already falling

While the post-mortem was still being written, the industry supplied a second data point. On 19 March 2026, the maintainers of two Trivy-related actions — trivy-action and setup-trivy — reported that their tags had been force-pushed to deliver a tampered Trivy build, v0.69.4. Containment came within the same evening, and the takeaway matched Meridian’s Tuesday exactly: a build component you trusted last week can be swapped under you this week, and the word “Immutable” on a package page stops nothing. Force-pushed tags are a known technique, not a novel one.

Anyone who has priced charter flying will recognize the structure — the factors behind private jet charter pricing include exactly one you cannot negotiate: repositioning flights that earn nothing. Compromised build tooling is the repositioning flight of software. It carries no payload, it was never the plan, and it must be flown before the revenue trip can happen. The only real defense is refusing to board unverified components in the first place.

The DevSecOps changes Meridian shipped by Friday

The rebuild followed the published playbooks rather than invention, and each item traced to a documented practice:

  • Every third-party action pinned to a full commit SHA. Tags point at whatever the publisher wants them to point at this morning.
  • Least-privilege job permissions, so a compromised step in the docs build cannot read production credentials.
  • Long-lived stored keys replaced by OIDC federation — the platform now mints short-lived identity on the spot.
  • Fork contributions stripped of secret access, and no untrusted code executed under pull_request_target.
  • Provenance and signing added to every build, with verification enforced at deploy time — GitHub’s artifact attestations reach SLSA Build Level 2, or Level 3 once reusable workflows are pinned.
  • An SBOM per release: the parts list whose absence had made Tuesday’s scoping so painful.

The gate policy came last and stayed narrow: refuse only fresh, severe, fixable findings held with high confidence; convert everything else to owned work with a ninety-day fuse. Nobody wanted to build the bypass culture that strict gates create. Alongside the gate went a calendar habit: any significant change earns three quarters of an hour with STRIDE and the manifesto’s four questions before its first build runs, with the decisions written where the next engineer will find them.

What DevSecOps felt like from the flight desk

Ops noticed nothing at first, which was the point. The scheduling board kept pairing empty legs the same way it always had — empty-leg jet flights work as margin recoverers, and no security change was going to interrupt that arithmetic. What changed was invisible until the next bad Tuesday, when the response would be a lookup instead of an excavation: which builds used the affected action, read from provenance records, not from memory. The XZ Utils episode of March 2024 had demonstrated the same value soberly — an inventory does not stop a backdoor, but it collapses the time to answer “where does this component run?” from days to minutes, which is what let responders act on the downgrade-to-5.4.6 advice quickly.

The DevSecOps ledger, one fortnight later

Meridian’s costs were two late evenings and a weekend of pinning. Its assets were new: a secret inventory, verified builds, a narrow and believed gate, and the habit of asking “how do we know?” about every component that touches production. The team kept two DORA findings pinned above the desk as a reminder that the human layer deserves as much engineering as the tooling — weak security practice comes packaged with burnout at roughly 1.4 times the usual odds, and culture is the stronger adoption predictor. A pipeline is a crew, not an autopilot. The release that almost shipped turned out to be the cheapest training Meridian ever bought, and it cost less than the repositioning flight it prevented.