Our CI Went From 22 Minutes to 6. Here's the Boring Part.

There is no shortage of articles about Kubernetes. Most of them cover the happy path. This one is about what happened after the happy path ended.

Most of the complexity was accidental. Over the years we'd layered three different approaches to blue-green deploys on top of each other, and each one made sense at the time.

What was going wrong

When we finally measured, the numbers were worse than anyone guessed. Around 42% of the time spent on a typical change went into working around blue-green deploys rather than building anything.

We had good intentions. We even had a document describing how blue-green deploys should work. The trouble was that the document and the code had quietly gone their separate ways.

A simpler approach

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what Kubernetes already gives you out of the box.

Here's a trimmed-down version of what we ended up with:

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ hashFiles('package-lock.json') }}
    restore-keys: npm-

It's tempting to generalize this into a library. Resist that urge until you've needed it in at least three places.

A few things that helped along the way:

Make the default the safe choice, not just the documented one.
Delete dead code aggressively; it's the cheapest refactor there is.
Measure before and after, even if it's just a rough number in a spreadsheet.

Trade-offs

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Prometheus for everything. That's a fair question with a boring answer: we tried, and it made blue-green deploys harder to reason about.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

Wrapping up

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.