Blue-Green Deploys on a Budget

For a long time I treated blue-green deploys as something to worry about later. Then a small change to our Ansible setup turned into a week of debugging, and "later" arrived all at once.

The setup was ordinary: Ansible at the edges, Terraform doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

Some background

The first symptom was subtle. Nothing failed outright, but blue-green deploys kept drifting in the wrong direction, and every fix seemed to make the next one harder.

When we finally measured, the numbers were worse than anyone guessed. Around 40% of the time spent on a typical change went into working around blue-green deploys rather than building anything.

A simpler approach

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

Here's a trimmed-down version of what we ended up with:

FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/server

FROM gcr.io/distroless/static
COPY --from=build /app /app
ENTRYPOINT ["/app"]

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

Keep the old path around until the new one has survived a real incident.
Pair on the first few changes so the knowledge doesn't live in one head.
Agree on what "done" means before the first pull request.

Trade-offs

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Prometheus for everything. That's a fair question with a boring answer: we tried, and it made blue-green deploys harder to reason about.

The biggest cost was social, not technical. Changing how people think about blue-green deploys takes much longer than changing the code.

Closing thoughts

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.