The Dashboard Nobody Looked At

There is no shortage of articles about Terraform. Most of them cover the happy path. This one is about what happened after the happy path ended.

The first symptom was subtle. Nothing failed outright, but incident response kept drifting in the wrong direction, and every fix seemed to make the next one harder.

Some background

We had good intentions. We even had a document describing how incident response should work. The trouble was that the document and the code had quietly gone their separate ways.

Most of the complexity was accidental. Over the years we'd layered three different approaches to incident response on top of each other, and each one made sense at the time.

The fix

We started by writing down the one rule we actually cared about, then deleted every bit of code that didn't serve it. It's not glamorous, but it works.

Here's a trimmed-down version of what we ended up with:

FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/server

FROM gcr.io/distroless/static
COPY --from=build /app /app
ENTRYPOINT ["/app"]

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Write the migration guide before writing the migration.
Pair on the first few changes so the knowledge doesn't live in one head.
Delete dead code aggressively; it's the cheapest refactor there is.

Where this breaks down

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use GitHub Actions for everything. That's a fair question with a boring answer: we tried, and it made incident response harder to reason about.

The biggest cost was social, not technical. Changing how people think about incident response takes much longer than changing the code.

Closing thoughts

That's it for now. Next time I'll dig into secrets management, which turned out to be closely related.