Kubernetes Is Not a Strategy

A colleague asked me last week why our team cares so much about container image size. I gave a rambling answer, so here is the longer, more organized version.

Most of the complexity was accidental. Over the years we'd layered three different approaches to container image size on top of each other, and each one made sense at the time.

Where we started

The first symptom was subtle. Nothing failed outright, but container image size kept drifting in the wrong direction, and every fix seemed to make the next one harder.

When we finally measured, the numbers were worse than anyone guessed. Around 26% of the time spent on a typical change went into working around container image size rather than building anything.

Trying something different

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what Kubernetes already gives you out of the box.

Here's a trimmed-down version of what we ended up with:

FROM golang:1.23 AS build
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -o /app ./cmd/server

FROM gcr.io/distroless/static
COPY --from=build /app /app
ENTRYPOINT ["/app"]

This alone let us delete a surprising amount of code elsewhere in the project.

A few things that helped along the way:

Pair on the first few changes so the knowledge doesn't live in one head.
Make the default the safe choice, not just the documented one.
Measure before and after, even if it's just a rough number in a spreadsheet.

Where this breaks down

We still haven't solved CI caching. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

What's next

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.