Alerts Should Wake You Up for a Reason

I've been working with Kubernetes for about 8 years now, and my opinions on secrets management have changed more than I expected.

When we finally measured, the numbers were worse than anyone guessed. Around 29% of the time spent on a typical change went into working around secrets management rather than building anything.

The problem

The first symptom was subtle. Nothing failed outright, but secrets management kept drifting in the wrong direction, and every fix seemed to make the next one harder.

Most of the complexity was accidental. Over the years we'd layered three different approaches to secrets management on top of each other, and each one made sense at the time.

How it works

The key insight was to treat secrets management as part of the interface rather than an implementation detail. Once we did that, most of the other decisions followed naturally.

Stripped of the project-specific bits, it looks like this:

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ hashFiles('package-lock.json') }}
    restore-keys: npm-

Nothing about this is clever, and that's the point. Anyone on the team can read it in a minute and know exactly what it does.

A few things that helped along the way:

Write the migration guide before writing the migration.
Pair on the first few changes so the knowledge doesn't live in one head.
Make the default the safe choice, not just the documented one.

What it cost us

We still haven't solved blue-green deploys. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

There are cases where this falls apart. If your team is very small or your traffic is very spiky, the extra moving parts may not pay for themselves.

What's next

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.