A Gentle Runbook for Your First On-Call Week

A colleague asked me last week why our team cares so much about alert fatigue. I gave a rambling answer, so here is the longer, more organized version.

Most of the complexity was accidental. Over the years we'd layered three different approaches to alert fatigue on top of each other, and each one made sense at the time.

The problem

The setup was ordinary: Kubernetes at the edges, Prometheus doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

We had good intentions. We even had a document describing how alert fatigue should work. The trouble was that the document and the code had quietly gone their separate ways.

A simpler approach

Rather than a big-bang rewrite, we migrated one module at a time over about 11 weeks, keeping both paths alive behind a flag.

Stripped of the project-specific bits, it looks like this:

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ hashFiles('package-lock.json') }}
    restore-keys: npm-

It's tempting to generalize this into a library. Resist that urge until you've needed it in at least three places.

A few things that helped along the way:

Write the migration guide before writing the migration.
Measure before and after, even if it's just a rough number in a spreadsheet.
Delete dead code aggressively; it's the cheapest refactor there is.

Caveats

We still haven't solved incident response. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Docker for everything. That's a fair question with a boring answer: we tried, and it made alert fatigue harder to reason about.

Wrapping up

I don't think there's a universal answer here, but I hope this gives you a few ideas to try on your own project. If you do, I'd love to hear how it goes.