Terraform Modules I Regret Writing

For a long time I treated CI caching as something to worry about later. Then a small change to our Terraform setup turned into a week of debugging, and "later" arrived all at once.

Most of the complexity was accidental. Over the years we'd layered three different approaches to CI caching on top of each other, and each one made sense at the time.

How we got here

The setup was ordinary: Terraform at the edges, Docker doing most of the heavy lifting, and a handful of scripts nobody wanted to own. It worked well until usage roughly doubled over a single quarter.

The first symptom was subtle. Nothing failed outright, but CI caching kept drifting in the wrong direction, and every fix seemed to make the next one harder.

A simpler approach

The fix turned out to be smaller than expected. Instead of adding another layer, we removed one and leaned on what Terraform already gives you out of the box.

Stripped of the project-specific bits, it looks like this:

- name: Cache dependencies
  uses: actions/cache@v4
  with:
    path: ~/.npm
    key: npm-${{ hashFiles('package-lock.json') }}
    restore-keys: npm-

It's tempting to generalize this into a library. Resist that urge until you've needed it in at least three places.

A few things that helped along the way:

• Measure before and after, even if it's just a rough number in a spreadsheet.
• Agree on what "done" means before the first pull request.
• Delete dead code aggressively; it's the cheapest refactor there is.
• Make the default the safe choice, not just the documented one.

Caveats

None of this is free. The new approach is a bit more verbose, and newcomers occasionally ask why we don't just use Docker for everything. That's a fair question with a boring answer: we tried, and it made CI caching harder to reason about.

We still haven't solved autoscaling. That's a post for another day, and honestly, I'm not sure we'll get it right the first time either.

What's next

If you take one thing away from this, let it be this: CI caching is a design decision, not a cleanup task. Make it early and make it on purpose.