Fixing slow Docker builds on ephemeral CI runners

TLDR: --mount=type=cache makes RUN layers non-deterministic. On ephemeral runners the mount is always empty, so BuildKit can’t match layers from registry cache. Removing cache mounts, switching to registry cache with dynamic fallback, gating exports to deploy branches, and restricting triggers dropped builds from ~27 min to ~2 min — and cut redundant builds entirely.

I’d been ignoring slow Docker builds on a project for a while — around 27 minutes per build on ephemeral GCP runners, most of that spent in uv sync downloading Python dependencies from scratch. Every single build. Even though BuildKit caching was configured.

The runners are ephemeral VMs — created for each job, then destroyed. No persistent BuildKit daemon between builds. The Dockerfiles used cache mounts for package managers:

RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen

And the workflow used GitHub Actions cache as the BuildKit backend:

cache-from: type=gha,scope=buildkit-${{ github.ref_name }}
cache-to: type=gha,scope=buildkit-${{ github.ref_name }},mode=max

This is the pattern you’ll find in most “optimise your Docker builds” guides. On ephemeral CI runners, it’s worse than useless.

Why cache mounts break registry caching Link to heading

--mount=type=cache attaches a persistent cache directory to a RUN step so package managers can reuse downloaded files. But the layer’s cache key includes the state of the mount, making it non-deterministic — the layer identity differs between machines.

On ephemeral runners the mount is always empty, so it provides zero benefit. The real damage is on the cache-matching side: when BuildKit pulls cache from a registry, it matches layers by content hash. A layer with --mount=type=cache can’t be matched this way because the mount state is part of its identity. So even with a warm registry cache, every RUN --mount=type=cache layer misses and runs from scratch.

The lockfile hadn’t changed, the dependencies hadn’t changed, but BuildKit couldn’t match the layer because the cache mount made it non-deterministic.

The fix: remove cache mounts Link to heading

# Before — non-deterministic
RUN --mount=type=cache,target=/root/.cache/uv \
    uv sync --frozen

# After — deterministic, cacheable by content hash
RUN uv sync --frozen

Without the cache mount, the layer is fully determined by its inputs. If uv.lock hasn’t changed, BuildKit matches from registry cache and skips it entirely. Same applies to npm, pnpm, etc.

I kept apt cache mounts though — system packages change rarely and those layers typically get cached at a higher level anyway.

Switch to registry cache Link to heading

I also switched from GHA cache to a container registry on the same cloud (in my case Google Artifact Registry):

Network locality. Pulling cache from the same cloud is significantly faster than GHA cache, which stores data in Azure blob storage — every read/write crosses cloud boundaries.
Size limits. GHA cache is limited to 10 GB per repo. With multiple branches and images, older entries get evicted and you’re back to cold builds.

cache-from: |
  type=registry,ref=europe-docker.pkg.dev/my-project/apps/myapp:buildcache-${{ steps.branch.outputs.name }}
  type=registry,ref=europe-docker.pkg.dev/my-project/apps/myapp:buildcache-${{ github.base_ref || github.event.repository.default_branch }}
cache-to: ...

github.base_ref is the PR’s target branch — so a PR targeting main pulls cache from main. On push events it’s empty, so the fallback github.event.repository.default_branch kicks in. Either way, new feature branches get a warm start rather than building cold.

You’ll also want to sanitise the branch name for use in cache tags (slashes aren’t valid):

- name: Sanitise branch name
  id: branch
  run: echo "name=${GITHUB_REF_NAME//\//-}" >> $GITHUB_OUTPUT

If your runners are on AWS, same idea with ECR.

Gate cache exports to deploy branches Link to heading

Writing cache from every feature branch pollutes the registry. I’d recommend only exporting on deploy branches. You can do this with an inline conditional — no extra step needed:

cache-to: >-
  ${{ github.event_name != 'pull_request'
    && format('type=registry,ref=europe-docker.pkg.dev/my-project/apps/myapp:buildcache-{0},mode=max',
              steps.branch.outputs.name)
    || '' }}

For PR builds, event_name is pull_request so the expression evaluates to an empty string — BuildKit skips the export. On push events to deploy branches, it writes the cache as normal. Feature branches still read from cache, they just don’t write back.

Restrict push triggers and add pull_request Link to heading

Most deploy workflows I’ve seen trigger on push to all branches. Every feature branch push kicks off a full build — even though you’re only actually deploying from main and dev. That’s a lot of wasted compute.

I’d recommend restricting push to deploy branches only, and adding a pull_request trigger so PRs still get build and test feedback:

on:
  push:
    branches:
      - main
      - dev
    paths-ignore:
      - "*.md"
      - "docs/**"
  pull_request:
    paths-ignore:
      - "*.md"
      - "docs/**"

One gotcha: PRs between deploy branches (e.g. dev → main) would trigger both the push event on dev and a pull_request event — running the build twice. You can skip these with a job-level condition:

build:
  if: >-
    github.event_name != 'pull_request' ||
    !contains(fromJSON('["main","dev"]'), github.head_ref)

This says: always run on push events, but for pull requests, skip if the source branch is itself a deploy branch. Those builds already ran when the source branch was pushed.

Combined with the cache gating above, the trigger setup looks like:

Push to main/dev: full build, writes cache, deploys
PR from feature branch: full build, reads cache only, no deploy
PR from dev → main: skipped entirely (already built on push)

Mirror base images Link to heading

Ephemeral runners hit Docker Hub’s anonymous pull rate limit fast. Each runner is a fresh VM with no auth tokens. I was seeing 429 Too Many Requests regularly.

The fix: mirror base images to your own registry and pass them as build args.

ARG base_image='python:3.12-slim'
FROM ${base_image} AS base

build-args: |
  base_image=europe-docker.pkg.dev/my-project/public/python:3.12-slim

The ARG default means local builds still pull from Docker Hub, which is fine for development.

Results Link to heading

	Before	After
Build time	~27 min	~2 min
Redundant builds	Every push	Deploy + PR only
Rate limit failures	Frequent	None
Cache exports	Every branch	Deploy branches only