3. Docker Layer Caching in Actions

Overview

You configured your build in CI with docker/build-push-action. It works. But it takes 5-8 minutes every time — because it installs every dependency from scratch. On your laptop, the second build takes seconds because Docker caches the layers on disk. In CI, every workflow run starts with a clean runner. There's no persistent disk. There's no local cache. Every build is like the first one.

For AI projects, this is especially painful. A pip install with torch, transformers, langchain, and their transitive dependencies can take 3-5 minutes. Multiplied by every push and every PR, those minutes add up to hours of waiting per month. Layer caching in CI solves this: you save the built layers in external storage and reuse them in later builds.

The real impact: A typical AI project with heavy dependencies goes from 6 minutes to 1.5 minutes per build with caching configured. If you do 20 builds a day, that's 90 minutes saved daily.

This capsule teaches you the two main cache backends for GitHub Actions — GitHub Cache and Registry Cache — with complete configuration, trade-offs, and before/after metrics.

Why CI has no cache by default

The runner is ephemeral

Every workflow run in GitHub Actions provisions a new runner — a clean virtual machine with a fresh operating system. When the run ends, the runner gets destroyed along with everything in it:

Run 1:
  New runner → Clean Docker daemon → Full build (6 min) → Runner destroyed

Run 2 (5 minutes later):
  New runner → Clean Docker daemon → Full build (6 min) → Runner destroyed

Run 3:
  New runner → Clean Docker daemon → Full build (6 min) → Runner destroyed

On your laptop, the Docker daemon persists between builds. The layers you built yesterday are still on disk today. That's why the second build is fast — Docker detects that nothing changed in requirements.txt and reuses the pip install layer.

In CI, that layer doesn't exist. Every build downloads the base image, copies requirements.txt, runs the full pip install, and copies the code. All from scratch. Every time.

The cost in AI projects

Typical dependencies of an AI project:

torch==2.2.0           ~800MB
transformers==4.38.0   ~50MB
langchain==0.1.0       ~30MB
openai==1.12.0         ~5MB
fastapi==0.109.0       ~5MB
uvicorn==0.27.0        ~2MB
+ transitive dependencies

Total: ~1-2 GB of packages

pip install time without a cache: 3-5 minutes
pip install time with a cache: 5-15 seconds (it only verifies, it doesn't download)

The two cache backends

Option 1: The GitHub Cache backend (type=gha)

It uses the same storage as actions/cache. The layers get saved in GitHub Actions' cache and restored in later builds.

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3

- name: Build with GitHub Cache
  uses: docker/build-push-action@v6
  with:
    context: .
    push: false
    tags: myapp:${{ github.sha }}
    cache-from: type=gha
    cache-to: type=gha,mode=max

How it works:

The first build:
  1. cache-from: type=gha → it looks for a cache → it doesn't exist → full build
  2. Build: download the base, pip install, copy src → 6 minutes
  3. cache-to: type=gha,mode=max → it saves ALL the layers in the GH Cache

The second build (only src/ changed):
  1. cache-from: type=gha → it restores the layers from the GH Cache
  2. Build: layers 1-4 from the cache, only rebuild layer 5 (COPY src) → 45 seconds
  3. cache-to: type=gha,mode=max → it updates the cache with the new layer

The parameters:

• cache-from: type=gha — It reads the cache from GitHub Actions' cache storage
• cache-to: type=gha,mode=max — It writes to the cache. mode=max saves all the intermediate layers (not just those of the final stage)

mode=min vs mode=max:

mode=min (default):
  It only caches the final image's layers
  → If you have multi-stage, it only caches the last stage
  → Smaller, but less useful

mode=max:
  It caches ALL the layers of ALL the stages
  → It includes intermediate stages (base, test, etc.)
  → Bigger, but it reuses more

For most projects, use mode=max. The extra cache is worth it for the time saved.

Option 2: The Registry Cache backend (type=registry)

It saves the layers as a special image in a container registry (GHCR, Docker Hub).

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v3

- name: Login to GHCR
  uses: docker/login-action@v3
  with:
    registry: ghcr.io
    username: ${{ github.actor }}
    password: ${{ secrets.GITHUB_TOKEN }}

- name: Build with Registry Cache
  uses: docker/build-push-action@v6
  with:
    context: .
    push: true
    tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
    cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
    cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max

How it works:

The first build:
  1. cache-from → it looks for ghcr.io/user/repo:buildcache → it doesn't exist → full build
  2. Full build → 6 minutes
  3. cache-to → it pushes the layers as ghcr.io/user/repo:buildcache

The second build:
  1. cache-from → it pulls ghcr.io/user/repo:buildcache → it restores the layers
  2. Partial build → 1 minute
  3. cache-to → it updates buildcache with the new layers

The cache gets stored as a cache image in the registry. It's not a runnable image — it's just the cache layers.

Comparison: GitHub Cache vs Registry Cache

Which one should you choose?

Does your project have < 5 GB of cacheable layers?
  → GitHub Cache (type=gha) — simpler, no extra config

Does your project have heavy dependencies (torch, cuda, transformers)?
  → Registry Cache (type=registry) — no 10 GB limit

Do you need to share the cache across multiple repos?
  → Registry Cache — accessible from any workflow

Do you want the simplest possible option?
  → GitHub Cache — two lines of configuration

The recommendation for most AI projects: Start with GitHub Cache (type=gha). If the cache fills up (the 10 GB limit per repo is shared with other Actions caches), switch to Registry Cache.

Complete configuration: GitHub Cache

The optimized workflow

# .github/workflows/docker.yml
name: Docker Build (GHA Cache)

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 20

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Build with GHA cache
        uses: docker/build-push-action@v6
        with:
          context: .
          push: false
          load: true
          tags: myapp:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Verify image
        run: |
          docker run --rm myapp:${{ github.sha }} python -c "
          from src.main import app
          print('Build OK')
          "

Verifying that the cache works

In the Actions logs, look for these lines:

The first build (no cache):

#6 [4/5] RUN pip install --no-cache-dir -r requirements.txt
#6 DONE 180.3s

Total build time: 195.2s

The second build (with cache):

#6 [4/5] RUN pip install --no-cache-dir -r requirements.txt
#6 CACHED

Total build time: 12.4s

The word CACHED confirms that the layer was restored from the cache. If you see the real time of the pip install, the cache didn't work — check the configuration.

Complete configuration: Registry Cache

A workflow with Registry Cache in GHCR

# .github/workflows/docker.yml
name: Docker Build (Registry Cache)

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read
      packages: write

    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build with Registry cache
        uses: docker/build-push-action@v6
        with:
          context: .
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
          cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max

The difference from GitHub Cache: you need a login to the registry and you specify a ref for the cache (buildcache). That ref is a special image that only contains layers — it isn't runnable.

Before/After: The impact of caching

An example project: an AI API with FastAPI + LangChain

Dependencies:
  langchain==0.1.0
  openai==1.12.0
  fastapi==0.109.0
  uvicorn==0.27.0
  pydantic==2.6.0
  tiktoken==0.6.0
  python-dotenv==1.0.0
  httpx==0.26.0

Measured times

The analysis

• 📋 The first build: It's always slow — there's no cache to restore
• 📋 Only the code changed: ~80% faster — the pip install layer gets reused
• 📋 requirements changed: ~25% faster — the base image's layer gets reused, but pip install re-runs
• 📋 No changes: ~90% faster — every layer from the cache

With heavy dependencies (torch, transformers)

For projects with torch, the cache turns 5-minute builds into 25-second builds. The difference is brutal.

Optimizing the Dockerfile for better caching

The order of the layers matters

Docker's cache works by layers: if one layer changes, every layer after it gets invalidated. Order your Dockerfile from least-changing to most-changing:

# ✅ CORRECT: dependencies before code
FROM python:3.12-slim
WORKDIR /app

# Layer 1: System dependencies (they almost never change)
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*

# Layer 2: Python dependencies (they change rarely)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Layer 3: The app's code (it changes frequently)
COPY src/ ./src/

CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

# ❌ INCORRECT: code before dependencies
FROM python:3.12-slim
WORKDIR /app

COPY . .     # Any change in the code invalidates ALL the following layers
RUN pip install --no-cache-dir -r requirements.txt   # It re-runs every time

CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

Separating requirements for better granularity

FROM python:3.12-slim
WORKDIR /app

# Layer 1: Core dependencies (they rarely change)
COPY requirements-core.txt .
RUN pip install --no-cache-dir -r requirements-core.txt

# Layer 2: AI dependencies (they change more often)
COPY requirements-ai.txt .
RUN pip install --no-cache-dir -r requirements-ai.txt

# Layer 3: The code
COPY src/ ./src/

CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

If you change an AI dependency (openai), only layer 2 re-runs. The core dependencies (fastapi, uvicorn) stay cached.

Cache across branches

GitHub Cache: branch restrictions

GitHub Cache has a scoping policy per branch:

The main branch: it can read and write its own cache
A feature branch: it can read main's cache + its own cache
                  it can only write to its own cache

A PR from feature → main:
  1. It looks for a cache on the feature branch → does it exist? → it uses that one
  2. If not → it looks for a cache on main → does it exist? → it uses that one
  3. If not → full build

This means that the first build on a new branch can restore main's cache, which is desirable — the dependencies are usually the same.

Registry Cache: no restrictions

Registry Cache has no branch restrictions — any build can read and write to the same cache ref:

cache-from: type=registry,ref=ghcr.io/user/repo:buildcache
cache-to: type=registry,ref=ghcr.io/user/repo:buildcache,mode=max

Every build, regardless of the branch, shares the same cache. This can be an advantage (maximum reuse) or a disadvantage (a broken build can contaminate the cache).

Troubleshooting

1. The cache never gets restored — always a full build

Symptom: The logs always show full build times, never CACHED.

Cause: cache-from doesn't find the cache — possibly the first build didn't save the cache correctly.

Solution: Verify that cache-to is configured correctly:

# Verify that you have BOTH: cache-from AND cache-to
cache-from: type=gha
cache-to: type=gha,mode=max    # mode=max is important

Check in GitHub → Settings → Actions → Cache that a cache entry exists for your workflow.

2. "cache export failed" in the logs

Symptom:

WARNING: cache export failed: error writing layer blob

Cause: GitHub Actions' cache is full (10 GB per repo, shared with every workflow).

Solution:

# Option 1: Clean the old caches
gh cache list
gh cache delete --all

# Option 2: Switch to Registry Cache (no limit)

Or in the workflow:

- name: Clean old caches
  run: |
    gh cache list --json key --jq '.[].key' | \
      grep "buildkit" | \
      xargs -I {} gh cache delete {}
  env:
    GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

3. The cache restores but the build is still slow

Symptom: The logs show importing cache manifest but the pip install runs in full.

Cause: You changed requirements.txt — the pip install layer got invalidated correctly.

Solution: This is correct behavior. If the dependencies change, the pip install re-runs. The cache only helps when the dependencies don't change. Verify that the COPY order in your Dockerfile is: requirements.txt before the source code.

4. "unsupported cache type" error

Symptom:

ERROR: unsupported cache type: gha

Cause: Buildx isn't configured. The classic builder doesn't support type=gha.

Solution: Make sure you have docker/setup-buildx-action before the build:

- name: Set up Docker Buildx      # THIS STEP IS NECESSARY
  uses: docker/setup-buildx-action@v3

- name: Build with cache
  uses: docker/build-push-action@v6
  with:
    context: .
    cache-from: type=gha
    cache-to: type=gha,mode=max

Exercises

Exercise 1: Add GHA cache to an existing build

You have this workflow with no cache. Add GitHub Cache to it:

name: Docker
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: false
          tags: myapp:latest

name: Docker
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - uses: docker/build-push-action@v6
        with:
          context: .
          push: false
          tags: myapp:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

Exercise 2: Configure Registry Cache with GHCR

Convert the previous exercise's workflow to use Registry Cache instead of GitHub Cache. The repo is github.com/user/ai-api.

name: Docker
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v4

      - uses: docker/setup-buildx-action@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - uses: docker/build-push-action@v6
        with:
          context: .
          push: ${{ github.event_name == 'push' }}
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache
          cache-to: type=registry,ref=ghcr.io/${{ github.repository }}:buildcache,mode=max

Exercise 3: Optimize the Dockerfile for caching

This Dockerfile has a bad layer order. Rewrite it to maximize the cache hit:

FROM python:3.12-slim
WORKDIR /app
COPY . .
RUN pip install --no-cache-dir -r requirements.txt
RUN apt-get update && apt-get install -y curl
EXPOSE 8000
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

FROM python:3.12-slim
WORKDIR /app

# Layer 1: System dependencies (they change very rarely)
RUN apt-get update && \
    apt-get install -y --no-install-recommends curl && \
    rm -rf /var/lib/apt/lists/*

# Layer 2: Python dependencies (they change occasionally)
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Layer 3: The app's code (it changes frequently)
COPY src/ ./src/

EXPOSE 8000
CMD ["uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8000"]

Exercise 4: Compare build times with and without a cache

Create a workflow that builds the same image twice in the same job — once without a cache and once with one — and shows the times. Use date to measure.

name: Cache Benchmark
on: workflow_dispatch

jobs:
  benchmark:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v4
      - uses: docker/setup-buildx-action@v3

      - name: Build WITHOUT cache
        run: |
          echo "=== Build without cache ==="
          START=$(date +%s)
          docker buildx build \
            --no-cache \
            -t myapp:nocache \
            --load \
            .
          END=$(date +%s)
          echo "Build without cache: $((END - START)) seconds"

      - name: Build WITH GHA cache (cold)
        uses: docker/build-push-action@v6
        with:
          context: .
          load: true
          tags: myapp:cached-cold
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Build WITH GHA cache (warm)
        uses: docker/build-push-action@v6
        with:
          context: .
          load: true
          tags: myapp:cached-warm
          cache-from: type=gha
          cache-to: type=gha,mode=max

      - name: Compare image sizes
        run: |
          echo "=== Image sizes ==="
          docker images --format "table {{.Repository}}:{{.Tag}}\t{{.Size}}" | grep myapp

Summary

• ✅ CI has no cache by default — every build starts with a clean runner
• ✅ GitHub Cache (type=gha) — simple, zero extra config, a 10 GB limit per repo
• ✅ Registry Cache (type=registry) — no limit, shareable across repos, it requires a login
• ✅ mode=max caches every intermediate layer — always use it
• ✅ The layer order matters: system → dependencies → code (from least- to most-changing)
• ✅ The impact is real: 5+ minute builds drop to 1-2 minutes with a cache
• ✅ Start with GitHub Cache — if the 10 GB limit becomes a problem, switch to Registry Cache
• ✅ Verify the cache: look for CACHED in the build logs to confirm it works
• ✅ Optimize your Dockerfile: COPY requirements.txt before COPY src/ for maximum cache hits

Additional resources

1. Docker build cache docs — Docker's official cache documentation
2. docker/build-push-action — Cache — The official action's cache guide
3. GitHub Actions Cache limits — GitHub's cache limits and policies
4. BuildKit cache backends — Documentation of BuildKit's cache backends
5. Dockerfile best practices — Leverage build cache — Best practices for optimizing the cache
6. GitHub Container Registry — GHCR's docs for registry cache