feat: LAN GitHub Actions cache backed by a writable registry:2 #215

Why

Rust jobs on the fleet move their build cache (Swatinem/rust-cache → actions/cache) to and from GitHub's cache service over the WAN. Measured over 317 Rust jobs, 2026-10-15 to 10-22:

That's 26% of all time in those jobs: about 11 runner-hours a week, and 0.4–1.6 GB per entry over the home uplink. The cache is worth keeping: a hit roughly halves the job (Test linux-x64: 4–6 min with a hit, 9–23 min without; pdo_mysql: 8–16 min vs 29–32). But only 12 of 19 sampled jobs got a hit. GitHub's 10 GB-per-repo cap holds only 7 or 8 ePHPm-sized entries, so eviction probably costs misses too; that part is inferred, not measured.

Serving the cache from the LAN should cut restore and save to about 10–15 s each, and a larger cap should raise the hit rate.

Design

job step (actions/cache, rust-cache)
   │  GitHub cache API v2 (twirp JSON + Azure-blob-style upload/download)
   ▼
ephemerd cache shim on the node   http://<bridge-gw>:<port>/<per-job-key>/
   │  OCI distribution API (holds the only write credential)
   ▼
registry:2, writable, on the mayfly cache VM (second container next to the pull-through mirror)

1. Redirect the client: runner patch + env

• @actions/cache gets the service from ACTIONS_RESULTS_URL. The runner sets that per job from the job message, overwriting the container env.
• In runner.Extract(), after unpacking, byte-patch bin/Runner.Worker.dll, renaming the UTF-16 string ACTIONS_RESULTS_URL to a same-length name such as ACTIONS_RESULTS_ORL. This is the known technique from falcondev-oss/github-actions-cache-server. The runner then exports GitHub's URL under the new name, and ephemerd's ACTIONS_RESULTS_URL survives.
• The patch applies to every OS: the DLL is the same .NET assembly on Linux, Windows and macOS, and ephemerd supplies the runner for all of them, independent of the job image. It's reapplied for each runner version because each version extracts to a fresh directory.
• Verify the patch applied. If the string isn't found (a future runner renamed it), log loudly and turn the cache off for that runner version. Never fail jobs over it.

2. Cache shim (pkg/proxies/actionscache)

• CreateCacheEntry returns a signed upload URL pointing back at the shim.
• Upload: the Azure block protocol (PUT ?comp=block&blockid=…, then PUT ?comp=blocklist). Stream each block into an OCI blob upload: POST /v2/<repo>/blobs/uploads/, PATCH chunks, PUT ?digest=. Never buffer whole entries in memory.
• FinalizeCacheEntryUpload writes an OCI manifest (one layer) under actions-cache/<owner>/<repo>/<ref>. The tag is a hash of the cache key; the annotations hold the real key, version, size and created time.
• GetCacheEntryDownloadURL:
  • look for an exact key match first;
  • otherwise the newest entry matching a restore-key prefix, from the tag list plus annotations, cached in memory;
  • search scopes in order: the job's ref, then the default branch.
• Download: stream the blob from the registry with Range support, because the client downloads in parallel ranges.
• Everything else under the results URL (artifact v4 ArtifactService, etc.) is reverse-proxied unchanged to GitHub's real results host, with the client's Authorization header. Artifacts must keep working exactly as they do today.

3. Scope enforcement: ephemerd decides, not the job

• <per-job-key> is a random secret minted when ephemerd creates the job. It maps to (provider, repo, ref, default branch), all already known from the webhook event.
• Writes go only to the job's own ref. Reads come from the job's own ref plus the default branch, matching GitHub's rule that a PR can read main's cache but never write it.
• Keys are revoked when the job's environment is destroyed.
• Jobs never get registry credentials and can't reach the writable registry directly. Restrict it to ephemerd node IPs, and require auth even so.

4. Failure mode

• Shim or registry down: the shim returns errors the client treats as a cache miss, so the job builds cold. No job fails because of the cache.
• Health-checked the same way the other proxies are, with self-withdrawal: stop setting ACTIONS_RESULTS_URL if the shim can't reach the registry.

5. Eviction

• Per-repo size cap, LRU by last read. Track last-read time in the shim and delete old manifests through the registry API (REGISTRY_STORAGE_DELETE_ENABLED=true).
• Run the registry's garbage collection weekly from a mayfly timer on the cache VM. It needs the registry read-only for a few minutes; cache misses during that window are fine.

6. mayfly

• Add a second registry:2 service in non-proxy mode to the registrycache compose assets, on a separate port and storage path, with auth on.
• Grow the cache VM's disk from 32 GB to about 150–200 GB, moving it to local-lvm on ducks the way the runner VM was moved.
• Render the shim's endpoint and credentials into each node's ephemerd config.

Rollout

1. Build and test against a local registry:2:
   • unit tests for scope rules and the restore-key prefix search;
   • an integration test driving the real @actions/cache client;
   • an artifact pass-through test.
2. Ship off by default. Turn it on for ephpm/ephpm only.
3. Compare against the baseline above: restore p50 47 s, save p90 4 min, 12 of 19 jobs hitting. Then turn it on fleet-wide.

Prerequisites

• fix(dind): fail container start when it cannot be networked or published #213 and perf: dind image cache on webhook jobs; bound stalled cargo git fetches #214 rolled out.
• Runner bump to 2.338.0 (due before about 2026-11-05). Remove the stale 2.335.1 Windows zip from pkg/runner/embed in the same change: it's the same first-match trap that once shipped a deprecated runner to macOS.
• Confirmed 2026-10-22: the existing pull-through mirror on the cache VM is reachable from every Linux node, with about a 98% hit rate (2,889 blobs served vs 62 upstream fetches in 7 days) and 0 errors. It's a sound base for this.