[luthermonson]

Part of #215. This is the ephemerd half. The mayfly half (a second writable registry:2, a bigger cache-VM disk, rendering [actions_cache], and a GC timer) is a separate PR.

Off by default. With [actions_cache] absent, nothing about a job changes.

What it does

Serves GitHub Actions' cache service v2 (what actions/cache, Swatinem/rust-cache and setup-* caching talk to) from the node. Entries are stored in a writable OCI registry, so Rust jobs stop moving 0.4–1.6 GB archives over the WAN. The baseline in #215: 26% of Rust job time goes to cache transfer.

Design changes from the #215 write-up (found while building it)

1. Patching in runner.Extract() would not have reached most jobs. The fleet's default Linux image (runner-ci-linux, built on actions-runner) counts as an official image: the job runs the image's own runner at /home/runner, not the embedded one (isOfficialRunnerImage). So the patch is now per job:
   • in the container's own snapshot, for official images (patchSnapshotRunner);
   • in the job's overlay upper, for the mounted runner.
     The shared runner tree and the image are never modified, and the patch always matches the runner version that will execute.
2. A per-job secret in the URL path does not work. The toolkit builds twirp URLs with new URL('/twirp/…', baseUrl) (cacheTwirpClient.ts:66), which discards the path. Requests are attributed by source address instead. networking.JobForIP reuses the existing per-job c2c membership; the runner and every dind sibling join under the runner's name, and job containers lack CAP_NET_RAW. The signed upload and download URLs are used verbatim, so they additionally carry an unguessable token.
3. The scope must not come from the job a runner was dispatched for. JIT runners are fungible and registered at org level, so a runner started for a main push can be handed a fork PR. A runner is tracked at create (every lookup misses, every save is refused) and bound on in_progress, to the job GitHub says actually took it. The scope is resolved from GitHub's record of that job's run (CacheScoper / RunCacheFacts).
4. container: jobs. Container steps get only the step environment through docker exec, where a patched runner exports GitHub's address under the renamed variable. dind restores ACTIONS_RESULTS_URL on container create and exec. Without that, container steps would see no ACTIONS_RESULTS_URL at all, and upload-artifact would fail.

Scope policy (cacheScopeFor)

Run	Writes	Reads
push / schedule / dispatch / release / merge_group	refs/heads/<branch>	own branch, then default
pull_request (same repo)	refs/pull/N/merge	own PR, then default
pull_request from a fork	refs/pull/N/merge, or run/<id> when GitHub omits the PR	own, then default
everything else (pull_request_target, issue_comment, workflow_run, …)	— (read-only)	default

Nothing a PR saves is ever visible to a branch. A fork whose branch is named main cannot write refs/heads/main; there's a test for exactly that.

Failure is always a miss

• Registry down, job unbound, or entry missing: answered ok=false. The client treats that as a miss or a refused save and warns. The server never sends a 5xx, which the client would retry five times and then log as an error.
• Patch can't be applied: setting ACTIONS_RESULTS_URL is harmless on its own, because an unpatched runner overwrites it with GitHub's address. The job just uses GitHub's cache.
• New jobs: they only get the cache while the registry answers its health probe.
• Non-cache requests (artifact v4 twirp, …) under the results URL are relayed to GitHub unchanged, Authorization included. Only tracked jobs may use the relay, so it isn't an open relay on the bridge.

Storage

• Layout: one manifest per entry under actions-cache/<owner>/<repo>-<hash>/s<hash(ref)>, tagged k<hash(version, key)>, with the key, version, ref, created time and size in annotations. Each uploaded Azure block becomes its own blob, so concurrent out-of-order Put Blocks stream straight through without being buffered or reassembled. The manifest lists them in block-list order.
• Download: concatenates the layers, with Content-Length and single-range support.
• Eviction: per-repo LRU, by deleting manifests through the registry API. The catalog walk is rate-limited. Registry GC (blob reclamation) is left to the host's timer, in the mayfly PR.

Verified

• The runner string: downloaded runner 2.337.0. Runner.Worker.dll (linux-x64 tarball, and the win-x64 zip we embed) contains exactly one UTF-16LE ACTIONS_RESULTS_URL; Runner.Common.dll and Runner.Sdk.dll contain none. TestPatchResultsURL_RealRunner patched both real DLLs with the size unchanged. It's opt-in via EPHEMERD_RUNNER_WORKER_DLL, so re-run it on every runner bump.
• Protocol fields: all taken from the toolkit source (generated/results/api/v1/cache.ts, cache.twirp-client.ts, cachemetadata.ts, cachescope.ts), none guessed:
  • requests use useProtoFieldName (key, version, restore_keys, and size_bytes as a string);
  • responses use ok, signed_upload_url, entry_id, signed_download_url, matched_key and message.
• Upload path: uploadCacheArchiveSDK (BlockBlobClient.uploadFile, 128 MiB single-shot).
• Download path: downloadCacheHttpClient, used for every non-*.blob.core.windows.net host, with a Content-Length check.
• Tests:
  • an end-to-end Go flow against an in-process fake registry:2 that, like the real one, refuses manifests whose blobs are missing. It covers single-shot and reversed-order block uploads, ranges, restore-key prefix/newest/version rules, PR-reads-main-but-cannot-write-it, repo isolation, unbound jobs, signed-URL binding, conflicting re-bind, immutability, size mismatch and limit, registry down, LRU eviction, Forget, and artifact relay;
  • scope policy, JobForIP, dind env translation, scheduler tracking and binding, and config.
• Commands:
  • go build ./... passes;
  • go vet ./... passes on Windows, and with CGO_ENABLED=0 GOOS=linux;
  • go test passes for every touched package on Windows;
  • Linux test binaries pass under WSL Ubuntu: runner, networking, actionscache, scheduler, dind, runtime, config, providers/github.

Not verified yet

• A real @actions/cache client against this server. The Go tests drive the same wire format, but I haven't run the Node client or the Azure SDK against it. The first on-metal job is that test. Things to watch:
  • whether the Azure SDK accepts the Put Block/Put Block List response headers;
  • whether it treats an http://10.88.0.1:8089/... signed URL as IP-style.
• A patched runner starting a real job. The byte patch is the technique falcondev-oss/github-actions-cache-server uses in production, but I haven't run it here.
• Real registry:2's acceptance of the custom config media type. The fake enforces blob existence, not media types.
• Linux only; Windows and macOS hosts leave it off.

Turning it on (after the mayfly PR provisions the registry)

[actions_cache]
enabled  = true
registry = "http://192.168.16.224:5001"   # writable registry:2, NOT the pull-through mirror
username = "ephemerd"
password_env = "EPHEMERD_ACTIONS_CACHE_PASSWORD"
# port = 8089, max_size_gb_per_repo = 100, max_entry_gb = 10

Roll it out to one node first, and compare against #215's baseline: restore p50 47 s, save p90 4 min, 12 of 19 jobs hitting.