Repository navigation
perf: dind image cache on webhook jobs; bound stalled cargo git fetches - #214
Merged
Merged
Conversation
The webhook/poll path built runtime.CreateConfig without Provider or Repo, so dind.CacheNamespace returned "" and the per-repo image cache was off for every job that path started — which on Linux nodes is all of them (only the WSL/macOS dispatch path set the fields). Every `docker pull` in a job went to the registry: ducks pulled mysql:8.0 78 times in three days, 7-45 s each. Pass the event's provider name and repo, as dispatch.go already does.
Git dependencies bypass the crate cache and are fetched by libgit2, which has no working stall detection once TLS is up. ephpm job 119658796412 (ducks, 2026-10-22) sat 13 minutes and then 5.5 more on a silent github.com connection fetching the turso git dep before libgit2 gave up with "SSL error: syscall failure" — 19 minutes of a 25 minute job. Set net.git-fetch-with-cli in both the caching and the inert container config, and export GIT_HTTP_LOW_SPEED_LIMIT=1000 / GIT_HTTP_LOW_SPEED_TIME=30 so git abandons a transfer that moves under 1 KB/s for 30 s; Cargo's net.retry then reconnects. Both are skipped for Windows containers, whose images ship no git binary (Cargo fails outright if it cannot run git).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes from a fleet performance review of 7 days of ephpm CI logs.
1. dind image cache never engaged for webhook/poll-dispatched jobs (
pkg/scheduler/scheduler.go)The main dispatch path built
runtime.CreateConfigwithoutProvider/Repo, sodind.CacheNamespacereturned""and the per-repo image cache was skipped. Only the WSL/macOS dispatch path (dispatch.go) set them. On ducks,mysql:8.0was pulled 78 times in 3 days, costing 7–45 s per DB/E2E job. The fix passes the event's provider and repo. The scheduler's runtime is concrete, so this has no unit test. I'll confirm the cache namespace shows up on ducks after the rollout.2. Stalled git-dependency fetches ran for up to 19 minutes (
pkg/proxies/cargo)Git dependencies bypass the crate cache, and libgit2 doesn't bound a silent TLS connection. In ephpm job 119658796412, the
tursogit dep stalled for 13 min, then another 5.5 min, before failing withSSL error: syscall failure. Both container configs now setnet.git-fetch-with-cli = true. The proxy exportsGIT_HTTP_LOW_SPEED_LIMIT=1000andGIT_HTTP_LOW_SPEED_TIME=30, so git drops a stalled transfer after 30 s andnet.retryreconnects. Windows containers are excluded because their images ship no git.Verification:
go vetpasses on Windows and on Linux (GOOS=linux),go test ./pkg/scheduler ./pkg/proxies/cargopasses, and there's a newTestNetTOML_GitFetchBounded.