Skip to content

fix(ci): expose runner cargo and reclaim stale coverage images - #2514

Merged
seonghobae merged 1 commit into
mainfrom
seonghobae/coverage-cargo-path-docker-hygiene
Sep 29, 2026
Merged

seonghobae merged 1 commit into
mainfrom
seonghobae/coverage-cargo-path-docker-hygiene

Conversation

@seonghobae

Copy link
Copy Markdown
Contributor

Why

  1. Every Rust coverage gate on cwlab-s1-04 fails.
    • Error: Could not materialize base Rust dependencies: could not run trusted cargo vendor for base manifest Cargo.lock, crates/fast-mlsirm-py/Cargo.lock, fuzz/Cargo.lock: FileNotFoundError, seen on the fast-mlsirm#2157 dispatch, run 36529717504, job 109392516857.
    • Cause: the coordinator checked the guest. rustup's cargo is in ~/.cargo/bin, but actions-runner/.path doesn't include it.
    • Effect: a failed coverage gate means opencode-agent can publish only COMMENTED, not APPROVED, so no fast-mlsirm PR could get the non-author approval its ruleset needs.
  2. The guest disk refilled to 96% within hours of a cleanup. Per-run opencode-coverage-tools:<run>-<attempt> images (2.4–4.6 GB each) accumulate. fix(ci): reclaim per-run OpenCode coverage resources #2491 removes each run's own image at job end, but dispatches created before fix(ci): reclaim per-run OpenCode coverage resources #2491 run the older workflow and leave their images behind.

Change

  • New step Expose runner Rust toolchain (before measurement). If cargo is already on PATH (hosted), it does nothing. If ~/.cargo/bin/cargo exists, it appends that directory to GITHUB_PATH. Otherwise it emits ::warning::, since repositories without Rust don't need cargo.
  • materialize_base_rust_dependencies.py now reports FileNotFoundError as "cargo is not installed or not on PATH on this runner (self-hosted runners keep it in ~/.cargo/bin)".
  • New step Reclaim stale coverage images (before measurement). It removes only reference=opencode-coverage-tools images with until=2h, then dangling images (image prune -f), then build cache older than 24 h. It never runs -a or system prune, so pinned base images stay. A failure emits ::warning:: and doesn't block measurement. One OpenCode job runs per runner, so the current run's image, built after this step, is never touched.
  • REVIEW_DISPATCH_BLOB_SHA is recomputed for the changed workflow.

Tests

  • tests/test_opencode_coverage_runner_hygiene.py (new, 6 tests) runs each step's actual run: script with a fake docker and fake HOME:

    • step order;
    • exact reclaim commands, with no -a;
    • the warn-and-continue path;
    • ~/.cargo/bin gets appended;
    • cargo already on PATH leaves things unchanged;
    • missing cargo warns readably.

    6 failed before the change and 6 pass after.

  • test_missing_cargo_is_reported_as_a_runner_toolchain_gap reproduced the production message before the fix.

  • All 36 test files referencing the dispatch workflow or the materializer: 900 passed, 1 skipped, both plain and with GITHUB_ACTIONS=true. The single failure, test_maturin_offline_build_contract, is environmental and fails identically on main in this local environment (the uv venv has no pip).

  • actionlint on the workflow.

Follow-up (not in this PR)

Content-addressed coverage image tags (a hash of the Dockerfile, lock inputs and pinned base digest) would reuse identical images across runs. That saves build time, but it changes the --pull --no-cache freshness semantics, so it needs its own review. Disk usage is already bounded by #2491 plus this reclaim step.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PtgtJk1WjqieqvLDb1w8uW

Every Rust coverage-evidence run on cwlab-s1-04 failed with
"could not run trusted cargo vendor ...: FileNotFoundError" (e.g. the
fast-mlsirm#2157 dispatch, run 36529717504): rustup's cargo is in
~/.cargo/bin, which the runner's .path omits. A failed coverage gate keeps
opencode-agent from approving, so no fast-mlsirm PR could get approval.

- New "Expose runner Rust toolchain" step adds ~/.cargo/bin to GITHUB_PATH
  when cargo is not already on PATH, and warns when it is absent.
- The materializer reports a missing cargo as a runner toolchain gap.
- New "Reclaim stale coverage images" step removes only
  opencode-coverage-tools images older than 2 h, dangling images and build
  cache older than 24 h; failure warns and never blocks measurement. The
  guest disk refilled to 96% in hours from per-run images left by runs
  dispatched before per-run cleanup (#2491).

Behavioral tests run each step with fake docker/HOME; the materializer test
reproduces the production message before the fix. Dispatch blob pin updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PtgtJk1WjqieqvLDb1w8uW
@seonghobae

Copy link
Copy Markdown
Contributor Author

Bypass merge record (coordinator-authorized, .github only): the author reviewed the diff directly at head 808299e9. It adds two workflow steps before measurement, neither of which touches PR content, a readable-error branch in the materializer, tests, the blob pin and a changelog fragment. The docker commands are scoped to reference=opencode-coverage-tools with until=2h, plus dangling images and build cache older than 24 h. There's no prune -a, and failures don't block measurement. Local tests: 900 passed (plain and GITHUB_ACTIONS=true); 1 environment-only failure also fails on main. actionlint passed. Hosted checks are stuck behind the org queue, and this is the fix blocking every fast-mlsirm approval, so it's merged with the bypass. After merge: dispatches created from now on use this workflow, and fast-mlsirm#2157 will be re-dispatched.

🤖 Generated with Claude Code

@seonghobae
seonghobae merged commit 38eff13 into main Sep 29, 2026
4 of 8 checks passed
@seonghobae
seonghobae deleted the seonghobae/coverage-cargo-path-docker-hygiene branch September 29, 2026 13:57
@coderabbitai

coderabbitai Bot commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 4726de28-1b49-4738-93f4-d5fbf814ad7e

📥 Commits

Reviewing files that changed from the base of the PR and between 2e28521 and 808299e.

📒 Files selected for processing (6)
  • .github/workflows/opencode-review-dispatch.yml
  • CHANGELOG.d/20260929-coverage-runner-hygiene.md
  • scripts/ci/materialize_base_rust_dependencies.py
  • tests/test_materialize_base_rust_dependencies.py
  • tests/test_opencode_coverage_runner_hygiene.py
  • tests/test_pr_review_autofix_nvidia_nim_contract.py
 ________________________________________________________________________________________________________________________________________
< Use saboteurs to test your testing. Introduce bugs on purpose in a separate copy of the source to verify that testing will catch them. >
 ----------------------------------------------------------------------------------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant