fix: double-mount runner tool cache at /host prefix for AWF chroot on GPU runners by Copilot · Pull Request #40033 · github/gh-aw

Copilot · 2026-06-18T11:23:31Z

All workflows on aw-gpu-runner-T4 have been failing since inception because the AWF chroot pre-flight check can't find node. The GPU runner stores its tool cache at /home/runner/work/_tool (not /opt/hostedtoolcache), and the AWF container's chroot root is /host — so a path mounted only at its normal container path is invisible to the chroot.

Root cause

toolCacheMountProbe was generating a single bind-mount for non-/opt/* tool caches:

GH_AW_TOOL_CACHE_MOUNT="$GH_AW_TOOL_CACHE:$GH_AW_TOOL_CACHE:ro"
# → container sees /home/runner/work/_tool
# → chroot needs /host/home/runner/work/_tool  ✗

The ${RUNNER_TEMP}/gh-aw path already uses the correct double-mount pattern (once at its normal path, once at /host-prefix), but the tool cache didn't.

Changes

pkg/workflow/awf_helpers.go — Added GH_AW_TOOL_CACHE_HOST_MOUNT variable. Both the RUNNER_TOOL_CACHE and /home/runner/work/_tool fallback branches now set it to $GH_AW_TOOL_CACHE:/host$GH_AW_TOOL_CACHE:ro. Added toolCacheHostMountRef and included it in all four BuildAWFCommand branches:

GH_AW_TOOL_CACHE_MOUNT="$GH_AW_TOOL_CACHE:$GH_AW_TOOL_CACHE:ro"
GH_AW_TOOL_CACHE_HOST_MOUNT="$GH_AW_TOOL_CACHE:/host$GH_AW_TOOL_CACHE:ro"
# AWF invocation now includes both:
${GH_AW_TOOL_CACHE_MOUNT:+--mount "$GH_AW_TOOL_CACHE_MOUNT"} \
${GH_AW_TOOL_CACHE_HOST_MOUNT:+--mount "$GH_AW_TOOL_CACHE_HOST_MOUNT"}

pkg/workflow/awf_config_test.go — Extended TestBuildAWFCommand_AddsToolCacheMountProbe to assert both the normal and /host-prefixed mount vars and refs are present in the generated command.
250 workflow .lock.yml files recompiled; golden test files updated.

Affects all four workflows using aw-gpu-runner-T4: daily-news, daily-issues-report, daily-fact, hippo-embed.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Root cause: The aw-gpu-runner-T4 runner stores its Node.js tool cache at /home/runner/work/_tool instead of /opt/hostedtoolcache. The AWF container's chroot root is /host, so paths only bind-mounted at their normal container path are inaccessible from within the chroot — causing the pre-flight check to fail with "node is not available inside AWF chroot". Fix: In toolCacheMountProbe, also set GH_AW_TOOL_CACHE_HOST_MOUNT with the /host-prefixed container path whenever the tool cache is not under /opt/*. Add ${GH_AW_TOOL_CACHE_HOST_MOUNT:+--mount "$GH_AW_TOOL_CACHE_HOST_MOUNT"} to all four AWF command-building branches so the runner tool cache is double-mounted (normal path + /host-prefixed path), matching the same pattern already used for ${RUNNER_TEMP}/gh-aw. All 250 workflow lock files recompiled. Golden test files updated. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

This PR fixes AWF chroot execution on aw-gpu-runner-T4 by ensuring the runner tool cache is bind-mounted both at its normal container path and at the /host-prefixed path required by AWF’s chroot root. This makes node (and other tool-cache-provided binaries) visible to the chroot pre-flight checks on GPU runners whose tool cache lives outside /opt.

Changes:

Extend the tool cache mount probe in BuildAWFCommand to also emit GH_AW_TOOL_CACHE_HOST_MOUNT and inject its --mount argument into all command assembly branches.
Update unit test coverage to assert the new /host-prefixed mount variable and mount ref are included in the generated AWF command.
Refresh generated golden fixtures and compiled workflow .lock.yml outputs to reflect the updated AWF invocation.

Show a summary per file

File	Description
pkg/workflow/awf_helpers.go	Adds `GH_AW_TOOL_CACHE_HOST_MOUNT` and injects the `/host` mount into all AWF command-building branches.
pkg/workflow/awf_config_test.go	Extends assertions to cover the new host-prefixed tool cache mount variable and mount ref.
actions/setup-cli/install.sh	Fixes sync note to correctly describe `install-gh-aw.sh` as the canonical source.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/with-imports.golden	Updates generated output to include `GH_AW_TOOL_CACHE_HOST_MOUNT` and its AWF `--mount` injection.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/smoke-copilot.golden	Updates generated output to include the host-prefixed tool cache mount.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/playwright-cli-mode.golden	Updates generated output to include the host-prefixed tool cache mount.
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/basic-copilot.golden	Updates generated output to include the host-prefixed tool cache mount.
pkg/workflow/testdata/TestWasmGolden_AllEngines/pi.golden	Updates generated output to include the host-prefixed tool cache mount.
pkg/workflow/testdata/TestWasmGolden_AllEngines/gemini.golden	Updates generated output to include the host-prefixed tool cache mount.
pkg/workflow/testdata/TestWasmGolden_AllEngines/copilot.golden	Updates generated output to include the host-prefixed tool cache mount.
pkg/workflow/testdata/TestWasmGolden_AllEngines/codex.golden	Updates generated output to include the host-prefixed tool cache mount.
pkg/workflow/testdata/TestWasmGolden_AllEngines/claude.golden	Updates generated output to include the host-prefixed tool cache mount.
.github/workflows/workflow-normalizer.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/video-analyzer.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/update-astro.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/test-workflow.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/test-project-url-default.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/test-dispatcher.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/smoke-test-tools.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/smoke-temporary-id.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/smoke-pi.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/smoke-opencode.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/smoke-gemini.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/smoke-crush.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/smoke-ci.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/smoke-antigravity.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/security-review.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/research.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/repo-tree-map.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/release.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/pdf-summary.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/notion-issue-summary.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/issue-triage-agent.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/hippo-embed.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/gpclean.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/functional-pragmatist.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/firewall.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/example-permissions-warning.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/example-failure-category-filter.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/dependabot-repair.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/dependabot-go-checker.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/daily-team-status.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/daily-malicious-code-scan.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/daily-credit-limit-test.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/daily-byok-ollama-test.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/copilot-pr-merged-report.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/codex-github-remote-mcp-test.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/changeset.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/brave.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/bot-detection.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/architecture-guardian.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/ai-moderator.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/agentic-token-optimizer.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.
.github/workflows/ace-editor.lock.yml	Regenerated compiled workflow output reflecting the additional tool cache `/host` mount.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 136/262 changed files
Comments generated: 0

pelikhan · 2026-06-18T12:25:06Z

@copilot merge main and recompile
Run pr-finisher skill

…ews-failure # Conflicts: # .github/workflows/daily-file-diet.lock.yml # .github/workflows/go-fan.lock.yml # .github/workflows/spec-librarian.lock.yml Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2026-06-18T12:38:33Z

@copilot merge main and recompile
Run pr-finisher skill

Done. Merged main into this branch (merge commit c347e85), resolved 3 .lock.yml conflicts by keeping our GH_AW_TOOL_CACHE_HOST_MOUNT changes, then ran make recompile — all 250 workflows compiled cleanly. make fmt, make lint, and make test-unit all pass locally.

CI must be re-triggered by a maintainer (close/reopen or push) to confirm on the runner.

pelikhan · 2026-06-18T12:55:14Z

@copilot still broken https://github.com/github/gh-aw/actions/runs/27760141601/job/82132181721#step:44:1

github-actions · 2026-06-18T21:08:31Z

@copilot please fix the GPU-runner tool-cache mount issue and rerun checks, then summarize any remaining blocker.

Generated by 👨‍🍳 PR Sous Chef · 46.5 AIC · ⌖ 0.993 AIC · ⊞ 17.2K · ◷

github-actions · 2026-06-19T00:22:57Z

@copilot rerun the GPU-runner validation and summarize the remaining blocker.

Generated by 👨‍🍳 PR Sous Chef · 71.6 AIC · ⌖ 1 AIC · ⊞ 17.2K · ◷

Initial plan

4156e1e

Copilot AI assigned Copilot and pelikhan Jun 18, 2026

Copilot started work on behalf of pelikhan June 18, 2026 11:37 View session

Copilot AI and others added 2 commits June 18, 2026 11:51

chore: report initial plan for daily-news fix

8b8cc62

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Fix daily news workflow issue~~ fix: double-mount runner tool cache at /host prefix for AWF chroot on GPU runners Jun 18, 2026

Copilot finished work on behalf of pelikhan June 18, 2026 12:10

Copilot AI requested a review from pelikhan June 18, 2026 12:10

pelikhan marked this pull request as ready for review June 18, 2026 12:12

Copilot AI review requested due to automatic review settings June 18, 2026 12:12

Copilot started reviewing on behalf of pelikhan June 18, 2026 12:13 View session

Copilot AI reviewed Jun 18, 2026

View reviewed changes

Copilot started work on behalf of pelikhan June 18, 2026 12:28 View session

github-actions Bot mentioned this pull request Jun 18, 2026

[team-status] Daily Team Status — June 18, 2026 #40049

Open

Copilot finished work on behalf of pelikhan June 18, 2026 12:38

Copilot started work on behalf of pelikhan June 18, 2026 13:00 View session

github-actions Bot mentioned this pull request Jun 18, 2026

[PR Triage Report] [PR Triage] Copilot Agent PR Queue — 2026-06-18 (Run #27762753975) #40066

Open

Copilot finished work on behalf of pelikhan June 18, 2026 13:49

pelikhan closed this Jun 19, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: double-mount runner tool cache at /host prefix for AWF chroot on GPU runners#40033

fix: double-mount runner tool cache at /host prefix for AWF chroot on GPU runners#40033
Copilot wants to merge 4 commits into
mainfrom
copilot/debug-daily-news-failure

Copilot AI commented Jun 18, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

pelikhan commented Jun 18, 2026

Uh oh!

Copilot AI commented Jun 18, 2026

Uh oh!

pelikhan commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Root cause

Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Copilot's findings

Uh oh!

pelikhan commented Jun 18, 2026

Uh oh!

Copilot AI commented Jun 18, 2026

Uh oh!

pelikhan commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 18, 2026

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Jun 18, 2026 •

edited

Loading