Skip to content

Promote agents to dev (2026-10-22, helper routing) - #789

Merged
zah merged 6 commits into
devfrom
agents-to-dev-2026-10-22-2
Oct 23, 2026
Merged

Promote agents to dev (2026-10-22, helper routing)#789
zah merged 6 commits into
devfrom
agents-to-dev-2026-10-22-2

Conversation

@zah

@zah zah commented Oct 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Route flake matrix evaluation and planning through the caller's existing helper-runner input. Planning no longer inherits capabilities needed only by later VM builds. The complete build job, matrix mapping, and Foundry override are retained.

The capability gate checks the actual planner bindings; the previous production bindings fail the new control. The existing downstream caller placement contract and its 23 negative/vacuity controls also passed.

Verified locally on Linux at cc41f18:

  • Native hooks, complete provider Go suite, 153 declared D tests, and all 14 governance scripts.
  • All 180 named Nix checks, secret integration, five setup-Nix suites, and cloud-init runtime. Outputs may be reused/substituted; no forced reexecution claim.
  • Exact restoration after the predecessor-failing control and unchanged committed Nix graph.

Original CI on this promotion remains mandatory, including Darwin. These source checks do not establish deployed runner or shared-store reuse acceptance.

Update 2026-10-23: agents merged in (c21ccb5), restarting the stabilisation. Brings in the release-train batch so the provider changes reach dev, and therefore the fleet, in one GARM restart window:

  • 2813949: garm-provider-agentharbor runs JIT jobs in the payload work directory (the JIT .runner workFolder now points at the work dir instead of RUNNER_ROOT/_work).
  • 4dc9100, 206d405: shared host Nix client wrappers (landed on agents after the cut).

Verified locally on Linux x86_64 at c21ccb5:

  • All 181 checks.x86_64-linux build. deployment-reconciler-timer-retry-vm and t_garm_capability_placement timed out at host load ~250 and passed on re-run.
  • The provider Go suite passes, except TestIncusDeleteReclaimsEvenWhenTheCallerContextExpires, which fails at that load when the whole internal/backend package runs. It fails the same way on dev's own source, and passes 5/5 on its own.
  • The 14 terraform/github governance and budget scripts pass, the 150 mcl-devops D tests pass, and prek run --all-files passes.

Not verified locally: Darwin. The darwin batch failures on the previous head (nixos-rebuild-ng tmpdir test, lsof missing, durable-state test on m3 runners) also appear on dependabot PR #577 against dev, so they are runner environment, not this change.

zah added 2 commits October 23, 2026 02:01
@zah
fix(ci): place matrix planners on helper runners
ed33c09 
Use the caller's existing helper route for evaluation and planning instead of inheriting VM build capabilities. Preserve the full build expression, matrix mappings and reserved-lane selection; cover the real planner bindings in the capability gate.
@zah
chore(lock): record the planner routing source
cc41f18 
Refresh only the canonical self revision and integrity after the helper-routing fix. Preserve every solved graph field and the complete Nix input graph.
@github-actions

github-actions Bot commented Oct 22, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Thanks for your Pull Request!

Below you will find a summary of the cachix status of each package, for each supported platform.

package x86_64-linux x86_64-darwin aarch64-darwin
attic-migrate-flake ✅ Cached at <0> 🚫 not supported ❌ build failed
attic-migrate-flake-idempotent ✅ Cached at <0> 🚫 not supported ❌ build failed
attic-push-flake-outputs-action ✅ Cached at <0> 🚫 not supported ❌ build failed
aztec ✅ Cached at <0> 🚫 not supported ❌ build failed
cachix ✅ Cached at <0> 🚫 not supported ❌ build failed
cachix-deploy-metrics ✅ Cached at <0> 🚫 not supported ❌ build failed
check-nixos-runner-hazards ✅ Cached at <0> 🚫 not supported ❌ build failed
ci-build-batching ❌ build failed 🚫 not supported ❌ build failed
ci-image ✅ Cached at <0> 🚫 not supported 🚫 not supported
cloudflare-mail-auth ✅ Cached at <0> 🚫 not supported ❌ build failed
consumer-flake-cachix-inventory ✅ Cached at <0> 🚫 not supported ❌ build failed
consumer-flake-cachix-inventory-tool ✅ Cached at <0> 🚫 not supported ❌ build failed
consumer-flake-no-cachix-residual ✅ Cached at <0> 🚫 not supported ❌ build failed
consumer-flake-no-cachix-residual-tool ✅ Cached at <0> 🚫 not supported ❌ build failed
dcd ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-attic-nginx-logs-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-attic-push-substitute-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-break-glass-runbook-sections ✅ Cached at <0> 🚫 not supported ❌ build failed
deployment-cache-backend-policy ❌ build failed 🚫 not supported ❌ build failed
deployment-cache-corruption-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-cachix-fallback-simulation ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-current-flow-inventory ❌ build failed 🚫 not supported ❌ build failed
deployment-darwin-activation-integration ❌ build failed 🚫 not supported ❌ build failed
deployment-dashboard-query-fixtures ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-direct-ssh-attic-restore-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-direct-ssh-rollback-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-direct-ssh-success-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-e2e-rehearsal-skill-doc ❌ build failed 🚫 not supported ❌ build failed
deployment-event-metrics ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-event-schema-examples ❌ build failed 🚫 not supported ❌ build failed
deployment-incus-rehearsal-image ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-incus-rehearsal-script-static ❌ build failed 🚫 not supported 🚫 not supported
deployment-metrics-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-no-default-cachix-deploy-call ❌ build failed 🚫 not supported 🚫 not supported
deployment-parallel-cache-push-local ❌ build failed 🚫 not supported 🚫 not supported
deployment-production-cutover-simulation-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-pull-agent-darwin-integration 🚫 not supported 🚫 not supported ❌ build failed
deployment-pull-agent-latest-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-pull-agent-lock-contention-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-pull-agent-locks-migration-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-pull-agent-rejects-invalid-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-pull-agent-selfheal-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-pull-agent-static ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-reconciler-lock-contention-vm ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-reconciler-skill-doc ❌ build failed 🚫 not supported ❌ build failed
deployment-reconciler-supersession-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-reconciler-timer-retry-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-reconciler-timer-static ✅ Cached at <0> 🚫 not supported 🚫 not supported
deployment-scheduled-canary-local-vm ❌ build failed 🚫 not supported 🚫 not supported
deployment-skill-doc-command-references ❌ build failed 🚫 not supported ❌ build failed
deployment-summary-artifact ❌ build failed 🚫 not supported ❌ build failed
desktop-vms-golden ✅ Cached at <0> 🚫 not supported 🚫 not supported
dev-certificates-minting ❌ build failed 🚫 not supported ❌ build failed
disko ✅ Cached at <0> 🚫 not supported 🚫 not supported
dmd ✅ Cached at <0> 🚫 not supported 🚫 not supported
dscanner ✅ Cached at <0> 🚫 not supported 🚫 not supported
dub ✅ Cached at <0> 🚫 not supported ❌ build failed
encrypted-s3-artifact ✅ Cached at <0> 🚫 not supported ❌ build failed
fleet-alertmanager-config ✅ Cached at <0> 🚫 not supported ❌ build failed
fleet-alertmanager-config-deadman ✅ Cached at <0> 🚫 not supported ❌ build failed
folder-size-metrics ✅ Cached at <0> 🚫 not supported 🚫 not supported
garm ✅ Cached at <0> 🚫 not supported ❌ build failed
garm-fleet-alert-rules ✅ Cached at <0> 🚫 not supported ❌ build failed
garm-fleet-external-checks ✅ Cached at <0> 🚫 not supported ❌ build failed
garm-provider-agentharbor ❌ build failed 🚫 not supported ❌ build failed
garm-provider-aws ✅ Cached at <0> 🚫 not supported ❌ build failed
garm-provider-vmharness ❌ build failed 🚫 not supported ❌ build failed
garm-shared-nix-client ❌ build failed 🚫 not supported 🚫 not supported
git-hooks-reprobuild-handoff ✅ Cached at <0> 🚫 not supported ❌ build failed
git-hooks-same-repo-git ✅ Cached at <0> 🚫 not supported ❌ build failed
github-actions-fit-alert-rules ✅ Cached at <0> 🚫 not supported ❌ build failed
github-actions-fit-exporter ✅ Cached at <0> 🚫 not supported ❌ build failed
google-workspace-dkim ✅ Cached at <0> 🚫 not supported ❌ build failed
google-workspace-dkim-test ✅ Cached at <0> 🚫 not supported ❌ build failed
google-workspace-domains ✅ Cached at <0> 🚫 not supported ❌ build failed
google-workspace-domains-test ✅ Cached at <0> 🚫 not supported ❌ build failed
google-workspace-dwd ✅ Cached at <0> 🚫 not supported ❌ build failed
google-workspace-setup ✅ Cached at <0> 🚫 not supported ❌ build failed
gosti ❌ build failed 🚫 not supported ❌ build failed
ldc ✅ Cached at <0> 🚫 not supported 🚫 not supported
lido-withdrawals-automation ✅ Cached at <0> 🚫 not supported ❌ build failed
linux-vm-cloud-init-image ✅ Cached at <0> 🚫 not supported 🚫 not supported
mcl-devops ❌ build failed 🚫 not supported ❌ build failed
mcl-devops-no-stale-references ❌ build failed 🚫 not supported ❌ build failed
mcl-devops-single-bin-mcl-provider ❌ build failed 🚫 not supported ❌ build failed
mint-dev-certificates ❌ build failed 🚫 not supported ❌ build failed
netbird-with-agenix-login-script ✅ Cached at <0> 🚫 not supported 🚫 not supported
nix ✅ Cached at <0> 🚫 not supported ✅ Cached at <0>
nix-eval-jobs ✅ Cached at <0> 🚫 not supported ✅ Cached at <0>
nix-fast-build ✅ Cached at <0> 🚫 not supported ✅ Cached at <0>
nixos-anywhere ✅ Cached at <0> 🚫 not supported 🚫 not supported
nixos-rebuild-ng ❌ build failed 🚫 not supported ❌ build failed
pyroscope ✅ Cached at <0> 🚫 not supported ❌ build failed
random-alerts ✅ Cached at <0> 🚫 not supported ❌ build failed
reusable-flake-checks-artifact-fallback ❌ build failed 🚫 not supported ❌ build failed
reusable-flake-checks-attic-credentials ❌ build failed 🚫 not supported ❌ build failed
reusable-flake-checks-build-timeout ❌ build failed 🚫 not supported ❌ build failed
reusable-flake-checks-cancellation ❌ build failed 🚫 not supported ❌ build failed
reusable-flake-checks-mcl-ref ❌ build failed 🚫 not supported ❌ build failed
reusable-flake-checks-nix-token ❌ build failed 🚫 not supported ❌ build failed
reusable-flake-checks-s3-mirror-compat ❌ build failed 🚫 not supported ❌ build failed
reusable-lint-hook-discovery ❌ build failed 🚫 not supported ❌ build failed
reusable-terraform-drift-workflow ❌ build failed 🚫 not supported ❌ build failed
reusable-terraform-policy-advisories ❌ build failed 🚫 not supported ❌ build failed
reusable-terraform-source-identity ❌ build failed 🚫 not supported ❌ build failed
runner-label-tool ✅ Cached at <0> 🚫 not supported ❌ build failed
rustToolchain ✅ Cached at <0> 🚫 not supported ❌ build failed
seal-alerting-secrets ✅ Cached at <0> 🚫 not supported ❌ build failed
secret-integration ❌ build failed 🚫 not supported 🚫 not supported
serve-d ✅ Cached at <0> 🚫 not supported 🚫 not supported
setup-nix-hosted-disk-reclamation ❌ build failed 🚫 not supported ❌ build failed
setup-nix-substituter-preflight ✅ Cached at <0> 🚫 not supported ❌ build failed
setup-nix-transfer-resilience ✅ Cached at <0> 🚫 not supported ❌ build failed
t_aws_burst_runners ❌ build failed 🚫 not supported 🚫 not supported
t_aws_spot_runners ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_central_garm_recovery ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_ci_nixos_runner_hazards ❌ build failed 🚫 not supported ❌ build failed
t_ci_runs_on_capability ❌ build failed 🚫 not supported ❌ build failed
t_cross_host_publish_substitute ❌ build failed 🚫 not supported 🚫 not supported
t_fit_exporter_repos_env_roundtrip ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_fleet_alert_routing ✅ Cached at <0> 🚫 not supported ❌ build failed
t_fleet_alerting ✅ Cached at <0> 🚫 not supported ❌ build failed
t_garm_api_watchdog ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_busy_runner_not_reaped ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_capability_placement ❌ build failed 🚫 not supported 🚫 not supported
t_garm_central_multi_host ❌ build failed 🚫 not supported 🚫 not supported
t_garm_credentials_no_race ❌ build failed 🚫 not supported 🚫 not supported
t_garm_credentials_no_race_restart_contract ❌ build failed 🚫 not supported 🚫 not supported
t_garm_deleting_capacity ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_incus_runner_host ❌ build failed 🚫 not supported 🚫 not supported
t_garm_incus_storage_pool_source ❌ build failed 🚫 not supported 🚫 not supported
t_garm_install_script_template_fixtures ❌ build failed 🚫 not supported ❌ build failed
t_garm_instance_lifecycle ❌ build failed 🚫 not supported 🚫 not supported
t_garm_job_cache_self_heal ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_macos_pools_supported ✅ Cached at <0> 🚫 not supported ❌ build failed
t_garm_macos_runner_install_wrapper ❌ build failed 🚫 not supported ❌ build failed
t_garm_multi_provider ❌ build failed 🚫 not supported 🚫 not supported
t_garm_nixos_service_boots ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_pools_labels ❌ build failed 🚫 not supported 🚫 not supported
t_garm_provider_agentharbor ❌ build failed 🚫 not supported 🚫 not supported
t_garm_provider_agentharbor_module ❌ build failed 🚫 not supported 🚫 not supported
t_garm_provider_remote ❌ build failed 🚫 not supported 🚫 not supported
t_garm_provider_remote_old_daemon ❌ build failed 🚫 not supported 🚫 not supported
t_garm_provider_vmharness_backend ❌ build failed 🚫 not supported ❌ build failed
t_garm_provider_vmharness_protocol ❌ build failed 🚫 not supported 🚫 not supported
t_garm_provider_vmharness_windows_toolchain ❌ build failed 🚫 not supported ❌ build failed
t_garm_reconcile ❌ build failed 🚫 not supported 🚫 not supported
t_garm_remote_incus_capabilities ❌ build failed 🚫 not supported 🚫 not supported
t_garm_remote_libvirt_settings ❌ build failed 🚫 not supported 🚫 not supported
t_garm_skip_queued_jobs_when_pools_full ✅ Cached at <0> 🚫 not supported ❌ build failed
t_garm_stale_scaleset_job_reaped ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_webhook_delivery ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_garm_webhook_public_paths ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_gosti_serve_rename ❌ build failed 🚫 not supported ❌ build failed
t_gosti_serve_survives_incus_blip ❌ build failed 🚫 not supported 🚫 not supported
t_gosti_serve_unit_rename_switch ❌ build failed 🚫 not supported 🚫 not supported
t_pools_cutover_complete ❌ build failed 🚫 not supported 🚫 not supported
t_private_repo_hybrid_fallback ❌ build failed 🚫 not supported ❌ build failed
t_public_repo_free_ci ✅ Cached at <0> 🚫 not supported ❌ build failed
t_repro_binary_cache_systemd_healthz ❌ build failed 🚫 not supported 🚫 not supported
t_repro_cache_client_render ❌ build failed 🚫 not supported 🚫 not supported
t_repro_deploy_agent_https_converges ❌ build failed 🚫 not supported 🚫 not supported
t_repro_lease_reaper_eval ❌ build failed 🚫 not supported 🚫 not supported
t_runner_fit_monitoring ✅ Cached at <0> 🚫 not supported ❌ build failed
t_runner_label_taxonomy ❌ build failed 🚫 not supported ❌ build failed
t_runner_mode_manager ❌ build failed 🚫 not supported ❌ build failed
t_runner_mode_switch ❌ build failed 🚫 not supported ❌ build failed
t_s3_artifact_store_acl_and_creds ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_vmharness_create_fails_on_dead_guest ❌ build failed 🚫 not supported ❌ build failed
t_vmharness_image_is_honoured ❌ build failed 🚫 not supported ❌ build failed
t_vmharness_serve_inventory_textfile ✅ Cached at <0> 🚫 not supported 🚫 not supported
t_vmharness_serve_linux_deploy ❌ build failed 🚫 not supported 🚫 not supported
t_vmharness_serve_win_libvirt ❌ build failed 🚫 not supported 🚫 not supported
terraform ✅ Cached at <0> 🚫 not supported 🚫 not supported
terraform-ci-matrix ✅ Cached at <0> 🚫 not supported ❌ build failed
terranix ✅ Cached at <0> 🚫 not supported 🚫 not supported
test-linux-vm-cloud-init ✅ Cached at <0> 🚫 not supported 🚫 not supported
test_darwin_pull_agent_already_current_is_transaction_free 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_already_current_recovers_lifecycle_before_convergence 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_assertion_gate 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_entrypoint_survives_generation_change 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_launchd_contract 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_lock_contention_is_non_destructive 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_recreates_missing_lock_dir ✅ Cached at <0> 🚫 not supported ❌ build failed
test_darwin_pull_agent_rejects_invalid_durable_state 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_rejects_untrusted_and_wrong_target_manifests 🚫 not supported 🚫 not supported ❌ build failed
test_darwin_pull_agent_same_id_sequence_high_water 🚫 not supported 🚫 not supported ❌ build failed
vm-harness ❌ build failed 🚫 not supported ❌ build failed
vmharness-serve-darwin-posture ❌ build failed 🚫 not supported ❌ build failed
workspace-admin-tools ✅ Cached at <0> 🚫 not supported ❌ build failed
yaml-automation-runner ✅ Cached at <0> 🚫 not supported 🚫 not supported

zah and others added 4 commits October 23, 2026 04:03
@zah
refactor(garm): package shared host Nix client wrappers
4dc9100 
Render the local Incus wrappers and immutable client package from one
producer script. Preserve existing daemon routing, tool selection and
default-off behavior while allowing an explicit target Nix override.
@zah
chore(lock): record the shared client producer source
206d405 
Pin the qualified producer source through the ordinary canonical lock
lifecycle without changing the dependency graph or selected toolchain.
Zahary Karadjov
garm-provider-agentharbor: run JIT jobs in the payload work directory
2813949 
The JIT .runner that GARM serves carries GitHub's workFolder ("_work",
relative to RUNNER_ROOT), so sandbox jobs ran in runner/_work instead of the
payload's work directory (HOME and the runner's cwd). The payload now
rewrites workFolder to the work directory before starting the runner, as
configure --work already does on the registration-token path. Other fields
are kept unchanged; a .runner without the field, or an empty one, gets it
added.

Covered by TestPayloadPointsRunnerWorkFolderAtWorkDir (GitHub's shape, an
absent field, an empty object), by the end-to-end fake listener, which now
refuses to start unless workFolder is its cwd, and by a ninth negative
control in t_garm_provider_agentharbor.
Zahary Karadjov
Merge agents into agents-to-dev-2026-10-22-2: bring in AH3-F1 (2813949)
c21ccb5 
The devops-modules release train batches the provider changes into one
promotion and so into one GARM restart window. AH3-F1 (JIT jobs run in the
payload work directory) is part of that batch, so agents is merged in; this
restarts the stabilisation. It also brings in 4dc9100/206d405d (shared host
Nix client wrappers), which landed on agents after the cut.
@zah

zah commented Oct 23, 2026

Copy link
Copy Markdown
Member Author

Landing at c21ccb5. Every Linux check is green. The one red check is ci / batch 3/6 | aarch64-darwin (and Final Results, which aggregates it): mcl.utils.cache_backends test_cache_probe_timeout_command_executes_inner_command wraps printf in a 1 s timeout and got 124 on the loaded m3-mcl-003. The same derivation passed 146/146 in that job's first attempt, and this promotion does not touch cache_backends.d. The 1 s budget is tight on loaded Darwin runners; that is a follow-up in its own change, not part of this stabilisation. dev has no required status checks.

@zah
zah merged commit 571c44b into dev Oct 23, 2026
113 of 115 checks passed
@zah
zah deleted the agents-to-dev-2026-10-22-2 branch October 23, 2026 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@zah