ci: run nested model workload race tests

[gfyrag]

Summary

• run the tests/antithesis/workload nested module's deterministic suite with -race in a dedicated mandatory Tests-Antithesis-Workload job
• preserve the existing 180-second three-node runtime Tests-Model campaign unchanged
• add a repository invariant that structurally enforces both distinct jobs and their mandatory failure behavior

Discovery and reproduction

• recorded target base: ae5346f1d183e22b498b6d429c37178370a80fa1
• rebased integration base: 7c88752e1afacb9afdfe11635ebdbee79aa93686
• root go test ./... passed in 256s and did not enumerate any tests/antithesis/workload package
• the existing runtime harness builds singleton_driver_model but does not invoke go test
• pinned Nix Go 1.26.5: go -C tests/antithesis/workload test -race ./... passed with no failures in 50.92s cold
• inspected open model/query PR test(antithesis): model-driver query coverage — filters, indexes, typed metadata (EN-1625) #1659 and Required CI PR ci: add fail-closed Required CI aggregate #1835; no workload driver sources are changed or absorbed

Placement

The deterministic suite has its own Tests-Antithesis-Workload job. This separates deterministic unit/race failures from the long runtime model campaign, lets both jobs run in parallel, and provides a dedicated required status. The additional checkout/Nix setup is acceptable for a roughly 51-second cold suite.

Release-Latest and Build-PR-Image depend on both jobs. When pending Required CI PR #1835 rebases after this PR lands, it must add Tests-Antithesis-Workload to Required-CI.needs; its fail-closed topology invariant will enforce that update.

Validation

• dedicated nested module race command: PASS
• unchanged just test-model-cluster 180: PASS with four rolling restart cycles and no findings (local 95% disk required documented HEALTH_THRESHOLD=0.99 accommodation)
• focused reachability tests: PASS
• repository invariants, including dashboard/operator/workload reachability: PASS
• AI_REVIEW_BASE_SHA=7c88752e... bash scripts/agent-check-pr: PASS (agent-check-full, including root go test -race ./...)
• trusted pre-push fixpoint: PRE_COMMIT=PASS, WORKTREE_CLEAN=YES
• exact review: APPROVE, 0 blocking / 0 non-blocking findings

No branch-protection bypass requested. Merge only after all mandatory CI and review requirements are green.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.77%. Comparing base (7c88752) to head (b66fc4e).

Additional details and impacted files

@@               Coverage Diff                @@
##           release/v3.0    #1842      +/-   ##
================================================
- Coverage         76.84%   76.77%   -0.08%     
================================================
  Files               472      472              
  Lines             50356    50356              
================================================
- Hits              38697    38659      -38     
- Misses             8262     8296      +34     
- Partials           3397     3401       +4     

Flag	Coverage Δ
e2e	76.77% <ø> (-0.08%)	⬇️
scenario	76.77% <ø> (-0.08%)	⬇️
unit	76.77% <ø> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[shipfox-ai]

Final review — PR #1842 "ci: run nested model workload race tests"

Summary. This PR adds a mandatory -race preflight for the nested tests/antithesis/workload module to the Tests-Model CI job, keeps the existing 180s runtime model campaign, and adds a repository-invariant checker (scripts/check_model_workload_reachability.go) that structurally verifies the preflight command, its placement inside Tests-Model, its ordering before the campaign, and that neither the job/step is conditional nor ignores failure. I verified the change against the PR diff and the checked-in code: the checker logic is correct for the cases it covers, its unit tests exercise the accepted form plus the rejected non-race / conditional-job / conditional-step / continue-on-error: true / wrong-job / missing-campaign / post-campaign-ordering forms, the go.yaml.in/yaml/v3 dependency is already present, and the check is wired into checkFile and run through go run ./scripts. No changes touch FSM/cache/preload, persisted/protobuf/audited state, or any AGENTS.md non-negotiable invariant, and the added files are package main under scripts/ so production go build ./... is unaffected. Recommendation: approve with comments — the two retained items are minor coverage gaps in a CI-tooling gate, not blockers.

I rejected the remaining candidate findings after verification: the byte-exact command matching (Reviewer A #2) is a documented, intentional trade-off with no correctness/security/compat impact; the doc-wording claim (Reviewer A #3) is not substantiated — the doc's "must be unconditional / mandatory failure behavior" describes exactly what the checker verifies (job if, step if, step continue-on-error, ordering), and it never claims to enforce required-status-for-merge, which the PR explicitly attributes to #1835; the non-bool continue-on-error note (Reviewer A #4) is, by the reviewer's own account, correct conservative behavior and only a defensive-branch test gap, not material.

Confirmed findings

1. Enforcement disappears if the workflow file is renamed or deleted — minor
scripts/check-repo-invariants.go (checkFile, the case path == defaultCIWorkflowPath branch) and scripts/check_model_workload_reachability.go.
The reachability check runs only when git ls-files enumerates a file whose path is exactly .github/workflows/main.yml. checkFile returns nil, nil for a non-existent path, and there is no independent "the workflow must exist" assertion. If main.yml is renamed, moved, or deleted, checkModelWorkloadTestReachability never executes and the gate passes with zero findings — defeating the stated goal of structurally guaranteeing the preflight cannot be removed. Impact is limited because deleting/renaming the primary CI workflow is highly visible in review; consider anchoring the assertion so a missing workflow fails the gate.

2. Job-level continue-on-error: true on Tests-Model bypasses the "cannot ignore failures" invariant — minor
scripts/check_model_workload_reachability.go (workflowReachability struct + checkModelWorkloadTestReachability).
The checker parses only a job-level if, and per-step if / continue-on-error. It does not parse the valid GitHub key jobs.<job_id>.continue-on-error. A Tests-Model job declaring continue-on-error: true at the job level would cause the runner to tolerate the whole job's failure, yet the gate would still pass with zero findings. This directly under-covers the documented claim in docs/technical/architecture/repository-invariant-gates.md ("The job and preflight must be unconditional, neither command can ignore failures"). Like finding 1, exploitation requires a visible workflow edit, so severity is minor; adding a job-level continue-on-error field to workflowReachability and rejecting a non-false value (mirroring workflowFailureIgnored) would close it, and a unit test for the string/expression continue-on-error case would lock in the intended conservative semantics.

Reviewed by Claude (claude-opus-4.8) and Codex (gpt-5.6-sol) via Shipfox; verified and synthesized by Claude.

[NumaryBot]

✅ Approve — automated review

The dedicated race-test job is valid, mandatory on workflow runs, and the repository invariant accurately verifies both workload and runtime campaign reachability.

No findings.