ci: run monitoring dashboard tests

[gfyrag]

Summary

• make just pre-commit run the nested monitoring-dashboard Go tests
• regenerate dashboard artifacts before the semantic tests execute
• fail closed through the repository invariant gate if the module/tests/recipe/CI reachability chain is removed
• document the executable reachability contract

Reproduction

• target base: ae5346f1d183e22b498b6d429c37178370a80fa1
• root go list -test ./...: 337 package entries, zero dashboard-module matches
• dashboard module discovery: TestGeneratedDashboards
• baseline uncached module run: PASS (15.846s package runtime, 37.10s wall)
• GitHub workflows/actions had zero dashboard-module or dashboard-test references

Required CI compatibility

The open Required CI implementation (#1835) already aggregates both Dirty and Repository-Invariants. This change adds the test to the existing Dirty producer via pre-commit and extends the existing invariant entry point, so it creates no unaggregated producer.

Validation

• just test-dashboards
• uncached go test -count=1 ./... in misc/devenv/monitoring-dashboards
• fail-closed mutation: removing test-dashboards from pre-commit is rejected
• bash scripts/check-repo-invariants
• bash scripts/agent-check
• base-pinned trusted scripts/agent-just pre-commit
• repeated candidate pre-commit: identical before/after fingerprint (cbf6b962fed0f5fa141a9c5dc303adb57c2a8400f83a6f7286836fc6638d44f0)
• git diff --check

Finding: test-reachability-enforcement/dashboard-tests-never-collected (P2).

No automatic merge requested.

[shipfox-ai]

Arbitration — PR #1839 (ci: run monitoring dashboard tests)

The core change is sound: pre-commit now depends on test-dashboards, which depends on generate-dashboards and then runs the nested module's go test ./..., so the previously-unreached misc/devenv/monitoring-dashboards tests now execute in the Default CI Dirty job. Generation ordering is preserved and no correctness regression exists in the recipe. I independently verified the diff at head 85802fb, the main.yml workflow, and the CI check runs: the new scripts/check-dashboard-test-reachability uses grep/grep -E/grep -F only (no rg), and the invariants check-run for this head is completed / success. Reviewer A's original Blocker (missing rg Nix dependency) and Major (shell probe short-circuits go run ./scripts in a red CI) were both factually wrong and have been correctly retracted by A — the CI is green and no such dependency exists. The remaining findings are all defensive-hardening nits on a fail-closed gate. Overall recommendation: approve with comments.

Agreed findings

• Minor — comment/disabled-command bypass (fail-open). scripts/check-dashboard-test-reachability:34 (grep -Fq "cd $dashboard_module && go test ./..." over just --show test-dashboards) and :38 (grep -Fq 'nix develop --command just pre-commit' over raw .github/workflows/main.yml) are unanchored fixed-substring matches with no comment stripping. Commenting out either command (# cd misc/...&& go test ./..., # nix develop --command just pre-commit) leaves the substring present, so the gate passes while the tests no longer run. This contradicts the guarantee added in this PR in docs/technical/architecture/repository-invariant-gates.md:48-51 ("fails if … the nested go test ./... command disappears, or CI stops invoking pre-commit"). Remediation: parse Just/YAML structure or strip comments, and add negative mutation tests for commented/disabled commands. (Both reviewers agree; A's refinement is also correct and adopted: the two dependency-header checks at :25/:30 are ^-anchored to pre-commit:/test-dashboards: and are not comment-vulnerable, so the defect is confined to the two -F substring checks.)

• Minor — brittle literal CI-command coupling (fail-positive). scripts/check-dashboard-test-reachability:38 requires one exact contiguous source substring. It passes today only because main.yml's - run: > folded scalar keeps nix develop --command just pre-commit on a single physical line (verified). Behavior-preserving refactors — splitting the command across YAML lines, re-wrapping the folded scalar, or moving the invocation into a composite action — would trip this fail-closed gate even though pre-commit still runs. repository-invariant-gates.md itself requires rules with "an unambiguous syntactic boundary and a low false-positive" rate; a whole-command literal match is a weak boundary. This is the false-positive twin of the comment bypass and should be addressed by the same shift to structural parsing. (A raised it; B accepted it.)

Ruled disputes

• Docs overstate execution vs. wiring. A: the README/gate docs imply verified execution while the gate only checks textual wiring. B: the README (misc/devenv/monitoring-dashboards/README.md:63-66) claims the tests run in CI via the dependency chain, not that the script executes them, and repository-invariant-gates.md:48 says the gate "inspects" — so wiring and execution are compatible, non-conflicting statements. Ruling: B. I verified the README wording ("… is also part of pre-commit and therefore runs in the Default CI workflow") describes the real, active chain test-dashboards → pre-commit → Dirty job runs \just pre-commit``, which is accurate; the docs do not attribute execution to the reachability script. No separate documentation defect stands beyond the already-counted comment-bypass gap. Optional nicety: the sentence describing the gate could say "verified wiring," but this is not a required change.

• Ordering: shell probe runs before exec go run ./scripts. A (reduced to a nit): prepending bash check-dashboard-test-reachability under set -euo pipefail means a future failure there short-circuits the primary Go invariant runner; fold it into go run ./scripts. B: this is ordinary fail-fast behavior, no requirement mandates that later checks run after an earlier failure, and the green CI disproves any current masking. Ruling: B, with a non-blocking note. There is no defect and nothing is currently masked. Consolidating the check into the existing go run ./scripts runner (consistent with how the other gates are implemented) is a reasonable consistency improvement, but it is a stylistic suggestion, not a review finding.

Reviewed by Claude (claude-opus-4.8) and Codex (gpt-5.6-sol) via Shipfox; arbitrated by Claude.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.87%. Comparing base (ae5346f) to head (b0cbb8d).

Additional details and impacted files

@@               Coverage Diff                @@
##           release/v3.0    #1839      +/-   ##
================================================
+ Coverage         76.84%   76.87%   +0.02%     
================================================
  Files               472      472              
  Lines             50356    50356              
================================================
+ Hits              38696    38709      +13     
+ Misses             8261     8248      -13     
  Partials           3399     3399              

Flag	Coverage Δ
e2e	76.87% <ø> (+0.02%)	⬆️
scenario	76.87% <ø> (+0.02%)	⬆️
unit	76.87% <ø> (+0.02%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.