[Deicyde]

Problem

site/ops/test_brain_release_authority.py tests site/ops/brain_release_authority.py, the release-neutral path policy that the exact-release promoter and the activation bundle freezer use to decide whether main may move past a built release. CI has never run it. It was added in 0a09877 (PR #45), and scripts/ci-python.sh on main has no line for it.

Nothing catches that kind of gap. A tracked test_*.py that no gate runs looks like coverage but isn't.

On main today, 101 test_*.py files are tracked and 95 are run by ci-python.sh. The other six:

• site/ops/test_brain_release_authority.py, which CI should run
• five Brain acceptance gates that fail on a fresh checkout. The Brain nightly runs the first four against its build (site/ops/brain-nightly.sh:713,742,765,773):
  • brain/test_acceptance.py needs the gitignored brain/data/edges_links.jsonl
  • brain/test_cells.py: its data checks pass on tracked brain/data, but its layout check imports numpy (brain/layout.py:32), which requirements-ci.txt doesn't pin
  • brain/test_cell_shards.py needs the gitignored site/assets/brain/cells/
  • brain/test_frontier.py needs the gitignored brain/data/synapses.jsonl
  • brain/test_fold_xref.py needs the private declaration cache plus a Mathlib checkout (CONTRIBUTING.md, "Corpus-dependent checks")

Change

• scripts/ci-python.sh runs site/ops/test_brain_release_authority.py, placed next to the other site/ops release tests.
• New scripts/check_ci_python_tests.py, which runs first in ci-python.sh. It lists tracked files with git ls-files and parses the run_check lines of ci-python.sh, recognizing script paths and -m unittest pkg.test_mod modules; commented-out lines don't count. It fails when:
  • a tracked test_*.py is neither run nor on its CORPUS_ONLY allowlist (the five files above, each with the corpus it needs)
  • an allowlisted file is no longer tracked, or ci-python.sh now runs it, so the allowlist can't go stale
  • a run_check names a test file that isn't tracked
  • it finds no tracked or no CI-run test files at all, so it can't pass vacuously
• it can't parse a run_check line. It then exits with that line instead of a traceback.
• New lines are placed next to related checks, not appended at the end, to avoid textual conflicts with open PRs that append there.

How tested

• python3 scripts/check_ci_python_tests.py prints 101 tracked test files: 96 run by CI, 5 corpus-only. and exits 0.
• python3 site/ops/test_brain_release_authority.py: 7 tests pass.
• Mutation checks against problems(), each producing exactly the expected error:
  • dropping the new run_check line
  • commenting out an existing one
  • adding an untracked or unlisted test
  • un-tracking an allowlisted file
  • running an allowlisted file
• The -m unittest brain.test_provenance_coverage and -I brain/test_replay_sandbox.py forms are recognized.

Risks

• A future PR that adds a test_*.py without a run_check line now fails CI. That is the point; the error message says what to do.
• Open PRs whose test files are already listed (Settle rule: recognise all 28 Mathlib maintainers; keep every tag in a conflict rebuild #57's bot/test_settle_gate.py comes with its own run_check) are unaffected.
• The parser only reads single-line run_check invocations, which is all ci-python.sh uses. A test run some other way would be reported as not run, so it fails loudly rather than passing silently.

What Jack must do

Nothing. There's an optional follow-up: pinning numpy in requirements-ci.txt would likely let brain/test_cells.py move from the allowlist into CI, since everything else it reads is tracked. I haven't verified its layout checks with numpy installed.

Release-neutral (per site/ops/brain_release_authority.py): the PR changes only scripts/.

[Deicyde]

Verdict: ready to merge. CI is green. The meta-check printed 101 tracked test files: 96 run by CI, 5 corpus-only, and 98 checks passed, including the release-policy tests that CI had never run before.

The pre-push review found wrong reasons on two allowlist entries; I corrected them before pushing.

• brain/test_cells.py is not missing data: its inputs are tracked. It only fails because numpy isn't in requirements-ci.txt.
• brain/test_frontier.py needs the gitignored synapses.jsonl.

Optional follow-up: pinning numpy would likely let test_cells.py run in CI. #57's change to ci-python.sh still applies cleanly on top of this.

[Deicyde]

Re-review on main (eaa5950b): fine. On main the check reports 106 tracked test files: 101 run by CI, 5 corpus-only with no problems. That includes every test file added since: bot/test_settle_gate.py, bot/test_publish_queue.py, scripts/test_workflow_credentials.py and site/test_agent_sandbox.py. Open #85's new scripts/atlas/test_build_store.py already has its run_check line. All five allowlist reasons still hold, and no Python test escapes the test_*.py naming rule. Node tests are glob-included by Vitest and Playwright.

By design, not a regression: the parser is lexical. A test path counts as run if it appears on any column-0 run_check line, even dead code or || true. None of main's 101 lines has that shape. A suite that skips every test (such as brain/test_replay_sandbox.py on Linux) also counts as run.