feat(ci): cross-check the documents against the configs

[Gerrrt]

Closes #154. Fixes #73, and closes the mechanism half of #72 and #68.

What changed

scripts/check_docs.py asserts the prose still agrees with the configs it describes, and runs in both make validate and CI. Nothing about the running lab changes — this is a build-time check plus the seven stale strings it found.

Six assertions, following the pattern scripts/snmp-targets.sh --check already established for the SNMP inventory:

	Assertion
1	Counted claims — rules, dashboards and panels quoted in prose
2	SNMP targets — snmp.yaml ↔ docs/network.md
3	Host and stack table — docs/architecture.md ↔ docs/network.md, and every stacks/*/ is named
4	Ports table — docs/architecture.md ↔ compose.yaml
5	Compute table — docs/hardware.md ↔ docs/network.md
6	Image versions quoted in prose ↔ compose.yaml

It found seven live disagreements on the first run, all in stacks/observability/README.md — the stale 34 alert rules from #72, and all six stale image versions from #73, whose title already named the cause: "and CI does not check Markdown". Fixed in this PR, which is why it is green.

$ python3 scripts/check_docs.py
  stacks/observability/README.md:26 claims 34 alert rules; the repository has 35 or 48
  ^ counted claims

  stacks/observability/README.md:11 says prom/prometheus:v3.1.0; compose.yaml pins v3.14.0
  stacks/observability/README.md:12 says prom/alertmanager:v0.28.0; compose.yaml pins v0.34.0
  stacks/observability/README.md:13 says grafana/loki:3.3.2; compose.yaml pins 3.7.6
  stacks/observability/README.md:14 says grafana/grafana-oss:11.5.2; compose.yaml pins 13.0.2
  stacks/observability/README.md:15 says prom/snmp-exporter:v0.28.0; compose.yaml pins v0.30.1
  stacks/observability/README.md:16 says grafana/alloy:v1.6.1; compose.yaml pins v1.18.1
  ^ image versions quoted in prose

7 disagreement(s) between the documents and the configs

It also wires scripts/snmp-targets.sh --check into CI. Comparing the scripts each side invokes, that was the only check make validate had and CI did not — so the SNMP inventory had never been verified on a pull request. That is #68 in the flesh, and adding a documentation check to one side only would have repeated it. The two lists are now identical:

$ comm -13 <ci-scripts> <validate-scripts>   # in validate.sh but not CI
$ comm -23 <ci-scripts> <validate-scripts>   # in CI but not validate.sh

Why

Documentation drift is this repository's most frequently-recurring defect and the only class that had no check. oracle was recorded as an i5-1235U with 32 GB when it is a dual-core A6-9200 with 4 GB — ADR-0008 notes that wrong entry was load-bearing in planning. shiva was described as the hypervisor for several revisions. roadmap.md says of itself that it "has already been wrong about the switch answering SNMP and about the history purge". #104 records a rule count that disagrees with reality.

Every other defect class here was answered by moving truth somewhere CI can check it. This does that for docs/.

Two decisions worth reviewing

Scope is a fixed file list, not an ignore list. docs/roadmap.md and docs/adr/ are excluded because they record what was true when the work landed — roadmap.md still says "(34 rules)" in a Done entry and that is correct as written. Failing on those would need a suppression mechanism, and this repository already learned where that leads with the .gitleaksignore deleted in the history purge: "an acknowledgement, not a fix… a CI job that is permanently red for a known reason gets ignored."

Two definitions are pinned in the checker rather than left implicit. A panel count includes rows (84 with, 73 without), matching the number already in the documents. And profile-gated services are excluded from the ports check, so renderer behind the capture profile does not make a correct document fail — the running stack is six services and the docs are right to say so.

Known consequence

Assertion 6 means a Dependabot image bump will now fail CI until stacks/observability/README.md is updated in the same PR. That is the intended reading of #73, but the alternative is worth naming: ci.yml already says "Image versions are NOT duplicated here… hardcoded copies went stale silently", and the stricter fix would be to drop version numbers from that table entirely and let compose.yaml be the only place they appear. Happy to switch if you prefer that reading.

Blast radius

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml — the checker reads tracked files only and never needs a decryption key

Build-time only. Nothing is deployed, no container is added, no service changes.

Verification

Each assertion was mutation-tested, because a check that passes and cannot fail is what #63 was. A deliberate error in each of the six inputs, one at a time:

Mutation	Result
48 → 49 alert rules in README.md	✅ README.md:48 claims 49 alert rules; the repository has 35 or 48
mjolnir address changed in network.md	✅ snmp.yaml polls mjolnir at 10.0.99.10 on VLAN 99, and docs/network.md does not list that host at that address
oracle address changed in architecture.md	✅ docs/architecture.md places oracle at 10.0.99.31 on VLAN 99; docs/network.md does not list it there
Loki port 3100 → 3200 in the ports table	✅ docs/architecture.md says loki publishes 3200; compose.yaml publishes 3100
oracle OS changed in hardware.md	✅ docs/hardware.md says oracle runs 'Ubuntu Server 22.04.1'; docs/network.md says ubuntu 24.04.3
grafana/loki version reverted in the stack README	✅ stacks/observability/README.md:13 says grafana/loki:3.3.2; compose.yaml pins 3.7.6

All six exit non-zero and name the file and line. Clean tree passes:

$ python3 scripts/check_docs.py
docs OK — 35 Prometheus + 13 Loki rules, 5 dashboards, 84 panels, 6 assertions

• make validate passes — not run in full, and saying so rather than implying otherwise. No docker daemon, promtool, amtool, alloy or gitleaks available here, so the compose, Prometheus, Alertmanager, Alloy and secret-scan sections would have skipped, and this repository is right that skipped checks prove nothing. What did run: check_docs.py, check_dashboards.py, check_compose_health.py, snmp-targets.sh --check, markdownlint-cli2, yamllint --strict, bash -n scripts/validate.sh — all clean. CI is the authority on the rest.
• Deployed to the lab and confirmed working — n/a, nothing deployable changed
• Docs updated — stacks/observability/README.md corrected, and README.md gains a Highlights bullet for the new check

🤖 Generated with Claude Code

https://claude.ai/code/session_01W1QX12EJLwMfRAWsbpfRJz

---

Generated by Claude Code