Alert-rule and panel counts are stale in 8 places, including a deploy-runbook check

[Gerrrt]

Counted from the files on main:

Claim	Where	Actual
40 alert rules (32 Prometheus + 8 Loki)	README.md:47-48	47 — 34 + 13
32 alert rules	README.md:146	34
32 alert rules	docs/security.md:17	34
32 alert rules	stacks/observability/README.md:26	34
40 rules in total: 32 + 8	docs/observability.md:73	47 — 34 + 13
32 rules across four files	docs/observability.md:102	34
Add alerting (32 rules)	docs/roadmap.md:140	historical, fine to leave
"32 rules loaded, none in error"	docs/runbooks/deploy-stack.md:46	34
79 panels	README.md:47	73 non-row, 84 including rows
5 dashboards, 79 panels	stacks/observability/README.md:36	same

Breakdown, for whoever fixes this:

• containers.rules.yaml 8, host.rules.yaml 7, network.rules.yaml 10,
  ups.rules.yaml 9 → 34
• loki/rules/security.rules.yaml → 13 across four groups
• Panels: docker 13, host 16, logs 11, network 18, ups 15 → 73; plus 11 rows
  → 84. scripts/check_dashboards.py prints 84, so neither published number
  matches either count.

The deploy runbook one is the only actively harmful entry: it tells an operator
following a deploy to expect 32, so the correct state now reads as a fault.

The Suricata, firewall-log and UPS self-test rules all landed without the prose
being updated, which suggests this wants a check rather than a one-off edit —
the counts are derivable from the files, and CI already parses all of them.

Found while verifying #12.

[Gerrrt]

#155 landed on main as 4f0e23b. It builds the mechanism this issue needs, but does not finish it — leaving this open deliberately rather than closing it on a partial.

What landed: scripts/check_docs.py recomputes the counts from the configs and fails when a document states a different number. It runs in make validate and in CI. On its first run it found one live instance:

stacks/observability/README.md:26 claims 34 alert rules; the repository has 35 or 48

Fixed in the same PR. Every other counted claim in the present-tense documents already agreed — recomputed today: 35 Prometheus rules, 13 Loki, 48 total, 5 dashboards, 84 panels. So the manual sweep this issue tracks mostly held; one place had drifted since.

What this issue still covers: the sweep itself. If the eight places are now seven-correct-and-one-fixed, this can close; if there are stale counts in files outside the checker's scope, they are still yours to find. The scope is a fixed list in PROSE at the top of the script — README.md, SECURITY.md, docs/*.md, docs/runbooks/*.md and the stack README.

Deliberately out of scope: docs/roadmap.md and docs/adr/. roadmap.md:182 still says "(34 rules)" inside a Done entry, and that is correct as written — it records what was true when the work landed. Failing on it would need a suppression mechanism, and the .gitleaksignore deleted in the history purge is this repository's own account of where that leads.

One thing worth knowing about the panel count: the checker pins the definition at 84, which includes type: row panels; excluding them gives 73. Both are defensible, 84 is what the documents already said, and check_dashboards.py independently arrives at the same figure. Pinning it is what makes the number reproducible rather than one somebody once arrived at.

---

Generated by Claude Code

[Gerrrt]

Closing this: the sweep it was held open for is done, and nothing is stale.

The sweep

The previous comment left this open on one condition — "if there are stale counts in files outside the checker's scope, they are still yours to find." PROSE in scripts/check_docs.py:69-81 covers README.md, SECURITY.md, five docs/*.md, the stack README and every docs/runbooks/*.md. Everything else is unguarded, so I grepped all of it:

File	Claim	Verdict
docs/roadmap.md:213	Add alerting (34 rules)	Correct as written — Done entry
docs/roadmap.md:218	Loki alerting rules (13)	Correct as written — Done entry
docs/adr/0007:32	dashboards and 40 alert rules already exist there	Correct as written — records the state at the decision
docs/images/README.md:50	four of the five dashboards	Accurate — 5 dashboard JSONs, capture-screenshots.sh:56-60 lists 4 uids
docs/images/README.md:84	five dashboards and four screenshots	Accurate — 5 JSON, 4 PNG, homelab-logs excluded on purpose
secrets/README.md, .github/pull_request_template.md	—	No count claims

The roadmap and ADR entries are the historical records this issue's design already argued should stay: they say what was true when the work landed. No stale count remains anywhere in the repository.

Ground truth today

python3 scripts/check_docs.py
  → docs OK — 39 Prometheus + 13 Loki rules, 5 dashboards, 84 panels, 6 assertions
python3 scripts/check_dashboards.py
  → 5 dashboards OK (84 panels, 83 PromQL expressions)

blackbox 3 + containers 9 + host 7 + network 10 + ups 9 + watchdog 1 = 39 Prometheus, 13 Loki, 52 total, 5 dashboards, 84 panels. Every one of the ten places named in the issue body now agrees, including the one that actually mattered — docs/runbooks/deploy-stack.md:107 reads "39 rules loaded, none in error", so an operator following a deploy is no longer told a correct state is a fault.

The panel ambiguity the issue raised — 73 non-row against 84 with rows, "neither published number matches either count" — is settled by pinning the definition at 84, rows included, in count_panels, which check_dashboards.py independently reproduces.

The part that is actual evidence

The counts in that earlier comment were 35 Prometheus and 48 total. They are 39 and 52 now. Rules were added in between, and every document tracked the move without anyone sweeping prose — the checker failed the moment a number went stale and it got fixed in the same change. That is the mechanism working under exactly the conditions that produced this issue: the Suricata, firewall-log and UPS self-test rules landed without prose updates, and that class of drift is now caught rather than noticed.

That is the argument the issue made — "this wants a check rather than a one-off edit" — coming back with a result.

Two residual gaps, not stale counts

Recording these so they are not lost with the issue. Both are holes in the guard, not defects; every number involved is correct today.

1. No make check-docs target. Every sibling has one — check-dashboards, check-rules, check-compose-health, check-loki-rules, check-image-pins at Makefile:113-133. check_docs.py is reachable from validate.sh and CI but not from the Makefile, the only checker with that asymmetry. It is the same shape as make validate and CI have drifted apart in both directions #68, which this script's own commit fixed for snmp-targets.sh.
2. Three present-tense claims fall outside the patterns. README.md:158 says "13 LogQL rules" and the Loki pattern is (\d+)\s+log-based, so it matches nothing. docs/observability.md:111 spells out "across six files". And docs/images/README.md is not in PROSE at all despite being present-tense, so its "five dashboards" would drift silently if a sixth were added.

Happy to open a follow-up for these — say the word and I will.

---

Generated by Claude Code