docs(runbooks): fit the UPS battery and rack the shelf switch

[Gerrrt]

What changed

Adds docs/runbooks/fit-the-ups-battery.md — the procedure for tomorrow's rack visit, covering the APCRBC115 pack (#93) and the 1U shelf (#110) as the one job they were bought to be. Linked from docs/roadmap.md (both items), docs/observability.md and the README file tree.

Nothing in the running lab changes. No config, dashboard or rule is touched.

Why

The pack and the shelf arrive tomorrow, and the step that matters most is not a physical one. UpsSelfTestFailed is silenced in Alertmanager until 2026-09-20 (54f1715c-e57b-4322-8a6d-5435bc8e1bd8). Between fitting the pack and that date, the silence would suppress the alert on a UPS that can finally report honestly — including a new pack that is faulty or badly seated, which is exactly the failure you have just introduced and most want to hear about. #93 says to delete it by hand and not let it expire; this puts that in the place you would actually look while standing at the rack.

The order of operations matters too, in a way that is easy to get wrong: moving the TP-Link onto the shelf without moving its power onto a UPS-fed outlet is tidier and no better protected — prometheus and oracle are laptops, so they still ride out a mains cut and go deaf. That is the whole of #110, and it is one step in the middle of the visit.

Written before the boxes are open rather than reconstructed at the rack.

Some specifics the runbook is opinionated about:

• Baseline first. Capture the six fabricated values before touching anything, so the after-state is provably different rather than assumed. A charge still reading exactly 100 and a runtime still at exactly 63 afterwards is the signature of a card that cannot see the new pack — not a coincidence to explain away.
• Delete the silence before the self-test, not after. In the other order the one result you most need to see is the one that is suppressed.
• Read the self-test from both ends — the NMC page and Prometheus — because the card agreeing with itself proves nothing about the scrape path.
• Prometheus queries, not snmpget. The community would otherwise sit on the command line where ps can read it, which is the hazard scripts/snmp-verify.sh was written around.
• Nothing claims a fitted battery. ups.rules.yaml, docs/security.md and the dashboard banner are correct until the runbook succeeds. They are listed as a post-fit checklist inside it, with the coupling flagged: deleting the banner panel changes the dashboard's panel count, and scripts/check_docs.py asserts every panel count quoted in prose against the live JSON.

Blast radius

Documentation only.

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

Verification

make validate — all available checks pass:

Compose health dependencies   PASS
Grafana dashboards            PASS  5 dashboards OK (84 panels, 83 PromQL expressions)
Documentation                 PASS  35 Prometheus + 13 Loki rules, 5 dashboards, 84 panels, 6 assertions
markdownlint                  PASS
SNMP inventory                PASS
Secrets                       PASS

Six checks skipped locally for missing binaries and no docker daemon (promtool, amtool, alloy, loki, yamllint, shellcheck, gitleaks) — CI runs the full set. The change is Markdown only, so nothing skipped covers it.

The new file enters check_docs.py's PROSE tuple automatically via the docs/runbooks/*.md glob, which is why it carries no counted claim: a count describing the post-fit state would fail CI today.

• make validate passes
• Deployed to the lab and confirmed working — n/a, and the runbook itself is unproven until it is followed tomorrow
• Docs updated (docs/, service README, or docs/roadmap.md)

---

Generated by Claude Code