docs(ups): record a battery that is proven, and what still is not

README.md

@@ -222,8 +222,9 @@ and runtime, under a banner explaining that every battery figure is fabricated
 because no battery pack is fitted.](docs/images/ups-power.png)
 
 That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The
-banner it shows has been rewritten since; re-shoot with `make screenshots` once
-the self-test has passed and the panels read something worth showing.
+banner it shows has been rewritten twice since, and the self-test has now
+passed, so the panels read measured values rather than invented ones — re-shoot
+the set with `make screenshots`.
 
 ## What runs it
 
@@ -255,12 +256,14 @@ Open work is tracked in
 [`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and
 why it is in that order.
 
-The current top items: prove the UPS battery and rack the shelf switch. The pack
-went in on 2026-08-28 and has not yet passed a self-test, so the rack is not
-demonstrably protected yet; the switch between the monitoring host and the
-network still has no battery at all
-([#93](https://github.com/Gerrrt/HomeLab/issues/93),
-[#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup
+The current top items: put the UPS battery under scheduled test and rack the
+shelf switch. The pack went in on 2026-08-28 and passed its self-test, so the
+rack itself is protected — but the card is not testing on a schedule, which
+leaves that pass a last-known result rather than a live one
+([#93](https://github.com/Gerrrt/HomeLab/issues/93)); the switch between the
+monitoring host and the network still has no battery at all, so both laptops
+stay running and go deaf on a mains cut
+([#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup
 off the machine it protects, and buy the spare that turns its restore runbook
 from a hypothesis into something rehearsed
 ([#92](https://github.com/Gerrrt/HomeLab/issues/92)); and take 64-bit interface

docs/observability.md

@@ -105,7 +105,7 @@ boot check.
 | --- | --- |
 | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots |
 | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware |
-| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 but has not yet passed a self-test, so most of these still read values the card may be fabricating — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
+| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
 | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself |
 | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable |
 

docs/roadmap.md

@@ -95,12 +95,15 @@ what left this one unfireable for months.
   [`restore-the-firewall.md`](runbooks/restore-the-firewall.md) stays a
   hypothesis until it has been restored onto a spare once.
 - **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.**
-  An APCRBC115 went into `mjolnir` on 2026-08-28. That is the hardware done, not
-  the issue: the management card reports a healthy pack whether or not it can see
-  one, so nothing yet separates a good fit from a bad one. **The `UpsSelfTestFailed`
-  silence is still in place until 2026-09-20, and now suppresses the only alert
-  that could tell you which this is** — delete it, then self-test, then enable
-  scheduled tests on the card.
+  An APCRBC115 went into `mjolnir` on 2026-08-28 and passed its self-test the
+  same day: `upsTestResultsSummary` `4` → `1`, `upsBatteryVoltage` off its
+  fabricated `480`, runtime no longer pinned to exactly `63`. The
+  `UpsSelfTestFailed` silence was deleted rather than left to expire in
+  September, so the rule that would report a bad pack is live again. **What is
+  left is the last step: scheduled self-tests on the card.** Until they are on,
+  `1` is a last-known result with nothing refreshing it, and `UpsBatteryUnproven`
+  cannot detect a card that has quietly stopped testing — it matches `6`
+  (noTestsInitiated), and this one reads `1`.
   → [runbook](runbooks/fit-the-ups-battery.md)
 - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.**
   A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and

docs/runbooks/fit-the-ups-battery.md

@@ -3,21 +3,28 @@
 **One rack visit, in an order that matters — and one alert that has to be
 un-silenced by hand at the right moment.**
 
-> **Status — 2026-08-28: the pack is in, and nothing else is done.**
+> **Status — 2026-08-28: the pack is in and proven. The shelf is not.**
 >
-> The APCRBC115 was fitted. **Steps 3 to 6 are all outstanding**, and step 3 is
-> the urgent one: `UpsSelfTestFailed` is still silenced until 2026-09-20, so
-> right now the only alert that could report a faulty or badly seated new pack
-> is suppressed — on a pack nobody has tested yet.
+> The APCRBC115 was fitted, the silence was deleted, and a self-test passed:
+> `upsTestResultsSummary` went `4` (aborted) to `1` (donePass) at 22:45 UTC.
+> Step 5's comparison holds — `upsBatteryVoltage` left its fabricated `480` for
+> a float reading that varies, and the runtime estimate no longer sits on
+> exactly `63`. Charge still reads exactly `100`, which is what a pack at float
+> voltage looks like rather than a card that cannot see one.
 >
-> The shelf and the switch move (step 2, items 1–4) were not done either, so
+> **Step 6 is outstanding.** Scheduled self-tests are not enabled on the NMC, so
+> `1` is a last-known result with nothing refreshing it, and nothing in
+> `ups.rules.yaml` detects a card that has quietly stopped testing.
+>
+> The shelf and the switch move (step 2, items 1–4) were not done, so
 > [#110](https://github.com/Gerrrt/HomeLab/issues/110) is untouched and
 > `prometheus` and `oracle` still go deaf on a mains cut.
 >
-> Step 1's baseline was not captured before the fit. It does not have to be
-> re-derived: the pre-fit values are recorded in the table under step 1 and in
-> the header of `ups.rules.yaml`, and comparing against those is what step 5
-> needs.
+> **One thing to do differently next time.** The silence was deleted at 23:14
+> UTC — *after* the 22:45 self-test, not before it. It cost nothing here because
+> the test passed, but for those 29 minutes a faulty pack would have been
+> reported into a suppressed alert. That inversion is the single failure mode
+> step 3 exists to prevent.
 
 `mjolnir` ran with no battery pack for the whole life of this stack. A mains
 loss was an immediate hard shutdown of the entire rack. The management card
@@ -121,7 +128,10 @@ the cases that genuinely need the wire, and reads the credential from SOPS.
 
 ## 3. Delete the silence — immediately, not on expiry
 
-`UpsSelfTestFailed` is silenced in Alertmanager until **2026-09-20**
+> **Done 2026-08-28**, at 23:14 UTC. The procedure below is kept for the next
+> time a pack is changed and a silence is standing over it.
+
+`UpsSelfTestFailed` was silenced in Alertmanager until **2026-09-20**
 (`54f1715c-e57b-4322-8a6d-5435bc8e1bd8`). It routes on `category=power` to the
 `urgent` receiver with `group_wait: 0s` and `repeat_interval: 30m`, so leaving
 it firing meant paging every half hour about a condition already known — which
@@ -241,17 +251,27 @@ actually true — a pack is fitted and unproven, with the outstanding silence an
 self-test named. That is `ups-power.json`, `ups.rules.yaml`, `docs/security.md`,
 `docs/observability.md`, `README.md`, `docs/roadmap.md` and `docs/hardware.md`.
 
-**Still to do, only once step 5 passes** — a separate commit made after
-verification, not before:
+**Done on 2026-08-28**, once the self-test passed: the same files moved from
+*fitted and unproven* to *proven*, with the two things that outlast the fix
+named in each — that history before 2026-08-28 is fabricated rather than
+measured, and that scheduled self-tests are still off. That is `ups-power.json`,
+`ups.rules.yaml`, `docs/security.md`, `docs/observability.md`, `README.md` and
+`docs/roadmap.md`.
+
+Two deliberate departures from the plan above. The banner panel was **rewritten
+rather than deleted**, because there is still something true and non-obvious for
+it to say and keeping it holds the dashboard at 84 panels, so the
+`scripts/check_docs.py` panel-count coupling stays untriggered. And **#93 was
+not moved into Done**, because this roadmap entry defines it as *delete the
+silence, then self-test, then enable scheduled tests* — the third is step 6, and
+it is outstanding.
+
+**Still to do:**
 
 | File | What changes |
 | --- | --- |
-| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "Battery fitted — not yet proven" banner panel; strip "(unproven — self-test pending)" from the five panel titles and reset their descriptions |
-| `stacks/observability/prometheus/rules/ups.rules.yaml` | Cut the header back to a short note. The rules themselves do not change |
-| `docs/security.md` | The "Mains power loss" row, and the closing paragraphs of "The UPS reported a battery it did not have" |
-| `docs/observability.md` | The `ups.rules.yaml` row of the rule-file table |
-| `README.md` | The UPS screenshot and the note under it, and the "current top items" paragraph. Re-capture with `make screenshots` |
-| `docs/roadmap.md` | Move #93 into Done. Existing entries stay as written; the roadmap is a record |
+| `docs/images/ups-power.png` | Re-capture the set with `make screenshots`. The committed image is from 2026-08-22 and shows the original no-battery banner; it needs the decrypted Grafana password, so it is a human step |
+| `docs/roadmap.md` | Move #93 into Done — but only once step 6 is done and the card is testing on a schedule |
 
 And when the shelf is racked, which is a separate visit and separate issue:
 

docs/security.md

@@ -15,7 +15,7 @@ What this network is actually built to survive:
 | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access |
 | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it |
 | Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs |
-| Mains power loss | **Partly defended.** A pack was fitted to `mjolnir` on 2026-08-28 and has not yet proven it can carry the load — see below |
+| Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below |
 
 What it explicitly does **not** defend against: a determined attacker with
 physical access to the rack, a supply-chain compromise in an upstream container
@@ -175,19 +175,25 @@ that file that can detect this condition.
 This is worth stating carefully: the monitoring did not fail, and neither did
 the rules. The device lied, and the rules trusted it.
 
-**A pack was fitted on 2026-08-28, and that has not yet closed this.** A card
-that cannot see a pack which *is* present — badly seated, or faulty out of the
-box — emits the same five fabricated values as one sitting over an empty bay. So
-a healthy-looking dashboard distinguishes nothing; only a passing self-test, and
-charge and runtime that have moved off the pre-fit baseline, do.
-
-Neither has happened yet. No self-test has been run since the fit, so
-`upsTestResultsSummary` still holds its pre-fit `4`. And `UpsSelfTestFailed` is
-still silenced in Alertmanager until 2026-09-20 — a silence that was correct
-while nothing could be done about a missing pack, and which now suppresses the
-one rule that would report the new pack being bad. Deleting it is step 3 of
-[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), it comes
-before the self-test rather than after, and it is outstanding.
+**A pack was fitted on 2026-08-28, and that closed this.** A card that cannot
+see a pack which *is* present — badly seated, or faulty out of the box — emits
+the same five fabricated values as one sitting over an empty bay, so a
+healthy-looking dashboard distinguished nothing. Only a passing self-test and
+readings that have left the pre-fit baseline do, and both now hold:
+`upsTestResultsSummary` went `4` (aborted) to `1` (donePass), `upsBatteryVoltage`
+left `480` for a float reading that varies, and the runtime estimate no longer
+sits on exactly `63`. The silence on `UpsSelfTestFailed` was deleted the same day
+rather than left to expire in September, so that rule is live again.
+
+Two things outlast the fix. Stored metrics older than 2026-08-28 *are* the
+fabricated values rather than measurements, so a dashboard or query whose range
+crosses that date is reading fiction on one side of it. And scheduled self-tests
+are not yet enabled on the card, which makes `1` a last-known result with
+nothing refreshing it — `UpsBatteryUnproven` cannot catch a card that stops
+testing, because it matches `6` (noTestsInitiated) and this one now reads `1`.
+That is step 6 of
+[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and it is
+outstanding.
 
 ### Why SNMPv2c is still a weak point