docs(ups): record a pack that is fitted and not yet proven

README.md

@@ -221,6 +221,10 @@ draw and hardware health.](docs/images/network-snmp.png)
 and runtime, under a banner explaining that every battery figure is fabricated
 because no battery pack is fitted.](docs/images/ups-power.png)
 
+That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The
+banner it shows has been rewritten since; re-shoot with `make screenshots` once
+the self-test has passed and the panels read something worth showing.
+
 ## What runs it
 
 The entire observability stack runs on a 2012 MacBook Pro with Ubuntu Server on
@@ -251,8 +255,10 @@ Open work is tracked in
 [`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and
 why it is in that order.
 
-The current top items: replace the UPS battery and rack the shelf switch, which
-are one purchase — the pack alone leaves the monitoring path half-protected
+The current top items: prove the UPS battery and rack the shelf switch. The pack
+went in on 2026-08-28 and has not yet passed a self-test, so the rack is not
+demonstrably protected yet; the switch between the monitoring host and the
+network still has no battery at all
 ([#93](https://github.com/Gerrrt/HomeLab/issues/93),
 [#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup
 off the machine it protects, and buy the spare that turns its restore runbook

docs/hardware.md

@@ -55,6 +55,7 @@ revisions of this repository treated `shiva` as the hypervisor itself.
 - 1U rackmount tray for the ProDesk Mini[^ProDeskRackmount]
 - Sliding rails for the ProLiant[^Sliderail]
 - 1U universal rack mount for the APC[^Rail]
+- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28
 - ViewSonic N1700W LCD, used as a rack console via the KVM
 - RJ45 Cat6 in-line couplers[^Couplers]
 - Cat6 patch cables[^Patchcables]

docs/images/README.md

@@ -25,6 +25,15 @@ carries a `renderer` series in two legends: that is the `capture` profile
 container taking the screenshot, which exists only for the duration of a
 capture, which is why the inventory below it lists six containers and not seven.
 
+`ups-power.png` is out of date in one specific way, and knowingly so. It was
+shot before a battery pack was fitted to `mjolnir` on 2026-08-28, so it still
+shows the old "No battery is installed in this UPS" banner and the
+"(fabricated — no battery fitted)" panel titles. It stays in place rather than
+being deleted — it is a real render of what the dashboard said that day, and the
+root `README.md` says underneath it when it was taken and what has changed
+since. Re-shoot it once the pack has passed a self-test, when the panels will
+read measured values instead of unproven ones.
+
 The Container inventory panel used to publish the absolute path of
 `compose.yaml` — and so a username — because it excluded fields by name and
 cAdvisor kept adding new ones. It now filters to an allowlist. That was caught

docs/observability.md

@@ -105,7 +105,7 @@ boot check.
 | --- | --- |
 | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots |
 | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware |
-| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. No pack is fitted, so most of these read values the management card fabricates — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
+| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 but has not yet passed a self-test, so most of these still read values the card may be fabricating — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
 | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself |
 | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable |
 

docs/roadmap.md

@@ -95,10 +95,12 @@ what left this one unfireable for months.
   [`restore-the-firewall.md`](runbooks/restore-the-firewall.md) stays a
   hypothesis until it has been restored onto a spare once.
 - **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.**
-  `mjolnir` has none, so a mains loss is an immediate hard shutdown of the rack.
-  The management card fabricates a healthy pack. **The `UpsSelfTestFailed`
-  silence must be deleted the moment the pack is fitted, not left to expire** —
-  the issue carries the command and the reasoning.
+  An APCRBC115 went into `mjolnir` on 2026-08-28. That is the hardware done, not
+  the issue: the management card reports a healthy pack whether or not it can see
+  one, so nothing yet separates a good fit from a bad one. **The `UpsSelfTestFailed`
+  silence is still in place until 2026-09-20, and now suppresses the only alert
+  that could tell you which this is** — delete it, then self-test, then enable
+  scheduled tests on the card.
   → [runbook](runbooks/fit-the-ups-battery.md)
 - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.**
   A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and

docs/runbooks/fit-the-ups-battery.md

@@ -3,12 +3,32 @@
 **One rack visit, in an order that matters — and one alert that has to be
 un-silenced by hand at the right moment.**
 
-`mjolnir` has no battery pack fitted. A mains loss is an immediate hard shutdown
-of the entire rack, and `docs/security.md` lists it under "Not currently
-defended". The management card does not report the missing pack; it fabricates a
-healthy one, so every alert keyed on charge, runtime or alarm count is quietly
-dead. Only `UpsSelfTestFailed` and `UpsBatteryUnproven` in `ups.rules.yaml` can
-see the real condition, because they read the one value the card will not fake.
+> **Status — 2026-08-28: the pack is in, and nothing else is done.**
+>
+> The APCRBC115 was fitted. **Steps 3 to 6 are all outstanding**, and step 3 is
+> the urgent one: `UpsSelfTestFailed` is still silenced until 2026-09-20, so
+> right now the only alert that could report a faulty or badly seated new pack
+> is suppressed — on a pack nobody has tested yet.
+>
+> The shelf and the switch move (step 2, items 1–4) were not done either, so
+> [#110](https://github.com/Gerrrt/HomeLab/issues/110) is untouched and
+> `prometheus` and `oracle` still go deaf on a mains cut.
+>
+> Step 1's baseline was not captured before the fit. It does not have to be
+> re-derived: the pre-fit values are recorded in the table under step 1 and in
+> the header of `ups.rules.yaml`, and comparing against those is what step 5
+> needs.
+
+`mjolnir` ran with no battery pack for the whole life of this stack. A mains
+loss was an immediate hard shutdown of the entire rack. The management card
+never reported the missing pack; it fabricated a healthy one, so every alert
+keyed on charge, runtime or alarm count was quietly dead. Only
+`UpsSelfTestFailed` and `UpsBatteryUnproven` in `ups.rules.yaml` can see the
+real condition, because they read the one value the card will not fake.
+
+Fitting a pack does not by itself end any of that. A card that cannot see a pack
+which *is* present reports the same fabricated values as one over an empty bay,
+so the steps below are what turns a fitted battery into a proven one.
 
 The pack alone is not the whole fix.
 
@@ -45,7 +65,11 @@ card's web UI for `mjolnir` at `10.0.99.10`.
 ## 1. Record the baseline before touching anything
 
 The point of this step is that the after-state is *provably* different rather
-than assumed to be. Capture what the card claims today:
+than assumed to be. If the pack is already in and this was not run first, use
+the recorded pre-fit values in the table below as the baseline — they are the
+observed readings from this UPS, and `ups.rules.yaml` carries the same five.
+
+Capture what the card claims:
 
 ```bash
 for m in upsBatteryStatus upsEstimatedChargeRemaining \
@@ -58,8 +82,8 @@ for m in upsBatteryStatus upsEstimatedChargeRemaining \
 done
 ```
 
-Expected on a UPS with no pack fitted — every one of these is derived, not
-measured:
+Observed on this UPS with no pack fitted — every one of these was derived,
+not measured:
 
 | Metric | Reads | Means |
 | --- | --- | --- |
@@ -206,26 +230,44 @@ to settle before concluding anything.
 uplink to port 3 of the main switch, not the UPS. Nothing in this runbook
 touches VLAN configuration.
 
-## Only when all of the above passes — flip the documents
+## Flipping the documents — the first half is done, the second is not
+
+This happens in two passes, because "a pack is fitted" and "the UPS works" are
+different claims and became true at different times.
+
+**Done on 2026-08-28**, when the pack went in: every file that asserted there is
+no battery said something false from that moment, so they were moved to what was
+actually true — a pack is fitted and unproven, with the outstanding silence and
+self-test named. That is `ups-power.json`, `ups.rules.yaml`, `docs/security.md`,
+`docs/observability.md`, `README.md`, `docs/roadmap.md` and `docs/hardware.md`.
+
+**Still to do, only once step 5 passes** — a separate commit made after
+verification, not before:
 
-Several files currently assert, correctly, that there is no battery. They become
-wrong the moment this runbook succeeds, and they are a separate commit made
-*after* verification, not before:
+| File | What changes |
+| --- | --- |
+| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "Battery fitted — not yet proven" banner panel; strip "(unproven — self-test pending)" from the five panel titles and reset their descriptions |
+| `stacks/observability/prometheus/rules/ups.rules.yaml` | Cut the header back to a short note. The rules themselves do not change |
+| `docs/security.md` | The "Mains power loss" row, and the closing paragraphs of "The UPS reported a battery it did not have" |
+| `docs/observability.md` | The `ups.rules.yaml` row of the rule-file table |
+| `README.md` | The UPS screenshot and the note under it, and the "current top items" paragraph. Re-capture with `make screenshots` |
+| `docs/roadmap.md` | Move #93 into Done. Existing entries stay as written; the roadmap is a record |
+
+And when the shelf is racked, which is a separate visit and separate issue:
 
 | File | What changes |
 | --- | --- |
-| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "No battery is installed in this UPS" banner panel; strip "(fabricated — no battery fitted)" from the six panel titles and their descriptions |
-| `stacks/observability/prometheus/rules/ups.rules.yaml` | Rewrite the header comment. The rules themselves do not change |
-| `docs/security.md` | The "Mains power loss" row in the threat table, and the "The UPS reports a battery it does not have" section |
-| `README.md` | The UPS dashboard screenshot alt text, and the "current top items" paragraph. Re-capture with `make screenshots` |
 | `docs/hardware.md` | The 1U shelf into the rack table at U4, and into Accessories |
 | `docs/network.md` | The TP-Link note — now racked and on UPS power |
-| `docs/roadmap.md` | Move #93 and #110 into Done. Existing entries stay as written; the roadmap is a record |
+| `docs/roadmap.md` | Move #110 into Done |
 
 One coupling to know about before you start editing: **deleting the banner panel
 changes the dashboard's panel count**, and `scripts/check_docs.py` asserts every
-panel count quoted in prose against the live JSON. Move the numbers in the same
-commit or CI will fail. Run `make validate` before pushing.
-
-Then close [#93](https://github.com/Gerrrt/HomeLab/issues/93) and
-[#110](https://github.com/Gerrrt/HomeLab/issues/110).
+panel count quoted in prose against the live JSON. The banner was retitled
+rather than removed on 2026-08-28 for exactly that reason — the count did not
+move. When you do delete it, 84 becomes 83 in both `README.md` and
+`stacks/observability/README.md`, in the same commit, or CI will fail. Run
+`make validate` before pushing.
+
+Then close [#93](https://github.com/Gerrrt/HomeLab/issues/93), and
+[#110](https://github.com/Gerrrt/HomeLab/issues/110) when the shelf follows.

docs/security.md

@@ -15,7 +15,7 @@ What this network is actually built to survive:
 | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access |
 | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it |
 | Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs |
-| Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below |
+| Mains power loss | **Partly defended.** A pack was fitted to `mjolnir` on 2026-08-28 and has not yet proven it can carry the load — see below |
 
 What it explicitly does **not** defend against: a determined attacker with
 physical access to the rack, a supply-chain compromise in an upstream container
@@ -155,26 +155,40 @@ strings at mode 664 in `~/.local/state/nvim/undodir/` were a real finding
 *because* the disk beneath them is readable. On an encrypted disk that is a much
 smaller problem. Neither fact is interesting alone.
 
-### The UPS reports a battery it does not have
+### The UPS reported a battery it did not have
 
-`mjolnir` has no battery installed. Its Network Management Card nonetheless
-reports 100% state of charge, 48.0 VDC, a battery temperature, an hour of
-runtime, a 2030 replacement date, and `upsAlarmsPresent = 0`. Every one of those
-values is derived rather than measured.
+Until 2026-08-28 `mjolnir` had no battery installed. Its Network Management Card
+nonetheless reported 100% state of charge, 48.0 VDC, a battery temperature, an
+hour of runtime, a 2030 replacement date, and `upsAlarmsPresent = 0`. Every one
+of those values was derived rather than measured.
 
 The single honest signal it emits is the self-test result. The management card
-renders it as **Refused — internal fault**; over SNMP it is
+rendered it as **Refused — internal fault**; over SNMP it is
 `upsTestResultsSummary = 4` (aborted), from the standard UPS-MIB the `apc_ups`
 module already walks. No extra OIDs were needed to see it.
 
-Any alert rule keyed on charge, runtime or alarm count will therefore never
-fire, no matter how bad things get. `UpsSelfTestFailed` and `UpsBatteryUnproven`
-in `ups.rules.yaml` key on the self-test instead, and are the only two rules in
+Any alert rule keyed on charge, runtime or alarm count therefore could not fire,
+no matter how bad things got. `UpsSelfTestFailed` and `UpsBatteryUnproven` in
+`ups.rules.yaml` key on the self-test instead, and are the only two rules in
 that file that can detect this condition.
 
 This is worth stating carefully: the monitoring did not fail, and neither did
 the rules. The device lied, and the rules trusted it.
 
+**A pack was fitted on 2026-08-28, and that has not yet closed this.** A card
+that cannot see a pack which *is* present — badly seated, or faulty out of the
+box — emits the same five fabricated values as one sitting over an empty bay. So
+a healthy-looking dashboard distinguishes nothing; only a passing self-test, and
+charge and runtime that have moved off the pre-fit baseline, do.
+
+Neither has happened yet. No self-test has been run since the fit, so
+`upsTestResultsSummary` still holds its pre-fit `4`. And `UpsSelfTestFailed` is
+still silenced in Alertmanager until 2026-09-20 — a silence that was correct
+while nothing could be done about a missing pack, and which now suppresses the
+one rule that would report the new pack being bad. Deleting it is step 3 of
+[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), it comes
+before the self-test rather than after, and it is outstanding.
+
 ### Why SNMPv2c is still a weak point
 
 The devices are polled with SNMPv2c, which transmits the community string in