diff --git a/README.md b/README.md index ae1c135..a44942c 100644 --- a/README.md +++ b/README.md @@ -165,7 +165,7 @@ rack; a dashed border means egress only. Full topology and data flow in │ ├── adr/ # 11 architecture decision records │ └── runbooks/ # deploy, add device, rotate creds, certs, key backup, │ # purge, restore the firewall, ship firewall logs, -│ # enable suricata +│ # enable suricata, fit the UPS battery └── Makefile # make help ``` diff --git a/docs/observability.md b/docs/observability.md index 3245a7f..900726d 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -105,7 +105,7 @@ boot check. | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | -| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature | +| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. No pack is fitted, so most of these read values the management card fabricates — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | diff --git a/docs/roadmap.md b/docs/roadmap.md index 4e3d8bb..a5f8d16 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -99,6 +99,7 @@ what left this one unfireable for months. The management card fabricates a healthy pack. **The `UpsSelfTestFailed` silence must be deleted the moment the pack is fitted, not left to expire** — the issue carries the command and the reasoning. + → [runbook](runbooks/fit-the-ups-battery.md) - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.** A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and `oracle` hang off. **Buy it with the UPS pack above, not after** — both shelf @@ -106,6 +107,7 @@ what left this one unfireable for months. switch between them and the network has no battery at all. Spec settled at the rack on 2026-08-21; the spare ProDesk from [#92](https://github.com/Gerrrt/HomeLab/issues/92) racks here too, powered off. + → [runbook](runbooks/fit-the-ups-battery.md) - **[#94](https://github.com/Gerrrt/HomeLab/issues/94) Decide what `oracle` is for.** A dual-core A6-9200 with 4 GB and a 5400 rpm disk — too little for anything demanding, and a candidate for the jobs that need a machine that is diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md new file mode 100644 index 0000000..c6bd0c2 --- /dev/null +++ b/docs/runbooks/fit-the-ups-battery.md @@ -0,0 +1,231 @@ +# Runbook: Fit the UPS battery and rack the shelf switch + +**One rack visit, in an order that matters — and one alert that has to be +un-silenced by hand at the right moment.** + +`mjolnir` has no battery pack fitted. A mains loss is an immediate hard shutdown +of the entire rack, and `docs/security.md` lists it under "Not currently +defended". The management card does not report the missing pack; it fabricates a +healthy one, so every alert keyed on charge, runtime or alarm count is quietly +dead. Only `UpsSelfTestFailed` and `UpsBatteryUnproven` in `ups.rules.yaml` can +see the real condition, because they read the one value the card will not fake. + +The pack alone is not the whole fix. + +## Why the shelf is part of this, not a follow-up + +`prometheus` and `oracle` hang off an unmanaged TP-Link that has no battery at +all. Both are laptops with their own batteries, so on a mains cut they stay +*running* and go *deaf* — the monitoring host survives the event it exists to +observe and cannot report it. + +Fitting the pack protects the rack and does nothing for that switch. Racking the +switch in U4 and moving it onto UPS power is what closes the gap, which is why +[#93](https://github.com/Gerrrt/HomeLab/issues/93) and +[#110](https://github.com/Gerrrt/HomeLab/issues/110) were bought together and +are done together. + +## Before you start + +Two boxes, checked before either is opened: + +- **The cartridge is an APCRBC115.** Not a compatible-with listing that turns + out to be a different form factor once the front bezel is off. +- **The shelf matches the spec measured at the rack on 2026-08-21:** 4-post + frame, square mounting holes, a **full 1U shelf with rear support** rather + than a cantilever, and **cage nuts plus screws** — most shelves include them, + not all. + +Depth is not a constraint. U4 is empty only because nothing had been chosen for +it, so there is no service clearance to preserve. + +You will also need the stack up on `prometheus`, and access to the management +card's web UI for `mjolnir` at `10.0.99.10`. + +## 1. Record the baseline before touching anything + +The point of this step is that the after-state is *provably* different rather +than assumed to be. Capture what the card claims today: + +```bash +for m in upsBatteryStatus upsEstimatedChargeRemaining \ + upsEstimatedMinutesRemaining upsBatteryVoltage \ + upsAlarmsPresent upsTestResultsSummary; do + printf '%-32s ' "$m" + curl -sG http://localhost:9090/api/v1/query \ + --data-urlencode "query=${m}{device=\"mjolnir\"}" | + python3 -c 'import json,sys; r=json.load(sys.stdin)["data"]["result"]; print(r[0]["value"][1] if r else "no data")' +done +``` + +Expected on a UPS with no pack fitted — every one of these is derived, not +measured: + +| Metric | Reads | Means | +| --- | --- | --- | +| `upsBatteryStatus` | `2` | batteryNormal | +| `upsEstimatedChargeRemaining` | `100` | percent | +| `upsEstimatedMinutesRemaining` | `63` | minutes of runtime it does not have | +| `upsBatteryVoltage` | `480` | 48.0 V | +| `upsAlarmsPresent` | `0` | no alarm raised | +| `upsTestResultsSummary` | `4` | aborted — **the one honest value** | + +Write the numbers down. In step 5 you compare against them, and a charge still +reading exactly `100` with a runtime still at exactly `63` is the signature of a +card that is still fabricating because it does not see the new pack. + +Query Prometheus rather than reaching for `snmpget`: the community would sit on +your command line where `ps` can read it. `scripts/snmp-verify.sh` exists for +the cases that genuinely need the wire, and reads the credential from SOPS. + +## 2. At the rack, in this order + +1. **Rack the 1U vented shelf in U4.** Cage nuts front and rear, screws through + the shelf, rear support engaged. +2. **Move the TP-Link onto the shelf** and re-seat its uplink to port 3 of the + main switch, plus the runs to `prometheus` and `oracle`. +3. **Move the TP-Link's power onto a UPS-fed outlet.** This is the step that + closes #110. A switch that has been relocated onto a shelf but left on a wall + socket is tidier and no better protected — the laptops still go deaf on a + mains cut. +4. **Rack the cold-spare ProDesk from + [#92](https://github.com/Gerrrt/HomeLab/issues/92) beside it**, cabled for + its cold-spare role and **left powered off**. A spare that is plugged in and + on the network is exposed to whatever took the primary. +5. **Fit the APCRBC115 pack** last, per the Smart-UPS front-bezel procedure, and + confirm the card comes back with the battery-replacement date reset. + +## 3. Delete the silence — immediately, not on expiry + +`UpsSelfTestFailed` is silenced in Alertmanager until **2026-09-20** +(`54f1715c-e57b-4322-8a6d-5435bc8e1bd8`). It routes on `category=power` to the +`urgent` receiver with `group_wait: 0s` and `repeat_interval: 30m`, so leaving +it firing meant paging every half hour about a condition already known — which +is how an urgent receiver stops being read. The silence was correct while +nothing could be done about it. + +It stops being correct the moment the pack is in. Between fitting and 20 +September it would suppress `UpsSelfTestFailed` on a UPS that can finally report +honestly — **including a new pack that is faulty or badly seated**, which is +precisely the thing you have just introduced and most want to hear about. + +```bash +curl -sS -X DELETE \ + http://localhost:9093/api/v2/silence/54f1715c-e57b-4322-8a6d-5435bc8e1bd8 +``` + +Confirm it is gone rather than merely expired: + +```bash +curl -sS http://localhost:9093/api/v2/silences | + python3 -c 'import json,sys; [print(s["id"], s["status"]["state"]) for s in json.load(sys.stdin)]' +``` + +Delete it **before** the self-test, not after. Run in the other order, the one +result you most need to see is the one that is suppressed. + +## 4. Run one self-test, and read it from both ends + +From the management card's diagnostics page, start a single self-test. Then +confirm the result twice: + +- **On the NMC** — the page that read "Refused — internal fault" should now + report a pass. +- **In Prometheus** — because the NMC agreeing with itself proves nothing about + the scrape path: + + ```bash + curl -sG http://localhost:9090/api/v1/query \ + --data-urlencode 'query=upsTestResultsSummary{device="mjolnir"}' + ``` + +`upsTestResultsSummary` must read `1` (donePass). The full enumeration, from +`ups.rules.yaml`: + +| Value | Meaning | +| --- | --- | +| `1` | donePass | +| `2` | doneWarning | +| `3` | doneError | +| `4` | aborted | +| `5` | inProgress | +| `6` | noTestsInitiated | + +Anything in `2`–`4` means `UpsSelfTestFailed` should fire within 5 minutes, and +it now will, because you deleted the silence first. That is the system working. + +## 5. Confirm the fabricated metrics became real + +Re-run the loop from step 1 and compare against the baseline. Charge, runtime +and voltage should now reflect an actual pack — a partially charged new +cartridge typically reads *below* 100% for the first hours, which is itself +evidence the value is measured. + +If they are byte-for-byte identical to the baseline, the card is still +fabricating. Treat that as "the pack is not seen", not as "the pack is fine and +the numbers happen to match" — go back to seating and the battery-replacement +date before believing anything else on the dashboard. + +## 6. Enable scheduled self-tests + +A value of `1` is the *last* result, not a fresh one. If the NMC stops testing +entirely the metric simply stops changing, and nothing in `ups.rules.yaml` +detects that — `upsTestStartTime` is `sysUpTime`-relative and does not survive +an agent restart, so it is not a usable staleness signal. + +Scheduled tests on the device are the control. Enable them on the NMC so +`UpsBatteryUnproven` — which reads `min_over_time`/`max_over_time` over 7 days +rather than a `for:` that a Prometheus restart would reset — has live evidence +to read instead of an absence of it. The reasoning is argued in full in the +header of `ups.rules.yaml`; this step is where it gets acted on. + +Do not go on running self-tests by hand to keep the metric warm. A test +transfers the load to battery, and on a pack that is weak that is the outage the +alert exists to warn about. + +## 7. What becomes true afterwards + +`UpsBatteryLow`, `UpsChargeLow` and `UpsRuntimeCritical` become meaningful rules +rather than decorative ones, reading values that finally correspond to hardware. +**No rule edits are needed** — the file was written so that its rules become +correct the moment a pack exists, and nothing in it changes today. + +## If something goes wrong + +**The self-test still reports `4` (aborted).** Do not re-create the silence. A +still-aborted test with a pack fitted is a finding: check the cartridge is fully +seated and its connector latched, and reset the battery-replacement date on the +NMC so the card re-evaluates. If it persists, the alert firing every 30 minutes +is the correct state of the world until it is resolved. + +**The self-test reads `5` (inProgress) and stays there.** `UpsSelfTestFailed` is +bounded below `5` deliberately so a test in flight does not trip it. Wait for it +to settle before concluding anything. + +**`prometheus` or `oracle` came back unreachable after the shelf move.** The +uplink to port 3 of the main switch, not the UPS. Nothing in this runbook +touches VLAN configuration. + +## Only when all of the above passes — flip the documents + +Several files currently assert, correctly, that there is no battery. They become +wrong the moment this runbook succeeds, and they are a separate commit made +*after* verification, not before: + +| File | What changes | +| --- | --- | +| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "No battery is installed in this UPS" banner panel; strip "(fabricated — no battery fitted)" from the six panel titles and their descriptions | +| `stacks/observability/prometheus/rules/ups.rules.yaml` | Rewrite the header comment. The rules themselves do not change | +| `docs/security.md` | The "Mains power loss" row in the threat table, and the "The UPS reports a battery it does not have" section | +| `README.md` | The UPS dashboard screenshot alt text, and the "current top items" paragraph. Re-capture with `make screenshots` | +| `docs/hardware.md` | The 1U shelf into the rack table at U4, and into Accessories | +| `docs/network.md` | The TP-Link note — now racked and on UPS power | +| `docs/roadmap.md` | Move #93 and #110 into Done. Existing entries stay as written; the roadmap is a record | + +One coupling to know about before you start editing: **deleting the banner panel +changes the dashboard's panel count**, and `scripts/check_docs.py` asserts every +panel count quoted in prose against the live JSON. Move the numbers in the same +commit or CI will fail. Run `make validate` before pushing. + +Then close [#93](https://github.com/Gerrrt/HomeLab/issues/93) and +[#110](https://github.com/Gerrrt/HomeLab/issues/110).