From 109f9b8723d094945c58ebb7d34ebca6b5de14f8 Mon Sep 17 00:00:00 2001 From: Claude Date: Fri, 28 Aug 2026 22:58:18 +0000 Subject: [PATCH] docs(ups): record a pack that is fitted and not yet proven MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The APCRBC115 went into `mjolnir` today. Every file that asserted there is no battery became false at that moment, so they move now rather than waiting — but they move to *fitted and unproven*, not to *working*, because nothing has yet demonstrated the second. The distinction is the whole point of the previous commit's runbook. A management card that cannot see a pack which is present — badly seated, or faulty out of the box — emits the same five fabricated values as one sitting over an empty bay: `batteryNormal`, 100%, 63 minutes, 48.0 V, no alarm. A healthy-looking dashboard therefore separates nothing. Only a passing self-test and a charge that has moved off the pre-fit baseline do, and neither has been done. What is outstanding is also what is dangerous. The `UpsSelfTestFailed` silence 54f1715c-e57b-4322-8a6d-5435bc8e1bd8 runs until 2026-09-20 and is now suppressing the one rule that could report a bad new pack — the runbook predicted exactly this window, and we are in it. Deleting that silence cannot be done from a commit, so every file that touches the subject names it instead: - the dashboard banner is retitled rather than deleted, so it now says a pack is fitted and untested and carries the two remaining steps. Keeping the panel means the count stays 84 and the check_docs.py coupling the runbook warned about is not triggered by this commit - the five "(fabricated — no battery fitted)" panel titles become "(unproven — self-test pending)", which is what they now are - `ups.rules.yaml` keeps its long header. The pre-fit readings stay because they are the baseline step 5 compares against, and step 1 was never run live; the rules themselves are untouched, as designed - `security.md` keeps the finding and its lesson, and stops claiming the bay is empty. "Mains power loss" goes from not defended to partly defended, which is the honest reading of an untested pack The screenshot is left as it is: it is a real render of what the dashboard said on 2026-08-22, and `docs/images/README.md` refuses mocked-up images. Both READMEs say what it predates and when to re-shoot it. #110 is untouched — the shelf and the switch move did not happen, so the monitoring path still has no battery behind it. The runbook's flip list is split accordingly: done now, done after the self-test, done after the shelf. Refs #93, #110 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_015ipPWFSYkEX3MtP226HzR1 --- README.md | 10 ++- docs/hardware.md | 1 + docs/images/README.md | 9 ++ docs/observability.md | 2 +- docs/roadmap.md | 10 ++- docs/runbooks/fit-the-ups-battery.md | 88 ++++++++++++++----- docs/security.md | 34 ++++--- .../grafana/dashboards/ups-power.json | 26 +++--- .../prometheus/rules/ups.rules.yaml | 75 ++++++++++------ 9 files changed, 175 insertions(+), 80 deletions(-) diff --git a/README.md b/README.md index a44942c..48b012c 100644 --- a/README.md +++ b/README.md @@ -221,6 +221,10 @@ draw and hardware health.](docs/images/network-snmp.png) and runtime, under a banner explaining that every battery figure is fabricated because no battery pack is fitted.](docs/images/ups-power.png) +That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The +banner it shows has been rewritten since; re-shoot with `make screenshots` once +the self-test has passed and the panels read something worth showing. + ## What runs it The entire observability stack runs on a 2012 MacBook Pro with Ubuntu Server on @@ -251,8 +255,10 @@ Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and why it is in that order. -The current top items: replace the UPS battery and rack the shelf switch, which -are one purchase — the pack alone leaves the monitoring path half-protected +The current top items: prove the UPS battery and rack the shelf switch. The pack +went in on 2026-08-28 and has not yet passed a self-test, so the rack is not +demonstrably protected yet; the switch between the monitoring host and the +network still has no battery at all ([#93](https://github.com/Gerrrt/HomeLab/issues/93), [#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup off the machine it protects, and buy the spare that turns its restore runbook diff --git a/docs/hardware.md b/docs/hardware.md index cbc84ce..fe40d38 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -55,6 +55,7 @@ revisions of this repository treated `shiva` as the hypervisor itself. - 1U rackmount tray for the ProDesk Mini[^ProDeskRackmount] - Sliding rails for the ProLiant[^Sliderail] - 1U universal rack mount for the APC[^Rail] +- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28 - ViewSonic N1700W LCD, used as a rack console via the KVM - RJ45 Cat6 in-line couplers[^Couplers] - Cat6 patch cables[^Patchcables] diff --git a/docs/images/README.md b/docs/images/README.md index 23805bf..e562603 100644 --- a/docs/images/README.md +++ b/docs/images/README.md @@ -25,6 +25,15 @@ carries a `renderer` series in two legends: that is the `capture` profile container taking the screenshot, which exists only for the duration of a capture, which is why the inventory below it lists six containers and not seven. +`ups-power.png` is out of date in one specific way, and knowingly so. It was +shot before a battery pack was fitted to `mjolnir` on 2026-08-28, so it still +shows the old "No battery is installed in this UPS" banner and the +"(fabricated — no battery fitted)" panel titles. It stays in place rather than +being deleted — it is a real render of what the dashboard said that day, and the +root `README.md` says underneath it when it was taken and what has changed +since. Re-shoot it once the pack has passed a self-test, when the panels will +read measured values instead of unproven ones. + The Container inventory panel used to publish the absolute path of `compose.yaml` — and so a username — because it excluded fields by name and cAdvisor kept adding new ones. It now filters to an allowlist. That was caught diff --git a/docs/observability.md b/docs/observability.md index 900726d..65ee792 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -105,7 +105,7 @@ boot check. | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | -| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. No pack is fitted, so most of these read values the management card fabricates — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | +| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 but has not yet passed a self-test, so most of these still read values the card may be fabricating — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | diff --git a/docs/roadmap.md b/docs/roadmap.md index a5f8d16..730a85d 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -95,10 +95,12 @@ what left this one unfireable for months. [`restore-the-firewall.md`](runbooks/restore-the-firewall.md) stays a hypothesis until it has been restored onto a spare once. - **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.** - `mjolnir` has none, so a mains loss is an immediate hard shutdown of the rack. - The management card fabricates a healthy pack. **The `UpsSelfTestFailed` - silence must be deleted the moment the pack is fitted, not left to expire** — - the issue carries the command and the reasoning. + An APCRBC115 went into `mjolnir` on 2026-08-28. That is the hardware done, not + the issue: the management card reports a healthy pack whether or not it can see + one, so nothing yet separates a good fit from a bad one. **The `UpsSelfTestFailed` + silence is still in place until 2026-09-20, and now suppresses the only alert + that could tell you which this is** — delete it, then self-test, then enable + scheduled tests on the card. → [runbook](runbooks/fit-the-ups-battery.md) - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.** A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index c6bd0c2..e27ca0b 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -3,12 +3,32 @@ **One rack visit, in an order that matters — and one alert that has to be un-silenced by hand at the right moment.** -`mjolnir` has no battery pack fitted. A mains loss is an immediate hard shutdown -of the entire rack, and `docs/security.md` lists it under "Not currently -defended". The management card does not report the missing pack; it fabricates a -healthy one, so every alert keyed on charge, runtime or alarm count is quietly -dead. Only `UpsSelfTestFailed` and `UpsBatteryUnproven` in `ups.rules.yaml` can -see the real condition, because they read the one value the card will not fake. +> **Status — 2026-08-28: the pack is in, and nothing else is done.** +> +> The APCRBC115 was fitted. **Steps 3 to 6 are all outstanding**, and step 3 is +> the urgent one: `UpsSelfTestFailed` is still silenced until 2026-09-20, so +> right now the only alert that could report a faulty or badly seated new pack +> is suppressed — on a pack nobody has tested yet. +> +> The shelf and the switch move (step 2, items 1–4) were not done either, so +> [#110](https://github.com/Gerrrt/HomeLab/issues/110) is untouched and +> `prometheus` and `oracle` still go deaf on a mains cut. +> +> Step 1's baseline was not captured before the fit. It does not have to be +> re-derived: the pre-fit values are recorded in the table under step 1 and in +> the header of `ups.rules.yaml`, and comparing against those is what step 5 +> needs. + +`mjolnir` ran with no battery pack for the whole life of this stack. A mains +loss was an immediate hard shutdown of the entire rack. The management card +never reported the missing pack; it fabricated a healthy one, so every alert +keyed on charge, runtime or alarm count was quietly dead. Only +`UpsSelfTestFailed` and `UpsBatteryUnproven` in `ups.rules.yaml` can see the +real condition, because they read the one value the card will not fake. + +Fitting a pack does not by itself end any of that. A card that cannot see a pack +which *is* present reports the same fabricated values as one over an empty bay, +so the steps below are what turns a fitted battery into a proven one. The pack alone is not the whole fix. @@ -45,7 +65,11 @@ card's web UI for `mjolnir` at `10.0.99.10`. ## 1. Record the baseline before touching anything The point of this step is that the after-state is *provably* different rather -than assumed to be. Capture what the card claims today: +than assumed to be. If the pack is already in and this was not run first, use +the recorded pre-fit values in the table below as the baseline — they are the +observed readings from this UPS, and `ups.rules.yaml` carries the same five. + +Capture what the card claims: ```bash for m in upsBatteryStatus upsEstimatedChargeRemaining \ @@ -58,8 +82,8 @@ for m in upsBatteryStatus upsEstimatedChargeRemaining \ done ``` -Expected on a UPS with no pack fitted — every one of these is derived, not -measured: +Observed on this UPS with no pack fitted — every one of these was derived, +not measured: | Metric | Reads | Means | | --- | --- | --- | @@ -206,26 +230,44 @@ to settle before concluding anything. uplink to port 3 of the main switch, not the UPS. Nothing in this runbook touches VLAN configuration. -## Only when all of the above passes — flip the documents +## Flipping the documents — the first half is done, the second is not + +This happens in two passes, because "a pack is fitted" and "the UPS works" are +different claims and became true at different times. + +**Done on 2026-08-28**, when the pack went in: every file that asserted there is +no battery said something false from that moment, so they were moved to what was +actually true — a pack is fitted and unproven, with the outstanding silence and +self-test named. That is `ups-power.json`, `ups.rules.yaml`, `docs/security.md`, +`docs/observability.md`, `README.md`, `docs/roadmap.md` and `docs/hardware.md`. + +**Still to do, only once step 5 passes** — a separate commit made after +verification, not before: -Several files currently assert, correctly, that there is no battery. They become -wrong the moment this runbook succeeds, and they are a separate commit made -*after* verification, not before: +| File | What changes | +| --- | --- | +| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "Battery fitted — not yet proven" banner panel; strip "(unproven — self-test pending)" from the five panel titles and reset their descriptions | +| `stacks/observability/prometheus/rules/ups.rules.yaml` | Cut the header back to a short note. The rules themselves do not change | +| `docs/security.md` | The "Mains power loss" row, and the closing paragraphs of "The UPS reported a battery it did not have" | +| `docs/observability.md` | The `ups.rules.yaml` row of the rule-file table | +| `README.md` | The UPS screenshot and the note under it, and the "current top items" paragraph. Re-capture with `make screenshots` | +| `docs/roadmap.md` | Move #93 into Done. Existing entries stay as written; the roadmap is a record | + +And when the shelf is racked, which is a separate visit and separate issue: | File | What changes | | --- | --- | -| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "No battery is installed in this UPS" banner panel; strip "(fabricated — no battery fitted)" from the six panel titles and their descriptions | -| `stacks/observability/prometheus/rules/ups.rules.yaml` | Rewrite the header comment. The rules themselves do not change | -| `docs/security.md` | The "Mains power loss" row in the threat table, and the "The UPS reports a battery it does not have" section | -| `README.md` | The UPS dashboard screenshot alt text, and the "current top items" paragraph. Re-capture with `make screenshots` | | `docs/hardware.md` | The 1U shelf into the rack table at U4, and into Accessories | | `docs/network.md` | The TP-Link note — now racked and on UPS power | -| `docs/roadmap.md` | Move #93 and #110 into Done. Existing entries stay as written; the roadmap is a record | +| `docs/roadmap.md` | Move #110 into Done | One coupling to know about before you start editing: **deleting the banner panel changes the dashboard's panel count**, and `scripts/check_docs.py` asserts every -panel count quoted in prose against the live JSON. Move the numbers in the same -commit or CI will fail. Run `make validate` before pushing. - -Then close [#93](https://github.com/Gerrrt/HomeLab/issues/93) and -[#110](https://github.com/Gerrrt/HomeLab/issues/110). +panel count quoted in prose against the live JSON. The banner was retitled +rather than removed on 2026-08-28 for exactly that reason — the count did not +move. When you do delete it, 84 becomes 83 in both `README.md` and +`stacks/observability/README.md`, in the same commit, or CI will fail. Run +`make validate` before pushing. + +Then close [#93](https://github.com/Gerrrt/HomeLab/issues/93), and +[#110](https://github.com/Gerrrt/HomeLab/issues/110) when the shelf follows. diff --git a/docs/security.md b/docs/security.md index 7ea8885..5c65e23 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs | -| Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below | +| Mains power loss | **Partly defended.** A pack was fitted to `mjolnir` on 2026-08-28 and has not yet proven it can carry the load — see below | What it explicitly does **not** defend against: a determined attacker with physical access to the rack, a supply-chain compromise in an upstream container @@ -155,26 +155,40 @@ strings at mode 664 in `~/.local/state/nvim/undodir/` were a real finding *because* the disk beneath them is readable. On an encrypted disk that is a much smaller problem. Neither fact is interesting alone. -### The UPS reports a battery it does not have +### The UPS reported a battery it did not have -`mjolnir` has no battery installed. Its Network Management Card nonetheless -reports 100% state of charge, 48.0 VDC, a battery temperature, an hour of -runtime, a 2030 replacement date, and `upsAlarmsPresent = 0`. Every one of those -values is derived rather than measured. +Until 2026-08-28 `mjolnir` had no battery installed. Its Network Management Card +nonetheless reported 100% state of charge, 48.0 VDC, a battery temperature, an +hour of runtime, a 2030 replacement date, and `upsAlarmsPresent = 0`. Every one +of those values was derived rather than measured. The single honest signal it emits is the self-test result. The management card -renders it as **Refused — internal fault**; over SNMP it is +rendered it as **Refused — internal fault**; over SNMP it is `upsTestResultsSummary = 4` (aborted), from the standard UPS-MIB the `apc_ups` module already walks. No extra OIDs were needed to see it. -Any alert rule keyed on charge, runtime or alarm count will therefore never -fire, no matter how bad things get. `UpsSelfTestFailed` and `UpsBatteryUnproven` -in `ups.rules.yaml` key on the self-test instead, and are the only two rules in +Any alert rule keyed on charge, runtime or alarm count therefore could not fire, +no matter how bad things got. `UpsSelfTestFailed` and `UpsBatteryUnproven` in +`ups.rules.yaml` key on the self-test instead, and are the only two rules in that file that can detect this condition. This is worth stating carefully: the monitoring did not fail, and neither did the rules. The device lied, and the rules trusted it. +**A pack was fitted on 2026-08-28, and that has not yet closed this.** A card +that cannot see a pack which *is* present — badly seated, or faulty out of the +box — emits the same five fabricated values as one sitting over an empty bay. So +a healthy-looking dashboard distinguishes nothing; only a passing self-test, and +charge and runtime that have moved off the pre-fit baseline, do. + +Neither has happened yet. No self-test has been run since the fit, so +`upsTestResultsSummary` still holds its pre-fit `4`. And `UpsSelfTestFailed` is +still silenced in Alertmanager until 2026-09-20 — a silence that was correct +while nothing could be done about a missing pack, and which now suppresses the +one rule that would report the new pack being bad. Deleting it is step 3 of +[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), it comes +before the self-test rather than after, and it is outstanding. + ### Why SNMPv2c is still a weak point The devices are polled with SNMPv2c, which transmits the community string in diff --git a/stacks/observability/grafana/dashboards/ups-power.json b/stacks/observability/grafana/dashboards/ups-power.json index 81c0ffe..f83df1e 100644 --- a/stacks/observability/grafana/dashboards/ups-power.json +++ b/stacks/observability/grafana/dashboards/ups-power.json @@ -1,7 +1,7 @@ { "uid": "homelab-ups", "title": "UPS & Power", - "description": "APC Smart-UPS (mjolnir) via UPS-MIB. Note: this UPS currently has no working battery, so runtime figures are not to be trusted until it is replaced — see docs/roadmap.md.", + "description": "APC Smart-UPS (mjolnir) via UPS-MIB. A replacement pack was fitted on 2026-08-28 and has not yet passed a self-test, so runtime figures are not proven — see docs/runbooks/fit-the-ups-battery.md.", "tags": [ "homelab", "power", @@ -258,7 +258,7 @@ }, { "type": "text", - "title": "No battery is installed in this UPS", + "title": "Battery fitted 2026-08-28 — not yet proven", "gridPos": { "x": 0, "y": 6, @@ -267,7 +267,7 @@ }, "options": { "mode": "markdown", - "content": "**Every battery figure on this dashboard is fabricated.** `mjolnir` has no battery pack fitted, yet its management card still reports `batteryNormal`, 100% charge, 48.0 V and a runtime estimate. It raises no alarm (`upsAlarmsPresent = 0`).\n\nA mains loss is therefore an **immediate hard shutdown of the rack**, with no warning from here. Treat charge, runtime, battery status and battery temperature below as decoration until a pack is fitted — every alert in `ups.rules.yaml` is dead for the same reason.\n\nDo **not** run a UPS self-test to get an honest reading: a self-test transfers the load to battery, and with none fitted that is the outage it is meant to warn about. Tracked in `docs/roadmap.md`." + "content": "**A pack is fitted. Nothing has yet confirmed the UPS can carry load.** An APCRBC115 cartridge went into `mjolnir` on 2026-08-28. Until a self-test passes, every battery figure below is unproven rather than measured: a management card that cannot see a fitted pack reports `batteryNormal`, 100% charge, 48.0 V and a runtime estimate exactly as it did when the bay was empty.\n\nThe test is comparison, not appearance. Charge still reading exactly `100` and runtime still at exactly `63` is the signature of a card that is still fabricating — compare against the pre-fit baseline in step 5 of `docs/runbooks/fit-the-ups-battery.md`.\n\n**Two steps are outstanding.** The `UpsSelfTestFailed` silence `54f1715c` runs until 2026-09-20 and is suppressing the one alert that can report a faulty or badly seated new pack — delete it *before* running the self-test, not after. Then enable scheduled self-tests on the card, so `upsTestResultsSummary` stays live evidence rather than a stale last result." } }, { @@ -361,8 +361,8 @@ }, { "type": "gauge", - "title": "Battery charge (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Battery charge (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -497,8 +497,8 @@ }, { "type": "stat", - "title": "Runtime remaining (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Runtime remaining (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -567,8 +567,8 @@ }, { "type": "stat", - "title": "Battery status (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Battery status (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -791,8 +791,8 @@ }, { "type": "stat", - "title": "Battery temp (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Battery temp (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -873,8 +873,8 @@ }, { "type": "timeseries", - "title": "Battery charge and runtime (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Battery charge and runtime (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index c4dd572..ed9acc7 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -3,53 +3,73 @@ # # READ THIS BEFORE TRUSTING ANY ALERT BELOW. # -# The UPS has no battery pack fitted — see docs/roadmap.md. A mains loss is an -# immediate hard shutdown of the whole rack. Every rule here is deliberately -# loud and routed to the `urgent` receiver by category=power in +# An APCRBC115 pack was fitted on 2026-08-28. Until then the UPS had none and a +# mains loss was an immediate hard shutdown of the whole rack. Every rule here +# is deliberately loud and routed to the `urgent` receiver by category=power in # alertmanager.yaml. # -# None of them can fire for that condition. The management card does not report -# the missing pack; it fabricates a healthy one. Observed on a UPS with no -# battery physically present: +# A fitted pack is not yet a proven one, and here that distinction is not +# pedantry. The management card never reported the missing pack; it fabricated +# a healthy one. Observed on this UPS with no battery physically present: # # upsBatteryStatus 2 (batteryNormal) # upsEstimatedChargeRemaining 100 (percent) -# upsEstimatedMinutesRemaining 63 (minutes of runtime it does not have) +# upsEstimatedMinutesRemaining 63 (minutes of runtime it did not have) # upsBatteryVoltage 480 (48.0 V) # upsAlarmsPresent 0 (no alarm raised) # -# So UpsBatteryLow, UpsChargeLow, UpsRuntimeCritical and UpsAlarmPresent are -# all comfortably inside their thresholds and will stay there. These rules are -# kept because they become correct the moment a pack is fitted — not because -# they are protecting anything today. A quiet `ups` group is not evidence of -# runtime. +# A card that cannot see a pack which *is* fitted — one badly seated, or dead on +# arrival — reports those same five values. So the numbers looking healthy is +# not evidence; only their having moved off that pre-fit baseline is. Step 5 of +# docs/runbooks/fit-the-ups-battery.md is that comparison, and it has not been +# made yet. +# +# Until it has, UpsBatteryLow, UpsChargeLow, UpsRuntimeCritical and +# UpsAlarmPresent sit comfortably inside their thresholds and still tell you +# nothing. They are kept because they become correct the moment the card is +# reading real hardware — not because they are protecting anything yet. A quiet +# `ups` group is not evidence of runtime. # # UpsOnBattery, UpsOverloaded and UpsBatteryTemperatureHigh read mains-side or -# load-side values and are unaffected. UpsOnBattery in particular would still -# fire on a transfer — it would just be reporting an outage already in progress -# rather than warning of one. +# load-side values and were unaffected throughout. UpsOnBattery in particular +# would still fire on a transfer — it would just be reporting an outage already +# in progress rather than warning of one. # # THE ONE METRIC THAT DOES NOT LIE. # # upsTestResultsSummary is the exception to everything above. It reports the -# result of the last diagnostics test, and the UPS will not fake one — with no -# pack fitted it cannot complete a test, so it reports the failure honestly +# result of the last diagnostics test, and the UPS will not fake one — without a +# usable pack it cannot complete a test, so it reports the failure honestly # while charge, runtime, voltage and alarm count all read healthy. # # 1 donePass 2 doneWarning 3 doneError # 4 aborted 5 inProgress 6 noTestsInitiated # -# Observed on this UPS: 4 (aborted). The management card's own diagnostics page -# renders the same state as "Refused — internal fault". +# Observed before the fit: 4 (aborted), which the management card's own +# diagnostics page rendered as "Refused — internal fault". No self-test has been +# run since the pack went in, so that remains the last stored result and this +# metric has not yet had anything new to say. # # UpsSelfTestFailed and UpsBatteryUnproven below are keyed on it, and they are -# the only two rules in this file that can detect the current condition. +# the only two rules in this file that can tell a UPS which works from one that +# merely looks like it does. +# +# WHAT IS STILL OUTSTANDING, AND WHY THIS HEADER IS NOT SHORTER YET. +# +# UpsSelfTestFailed is silenced in Alertmanager until 2026-09-20 +# (54f1715c-e57b-4322-8a6d-5435bc8e1bd8). That silence was correct while nothing +# could be done about a missing pack — it fired every 30 minutes about a known +# condition, which is how an urgent receiver stops being read. It stopped being +# correct the moment a pack was fitted, because it now suppresses the single +# rule that could report that pack being faulty or badly seated. Delete it +# before running the self-test, not after: run them in the other order and the +# one result most worth seeing is the one that is hidden. # -# Still do not run a UPS self-test by hand to force a reading. A test transfers -# the load to battery; with a pack fitted but weak, that is the outage it exists -# to warn about. The refusal above was safe only because no transfer occurred. -# Once a pack is in, enable the NMC's *scheduled* self-test instead so this -# metric stays live evidence rather than a stale last-known result. +# Still do not run self-tests by hand to keep the metric warm. A test transfers +# the load to battery, and on a pack that is weak that is the outage the alert +# exists to warn about. The pre-fit refusals were safe only because no transfer +# ever occurred. Enable the NMC's *scheduled* self-test instead, so this metric +# stays live evidence rather than a stale last-known result. # # Known limit: a value of 1 (donePass) is the last result, not a fresh one. If # the NMC stops testing entirely the value simply stops changing, and nothing @@ -156,8 +176,9 @@ groups: 2=doneWarning 3=doneError 4=aborted. A UPS that cannot complete a self-test cannot be assumed to carry the load, whatever its charge and runtime metrics claim. Check the management card's diagnostics - page — "Refused / internal fault" here means no pack is fitted or - the pack is not seen. + page — "Refused / internal fault" means the pack is not seen, and + now that one is fitted that is a finding about the pack or its + seating rather than about an empty bay. - alert: UpsBatteryUnproven # 6 = noTestsInitiated. Not a safe resting state: it is the state in