From 0c00f9f8ccc2128a5f1afecbf37e761534a787e1 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Fri, 28 Aug 2026 23:34:24 +0000 Subject: [PATCH] docs(ups): record a battery that is proven, and what still is not MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The self-test passed. upsTestResultsSummary went 4 (aborted) to 1 (donePass) at 22:45 UTC, and step 5's comparison — the part the card cannot fake — holds: upsBatteryVoltage left its fabricated 480 for a float reading that varies between 543 and 546, and the runtime estimate moves rather than sitting on exactly 63. The silence 54f1715c was deleted at 23:14 rather than left to run to 2026-09-20, so UpsSelfTestFailed is live again. Charge still reads exactly 100. On its own that is the fabrication signature the previous commit warned about, so it is worth saying why it is not one here: voltage and runtime both moved and the test passed, which is float voltage on a topped-off pack. Charge alone was never the evidence. Two things outlast the fix, and every file that claims the battery works now says both: - stored metrics older than 2026-08-28 *are* the fabricated values, not measurements. The dashboard defaults to a 24h window and its trend panels cross that boundary today; any wider range crosses it forever - scheduled self-tests are still off, so 1 is a last-known result with nothing refreshing it. UpsBatteryUnproven cannot catch a card that stops testing — it matches 6 (noTestsInitiated) and this one reads 1 Two departures from the runbook's flip list, both deliberate: - the banner panel is rewritten rather than deleted. It still has something true and non-obvious to say — the fabricated history above — and keeping it holds the dashboard at 84 panels, so the check_docs.py panel-count coupling the runbook warned about is never triggered - #93 is not moved into Done. The roadmap defines it as delete the silence, then self-test, then enable scheduled tests; the third is outstanding, so the entry stays open with its remaining scope narrowed to that Two files the runbook's own list had missed: docs/observability.md's rule-file row, and the runbook itself, whose step 3 still said in the present tense that the silence was in place. Dropped a stale claim from the ups.rules.yaml header while rewriting it — it grouped UpsBatteryTemperatureHigh with rules reading "mains-side or load-side values" and called it unaffected, but it reads upsBatteryTemperature, which was fabricated like the rest. Moot now that all of them are real. The screenshot is left as it is. It is an honest render of what the dashboard said on 2026-08-22, docs/images/README.md refuses mocked-up images, and `make screenshots` needs the decrypted Grafana password. #110 is untouched — the shelf and switch move did not happen, so `prometheus` and `oracle` still go deaf on a mains cut and the rack's own protection is the only thing this commit can claim. Refs #93, #110 Co-Authored-By: Claude Opus 5 --- README.md | 19 ++-- docs/observability.md | 2 +- docs/roadmap.md | 15 +-- docs/runbooks/fit-the-ups-battery.md | 58 +++++++---- docs/security.md | 34 ++++--- .../grafana/dashboards/ups-power.json | 26 ++--- .../prometheus/rules/ups.rules.yaml | 95 ++++++++----------- 7 files changed, 133 insertions(+), 116 deletions(-) diff --git a/README.md b/README.md index 48b012c..cd141e2 100644 --- a/README.md +++ b/README.md @@ -222,8 +222,9 @@ and runtime, under a banner explaining that every battery figure is fabricated because no battery pack is fitted.](docs/images/ups-power.png) That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The -banner it shows has been rewritten since; re-shoot with `make screenshots` once -the self-test has passed and the panels read something worth showing. +banner it shows has been rewritten twice since, and the self-test has now +passed, so the panels read measured values rather than invented ones — re-shoot +the set with `make screenshots`. ## What runs it @@ -255,12 +256,14 @@ Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and why it is in that order. -The current top items: prove the UPS battery and rack the shelf switch. The pack -went in on 2026-08-28 and has not yet passed a self-test, so the rack is not -demonstrably protected yet; the switch between the monitoring host and the -network still has no battery at all -([#93](https://github.com/Gerrrt/HomeLab/issues/93), -[#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup +The current top items: put the UPS battery under scheduled test and rack the +shelf switch. The pack went in on 2026-08-28 and passed its self-test, so the +rack itself is protected — but the card is not testing on a schedule, which +leaves that pass a last-known result rather than a live one +([#93](https://github.com/Gerrrt/HomeLab/issues/93)); the switch between the +monitoring host and the network still has no battery at all, so both laptops +stay running and go deaf on a mains cut +([#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup off the machine it protects, and buy the spare that turns its restore runbook from a hypothesis into something rehearsed ([#92](https://github.com/Gerrrt/HomeLab/issues/92)); and take 64-bit interface diff --git a/docs/observability.md b/docs/observability.md index 65ee792..bf47411 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -105,7 +105,7 @@ boot check. | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | -| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 but has not yet passed a self-test, so most of these still read values the card may be fabricating — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | +| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | diff --git a/docs/roadmap.md b/docs/roadmap.md index 730a85d..7c317b2 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -95,12 +95,15 @@ what left this one unfireable for months. [`restore-the-firewall.md`](runbooks/restore-the-firewall.md) stays a hypothesis until it has been restored onto a spare once. - **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.** - An APCRBC115 went into `mjolnir` on 2026-08-28. That is the hardware done, not - the issue: the management card reports a healthy pack whether or not it can see - one, so nothing yet separates a good fit from a bad one. **The `UpsSelfTestFailed` - silence is still in place until 2026-09-20, and now suppresses the only alert - that could tell you which this is** — delete it, then self-test, then enable - scheduled tests on the card. + An APCRBC115 went into `mjolnir` on 2026-08-28 and passed its self-test the + same day: `upsTestResultsSummary` `4` → `1`, `upsBatteryVoltage` off its + fabricated `480`, runtime no longer pinned to exactly `63`. The + `UpsSelfTestFailed` silence was deleted rather than left to expire in + September, so the rule that would report a bad pack is live again. **What is + left is the last step: scheduled self-tests on the card.** Until they are on, + `1` is a last-known result with nothing refreshing it, and `UpsBatteryUnproven` + cannot detect a card that has quietly stopped testing — it matches `6` + (noTestsInitiated), and this one reads `1`. → [runbook](runbooks/fit-the-ups-battery.md) - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.** A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index e27ca0b..b720b8a 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -3,21 +3,28 @@ **One rack visit, in an order that matters — and one alert that has to be un-silenced by hand at the right moment.** -> **Status — 2026-08-28: the pack is in, and nothing else is done.** +> **Status — 2026-08-28: the pack is in and proven. The shelf is not.** > -> The APCRBC115 was fitted. **Steps 3 to 6 are all outstanding**, and step 3 is -> the urgent one: `UpsSelfTestFailed` is still silenced until 2026-09-20, so -> right now the only alert that could report a faulty or badly seated new pack -> is suppressed — on a pack nobody has tested yet. +> The APCRBC115 was fitted, the silence was deleted, and a self-test passed: +> `upsTestResultsSummary` went `4` (aborted) to `1` (donePass) at 22:45 UTC. +> Step 5's comparison holds — `upsBatteryVoltage` left its fabricated `480` for +> a float reading that varies, and the runtime estimate no longer sits on +> exactly `63`. Charge still reads exactly `100`, which is what a pack at float +> voltage looks like rather than a card that cannot see one. > -> The shelf and the switch move (step 2, items 1–4) were not done either, so +> **Step 6 is outstanding.** Scheduled self-tests are not enabled on the NMC, so +> `1` is a last-known result with nothing refreshing it, and nothing in +> `ups.rules.yaml` detects a card that has quietly stopped testing. +> +> The shelf and the switch move (step 2, items 1–4) were not done, so > [#110](https://github.com/Gerrrt/HomeLab/issues/110) is untouched and > `prometheus` and `oracle` still go deaf on a mains cut. > -> Step 1's baseline was not captured before the fit. It does not have to be -> re-derived: the pre-fit values are recorded in the table under step 1 and in -> the header of `ups.rules.yaml`, and comparing against those is what step 5 -> needs. +> **One thing to do differently next time.** The silence was deleted at 23:14 +> UTC — *after* the 22:45 self-test, not before it. It cost nothing here because +> the test passed, but for those 29 minutes a faulty pack would have been +> reported into a suppressed alert. That inversion is the single failure mode +> step 3 exists to prevent. `mjolnir` ran with no battery pack for the whole life of this stack. A mains loss was an immediate hard shutdown of the entire rack. The management card @@ -121,7 +128,10 @@ the cases that genuinely need the wire, and reads the credential from SOPS. ## 3. Delete the silence — immediately, not on expiry -`UpsSelfTestFailed` is silenced in Alertmanager until **2026-09-20** +> **Done 2026-08-28**, at 23:14 UTC. The procedure below is kept for the next +> time a pack is changed and a silence is standing over it. + +`UpsSelfTestFailed` was silenced in Alertmanager until **2026-09-20** (`54f1715c-e57b-4322-8a6d-5435bc8e1bd8`). It routes on `category=power` to the `urgent` receiver with `group_wait: 0s` and `repeat_interval: 30m`, so leaving it firing meant paging every half hour about a condition already known — which @@ -241,17 +251,27 @@ actually true — a pack is fitted and unproven, with the outstanding silence an self-test named. That is `ups-power.json`, `ups.rules.yaml`, `docs/security.md`, `docs/observability.md`, `README.md`, `docs/roadmap.md` and `docs/hardware.md`. -**Still to do, only once step 5 passes** — a separate commit made after -verification, not before: +**Done on 2026-08-28**, once the self-test passed: the same files moved from +*fitted and unproven* to *proven*, with the two things that outlast the fix +named in each — that history before 2026-08-28 is fabricated rather than +measured, and that scheduled self-tests are still off. That is `ups-power.json`, +`ups.rules.yaml`, `docs/security.md`, `docs/observability.md`, `README.md` and +`docs/roadmap.md`. + +Two deliberate departures from the plan above. The banner panel was **rewritten +rather than deleted**, because there is still something true and non-obvious for +it to say and keeping it holds the dashboard at 84 panels, so the +`scripts/check_docs.py` panel-count coupling stays untriggered. And **#93 was +not moved into Done**, because this roadmap entry defines it as *delete the +silence, then self-test, then enable scheduled tests* — the third is step 6, and +it is outstanding. + +**Still to do:** | File | What changes | | --- | --- | -| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "Battery fitted — not yet proven" banner panel; strip "(unproven — self-test pending)" from the five panel titles and reset their descriptions | -| `stacks/observability/prometheus/rules/ups.rules.yaml` | Cut the header back to a short note. The rules themselves do not change | -| `docs/security.md` | The "Mains power loss" row, and the closing paragraphs of "The UPS reported a battery it did not have" | -| `docs/observability.md` | The `ups.rules.yaml` row of the rule-file table | -| `README.md` | The UPS screenshot and the note under it, and the "current top items" paragraph. Re-capture with `make screenshots` | -| `docs/roadmap.md` | Move #93 into Done. Existing entries stay as written; the roadmap is a record | +| `docs/images/ups-power.png` | Re-capture the set with `make screenshots`. The committed image is from 2026-08-22 and shows the original no-battery banner; it needs the decrypted Grafana password, so it is a human step | +| `docs/roadmap.md` | Move #93 into Done — but only once step 6 is done and the card is testing on a schedule | And when the shelf is racked, which is a separate visit and separate issue: diff --git a/docs/security.md b/docs/security.md index 5c65e23..b895f4e 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs | -| Mains power loss | **Partly defended.** A pack was fitted to `mjolnir` on 2026-08-28 and has not yet proven it can carry the load — see below | +| Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | What it explicitly does **not** defend against: a determined attacker with physical access to the rack, a supply-chain compromise in an upstream container @@ -175,19 +175,25 @@ that file that can detect this condition. This is worth stating carefully: the monitoring did not fail, and neither did the rules. The device lied, and the rules trusted it. -**A pack was fitted on 2026-08-28, and that has not yet closed this.** A card -that cannot see a pack which *is* present — badly seated, or faulty out of the -box — emits the same five fabricated values as one sitting over an empty bay. So -a healthy-looking dashboard distinguishes nothing; only a passing self-test, and -charge and runtime that have moved off the pre-fit baseline, do. - -Neither has happened yet. No self-test has been run since the fit, so -`upsTestResultsSummary` still holds its pre-fit `4`. And `UpsSelfTestFailed` is -still silenced in Alertmanager until 2026-09-20 — a silence that was correct -while nothing could be done about a missing pack, and which now suppresses the -one rule that would report the new pack being bad. Deleting it is step 3 of -[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), it comes -before the self-test rather than after, and it is outstanding. +**A pack was fitted on 2026-08-28, and that closed this.** A card that cannot +see a pack which *is* present — badly seated, or faulty out of the box — emits +the same five fabricated values as one sitting over an empty bay, so a +healthy-looking dashboard distinguished nothing. Only a passing self-test and +readings that have left the pre-fit baseline do, and both now hold: +`upsTestResultsSummary` went `4` (aborted) to `1` (donePass), `upsBatteryVoltage` +left `480` for a float reading that varies, and the runtime estimate no longer +sits on exactly `63`. The silence on `UpsSelfTestFailed` was deleted the same day +rather than left to expire in September, so that rule is live again. + +Two things outlast the fix. Stored metrics older than 2026-08-28 *are* the +fabricated values rather than measurements, so a dashboard or query whose range +crosses that date is reading fiction on one side of it. And scheduled self-tests +are not yet enabled on the card, which makes `1` a last-known result with +nothing refreshing it — `UpsBatteryUnproven` cannot catch a card that stops +testing, because it matches `6` (noTestsInitiated) and this one now reads `1`. +That is step 6 of +[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and it is +outstanding. ### Why SNMPv2c is still a weak point diff --git a/stacks/observability/grafana/dashboards/ups-power.json b/stacks/observability/grafana/dashboards/ups-power.json index f83df1e..f0c9cd3 100644 --- a/stacks/observability/grafana/dashboards/ups-power.json +++ b/stacks/observability/grafana/dashboards/ups-power.json @@ -1,7 +1,7 @@ { "uid": "homelab-ups", "title": "UPS & Power", - "description": "APC Smart-UPS (mjolnir) via UPS-MIB. A replacement pack was fitted on 2026-08-28 and has not yet passed a self-test, so runtime figures are not proven — see docs/runbooks/fit-the-ups-battery.md.", + "description": "APC Smart-UPS (mjolnir) via UPS-MIB. An APCRBC115 pack was fitted on 2026-08-28 and passed its self-test the same day, so the battery figures are measured — but anything before that date is fabricated, and scheduled self-tests are not yet enabled on the card. See docs/runbooks/fit-the-ups-battery.md.", "tags": [ "homelab", "power", @@ -258,7 +258,7 @@ }, { "type": "text", - "title": "Battery fitted 2026-08-28 — not yet proven", + "title": "Battery proven 2026-08-28 — the history before it is not", "gridPos": { "x": 0, "y": 6, @@ -267,7 +267,7 @@ }, "options": { "mode": "markdown", - "content": "**A pack is fitted. Nothing has yet confirmed the UPS can carry load.** An APCRBC115 cartridge went into `mjolnir` on 2026-08-28. Until a self-test passes, every battery figure below is unproven rather than measured: a management card that cannot see a fitted pack reports `batteryNormal`, 100% charge, 48.0 V and a runtime estimate exactly as it did when the bay was empty.\n\nThe test is comparison, not appearance. Charge still reading exactly `100` and runtime still at exactly `63` is the signature of a card that is still fabricating — compare against the pre-fit baseline in step 5 of `docs/runbooks/fit-the-ups-battery.md`.\n\n**Two steps are outstanding.** The `UpsSelfTestFailed` silence `54f1715c` runs until 2026-09-20 and is suppressing the one alert that can report a faulty or badly seated new pack — delete it *before* running the self-test, not after. Then enable scheduled self-tests on the card, so `upsTestResultsSummary` stays live evidence rather than a stale last result." + "content": "**The pack is fitted and proven.** An APCRBC115 went into `mjolnir` on 2026-08-28 and `upsTestResultsSummary` moved `4` (aborted) → `1` (donePass) at 22:45 UTC. `upsBatteryVoltage` left its fabricated `480` for a float reading that varies, and the runtime estimate moves instead of sitting on exactly `63` — the step 5 comparison a card cannot fake. Charge still reads exactly `100`, which is what a pack at float voltage looks like.\n\n**Anything on these panels before 2026-08-28 is fabricated.** With an empty bay the card invented a flat 100%, 63 minutes and 48.0 V. The trend panels below cross that boundary — widen the time range with that in mind.\n\n**Scheduled self-tests are not enabled yet.** `1` is the *last* result, not a fresh one; if the NMC never tests again this stops changing and nothing notices. `UpsBatteryUnproven` cannot catch it — that rule matches `6` (noTestsInitiated), and this card now reads `1`. Step 6 of `docs/runbooks/fit-the-ups-battery.md`." } }, { @@ -361,8 +361,8 @@ }, { "type": "gauge", - "title": "Battery charge (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Battery charge", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -497,8 +497,8 @@ }, { "type": "stat", - "title": "Runtime remaining (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Runtime remaining", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -567,8 +567,8 @@ }, { "type": "stat", - "title": "Battery status (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Battery status", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -791,8 +791,8 @@ }, { "type": "stat", - "title": "Battery temp (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Battery temp", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -873,8 +873,8 @@ }, { "type": "timeseries", - "title": "Battery charge and runtime (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Battery charge and runtime", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index ed9acc7..67b094a 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -1,16 +1,17 @@ --- # APC Smart-UPS (mjolnir, 10.0.99.10) via UPS-MIB. # -# READ THIS BEFORE TRUSTING ANY ALERT BELOW. +# These rules now read real hardware. An APCRBC115 pack was fitted on 2026-08-28 +# and passed its self-test the same day: upsTestResultsSummary went 4 (aborted) +# to 1 (donePass) at 22:45 UTC, upsBatteryVoltage left its fabricated 480 for a +# float reading that varies, and the runtime estimate moves rather than sitting +# on exactly 63. Every rule here is deliberately loud and routed to the `urgent` +# receiver by category=power in alertmanager.yaml. # -# An APCRBC115 pack was fitted on 2026-08-28. Until then the UPS had none and a -# mains loss was an immediate hard shutdown of the whole rack. Every rule here -# is deliberately loud and routed to the `urgent` receiver by category=power in -# alertmanager.yaml. +# WHY THE PRE-FIT BASELINE IS STILL RECORDED HERE. # -# A fitted pack is not yet a proven one, and here that distinction is not -# pedantry. The management card never reported the missing pack; it fabricated -# a healthy one. Observed on this UPS with no battery physically present: +# Until 2026-08-28 this UPS had no pack, and the management card never reported +# that. It fabricated a healthy one: # # upsBatteryStatus 2 (batteryNormal) # upsEstimatedChargeRemaining 100 (percent) @@ -18,64 +19,48 @@ # upsBatteryVoltage 480 (48.0 V) # upsAlarmsPresent 0 (no alarm raised) # -# A card that cannot see a pack which *is* fitted — one badly seated, or dead on -# arrival — reports those same five values. So the numbers looking healthy is -# not evidence; only their having moved off that pre-fit baseline is. Step 5 of -# docs/runbooks/fit-the-ups-battery.md is that comparison, and it has not been -# made yet. +# Those numbers stay for two reasons. Stored metrics older than 2026-08-28 *are* +# these values rather than measurements, so any query or dashboard whose range +# crosses that date is reading fiction on one side of it. And a card that cannot +# see a pack which *is* fitted — badly seated, or dead on arrival — emits this +# same set again, so it is the comparison to make the next time a pack changes. # -# Until it has, UpsBatteryLow, UpsChargeLow, UpsRuntimeCritical and -# UpsAlarmPresent sit comfortably inside their thresholds and still tell you -# nothing. They are kept because they become correct the moment the card is -# reading real hardware — not because they are protecting anything yet. A quiet -# `ups` group is not evidence of runtime. -# -# UpsOnBattery, UpsOverloaded and UpsBatteryTemperatureHigh read mains-side or -# load-side values and were unaffected throughout. UpsOnBattery in particular -# would still fire on a transfer — it would just be reporting an outage already -# in progress rather than warning of one. +# upsEstimatedChargeRemaining still reads exactly 100. On its own that is the +# fabrication signature; here it is consistent with a pack at float voltage, +# because voltage and runtime both moved and the self-test passed. Charge alone +# was never the evidence. # # THE ONE METRIC THAT DOES NOT LIE. # -# upsTestResultsSummary is the exception to everything above. It reports the -# result of the last diagnostics test, and the UPS will not fake one — without a -# usable pack it cannot complete a test, so it reports the failure honestly -# while charge, runtime, voltage and alarm count all read healthy. +# upsTestResultsSummary reports the result of the last diagnostics test, and the +# UPS will not fake one — without a usable pack it cannot complete a test, so it +# reports the failure honestly while charge, runtime, voltage and alarm count +# all read healthy. # # 1 donePass 2 doneWarning 3 doneError # 4 aborted 5 inProgress 6 noTestsInitiated # -# Observed before the fit: 4 (aborted), which the management card's own -# diagnostics page rendered as "Refused — internal fault". No self-test has been -# run since the pack went in, so that remains the last stored result and this -# metric has not yet had anything new to say. -# # UpsSelfTestFailed and UpsBatteryUnproven below are keyed on it, and they are # the only two rules in this file that can tell a UPS which works from one that -# merely looks like it does. -# -# WHAT IS STILL OUTSTANDING, AND WHY THIS HEADER IS NOT SHORTER YET. -# -# UpsSelfTestFailed is silenced in Alertmanager until 2026-09-20 -# (54f1715c-e57b-4322-8a6d-5435bc8e1bd8). That silence was correct while nothing -# could be done about a missing pack — it fired every 30 minutes about a known -# condition, which is how an urgent receiver stops being read. It stopped being -# correct the moment a pack was fitted, because it now suppresses the single -# rule that could report that pack being faulty or badly seated. Delete it -# before running the self-test, not after: run them in the other order and the -# one result most worth seeing is the one that is hidden. -# -# Still do not run self-tests by hand to keep the metric warm. A test transfers -# the load to battery, and on a pack that is weak that is the outage the alert +# merely looks like it does. The silence that held UpsSelfTestFailed until +# 2026-09-20 (54f1715c-e57b-4322-8a6d-5435bc8e1bd8) was deleted on 2026-08-28, +# so both rules are live. +# +# WHAT IS STILL OUTSTANDING. +# +# Scheduled self-tests are not enabled on the NMC. A value of 1 is the *last* +# result, not a fresh one: if the card stops testing the metric simply stops +# changing, and nothing here detects that. UpsBatteryUnproven cannot — it +# matches 6 (noTestsInitiated), and this card now reads 1. upsTestStartTime is +# sysUpTime-relative and does not survive an agent restart, so it is not a +# usable staleness signal either. Scheduled tests on the device are the control +# and this file cannot substitute for them; step 6 of +# docs/runbooks/fit-the-ups-battery.md. +# +# Do not run self-tests by hand to keep the metric warm. A test transfers the +# load to battery, and on a pack that is weak that is the outage the alert # exists to warn about. The pre-fit refusals were safe only because no transfer -# ever occurred. Enable the NMC's *scheduled* self-test instead, so this metric -# stays live evidence rather than a stale last-known result. -# -# Known limit: a value of 1 (donePass) is the last result, not a fresh one. If -# the NMC stops testing entirely the value simply stops changing, and nothing -# here detects that. upsTestStartTime is sysUpTime-relative and does not survive -# an agent restart, so it is not a usable staleness signal. Scheduled tests on -# the device are the control; this file cannot substitute for them. +# ever occurred. groups: - name: ups interval: 60s