diff --git a/README.md b/README.md index 48b012c..cd141e2 100644 --- a/README.md +++ b/README.md @@ -222,8 +222,9 @@ and runtime, under a banner explaining that every battery figure is fabricated because no battery pack is fitted.](docs/images/ups-power.png) That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The -banner it shows has been rewritten since; re-shoot with `make screenshots` once -the self-test has passed and the panels read something worth showing. +banner it shows has been rewritten twice since, and the self-test has now +passed, so the panels read measured values rather than invented ones — re-shoot +the set with `make screenshots`. ## What runs it @@ -255,12 +256,14 @@ Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and why it is in that order. -The current top items: prove the UPS battery and rack the shelf switch. The pack -went in on 2026-08-28 and has not yet passed a self-test, so the rack is not -demonstrably protected yet; the switch between the monitoring host and the -network still has no battery at all -([#93](https://github.com/Gerrrt/HomeLab/issues/93), -[#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup +The current top items: put the UPS battery under scheduled test and rack the +shelf switch. The pack went in on 2026-08-28 and passed its self-test, so the +rack itself is protected — but the card is not testing on a schedule, which +leaves that pass a last-known result rather than a live one +([#93](https://github.com/Gerrrt/HomeLab/issues/93)); the switch between the +monitoring host and the network still has no battery at all, so both laptops +stay running and go deaf on a mains cut +([#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup off the machine it protects, and buy the spare that turns its restore runbook from a hypothesis into something rehearsed ([#92](https://github.com/Gerrrt/HomeLab/issues/92)); and take 64-bit interface diff --git a/docs/observability.md b/docs/observability.md index 65ee792..bf47411 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -105,7 +105,7 @@ boot check. | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | -| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 but has not yet passed a self-test, so most of these still read values the card may be fabricating — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | +| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | diff --git a/docs/roadmap.md b/docs/roadmap.md index 730a85d..7c317b2 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -95,12 +95,15 @@ what left this one unfireable for months. [`restore-the-firewall.md`](runbooks/restore-the-firewall.md) stays a hypothesis until it has been restored onto a spare once. - **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.** - An APCRBC115 went into `mjolnir` on 2026-08-28. That is the hardware done, not - the issue: the management card reports a healthy pack whether or not it can see - one, so nothing yet separates a good fit from a bad one. **The `UpsSelfTestFailed` - silence is still in place until 2026-09-20, and now suppresses the only alert - that could tell you which this is** — delete it, then self-test, then enable - scheduled tests on the card. + An APCRBC115 went into `mjolnir` on 2026-08-28 and passed its self-test the + same day: `upsTestResultsSummary` `4` → `1`, `upsBatteryVoltage` off its + fabricated `480`, runtime no longer pinned to exactly `63`. The + `UpsSelfTestFailed` silence was deleted rather than left to expire in + September, so the rule that would report a bad pack is live again. **What is + left is the last step: scheduled self-tests on the card.** Until they are on, + `1` is a last-known result with nothing refreshing it, and `UpsBatteryUnproven` + cannot detect a card that has quietly stopped testing — it matches `6` + (noTestsInitiated), and this one reads `1`. → [runbook](runbooks/fit-the-ups-battery.md) - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.** A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index e27ca0b..b720b8a 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -3,21 +3,28 @@ **One rack visit, in an order that matters — and one alert that has to be un-silenced by hand at the right moment.** -> **Status — 2026-08-28: the pack is in, and nothing else is done.** +> **Status — 2026-08-28: the pack is in and proven. The shelf is not.** > -> The APCRBC115 was fitted. **Steps 3 to 6 are all outstanding**, and step 3 is -> the urgent one: `UpsSelfTestFailed` is still silenced until 2026-09-20, so -> right now the only alert that could report a faulty or badly seated new pack -> is suppressed — on a pack nobody has tested yet. +> The APCRBC115 was fitted, the silence was deleted, and a self-test passed: +> `upsTestResultsSummary` went `4` (aborted) to `1` (donePass) at 22:45 UTC. +> Step 5's comparison holds — `upsBatteryVoltage` left its fabricated `480` for +> a float reading that varies, and the runtime estimate no longer sits on +> exactly `63`. Charge still reads exactly `100`, which is what a pack at float +> voltage looks like rather than a card that cannot see one. > -> The shelf and the switch move (step 2, items 1–4) were not done either, so +> **Step 6 is outstanding.** Scheduled self-tests are not enabled on the NMC, so +> `1` is a last-known result with nothing refreshing it, and nothing in +> `ups.rules.yaml` detects a card that has quietly stopped testing. +> +> The shelf and the switch move (step 2, items 1–4) were not done, so > [#110](https://github.com/Gerrrt/HomeLab/issues/110) is untouched and > `prometheus` and `oracle` still go deaf on a mains cut. > -> Step 1's baseline was not captured before the fit. It does not have to be -> re-derived: the pre-fit values are recorded in the table under step 1 and in -> the header of `ups.rules.yaml`, and comparing against those is what step 5 -> needs. +> **One thing to do differently next time.** The silence was deleted at 23:14 +> UTC — *after* the 22:45 self-test, not before it. It cost nothing here because +> the test passed, but for those 29 minutes a faulty pack would have been +> reported into a suppressed alert. That inversion is the single failure mode +> step 3 exists to prevent. `mjolnir` ran with no battery pack for the whole life of this stack. A mains loss was an immediate hard shutdown of the entire rack. The management card @@ -121,7 +128,10 @@ the cases that genuinely need the wire, and reads the credential from SOPS. ## 3. Delete the silence — immediately, not on expiry -`UpsSelfTestFailed` is silenced in Alertmanager until **2026-09-20** +> **Done 2026-08-28**, at 23:14 UTC. The procedure below is kept for the next +> time a pack is changed and a silence is standing over it. + +`UpsSelfTestFailed` was silenced in Alertmanager until **2026-09-20** (`54f1715c-e57b-4322-8a6d-5435bc8e1bd8`). It routes on `category=power` to the `urgent` receiver with `group_wait: 0s` and `repeat_interval: 30m`, so leaving it firing meant paging every half hour about a condition already known — which @@ -241,17 +251,27 @@ actually true — a pack is fitted and unproven, with the outstanding silence an self-test named. That is `ups-power.json`, `ups.rules.yaml`, `docs/security.md`, `docs/observability.md`, `README.md`, `docs/roadmap.md` and `docs/hardware.md`. -**Still to do, only once step 5 passes** — a separate commit made after -verification, not before: +**Done on 2026-08-28**, once the self-test passed: the same files moved from +*fitted and unproven* to *proven*, with the two things that outlast the fix +named in each — that history before 2026-08-28 is fabricated rather than +measured, and that scheduled self-tests are still off. That is `ups-power.json`, +`ups.rules.yaml`, `docs/security.md`, `docs/observability.md`, `README.md` and +`docs/roadmap.md`. + +Two deliberate departures from the plan above. The banner panel was **rewritten +rather than deleted**, because there is still something true and non-obvious for +it to say and keeping it holds the dashboard at 84 panels, so the +`scripts/check_docs.py` panel-count coupling stays untriggered. And **#93 was +not moved into Done**, because this roadmap entry defines it as *delete the +silence, then self-test, then enable scheduled tests* — the third is step 6, and +it is outstanding. + +**Still to do:** | File | What changes | | --- | --- | -| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "Battery fitted — not yet proven" banner panel; strip "(unproven — self-test pending)" from the five panel titles and reset their descriptions | -| `stacks/observability/prometheus/rules/ups.rules.yaml` | Cut the header back to a short note. The rules themselves do not change | -| `docs/security.md` | The "Mains power loss" row, and the closing paragraphs of "The UPS reported a battery it did not have" | -| `docs/observability.md` | The `ups.rules.yaml` row of the rule-file table | -| `README.md` | The UPS screenshot and the note under it, and the "current top items" paragraph. Re-capture with `make screenshots` | -| `docs/roadmap.md` | Move #93 into Done. Existing entries stay as written; the roadmap is a record | +| `docs/images/ups-power.png` | Re-capture the set with `make screenshots`. The committed image is from 2026-08-22 and shows the original no-battery banner; it needs the decrypted Grafana password, so it is a human step | +| `docs/roadmap.md` | Move #93 into Done — but only once step 6 is done and the card is testing on a schedule | And when the shelf is racked, which is a separate visit and separate issue: diff --git a/docs/security.md b/docs/security.md index 5c65e23..b895f4e 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs | -| Mains power loss | **Partly defended.** A pack was fitted to `mjolnir` on 2026-08-28 and has not yet proven it can carry the load — see below | +| Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | What it explicitly does **not** defend against: a determined attacker with physical access to the rack, a supply-chain compromise in an upstream container @@ -175,19 +175,25 @@ that file that can detect this condition. This is worth stating carefully: the monitoring did not fail, and neither did the rules. The device lied, and the rules trusted it. -**A pack was fitted on 2026-08-28, and that has not yet closed this.** A card -that cannot see a pack which *is* present — badly seated, or faulty out of the -box — emits the same five fabricated values as one sitting over an empty bay. So -a healthy-looking dashboard distinguishes nothing; only a passing self-test, and -charge and runtime that have moved off the pre-fit baseline, do. - -Neither has happened yet. No self-test has been run since the fit, so -`upsTestResultsSummary` still holds its pre-fit `4`. And `UpsSelfTestFailed` is -still silenced in Alertmanager until 2026-09-20 — a silence that was correct -while nothing could be done about a missing pack, and which now suppresses the -one rule that would report the new pack being bad. Deleting it is step 3 of -[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), it comes -before the self-test rather than after, and it is outstanding. +**A pack was fitted on 2026-08-28, and that closed this.** A card that cannot +see a pack which *is* present — badly seated, or faulty out of the box — emits +the same five fabricated values as one sitting over an empty bay, so a +healthy-looking dashboard distinguished nothing. Only a passing self-test and +readings that have left the pre-fit baseline do, and both now hold: +`upsTestResultsSummary` went `4` (aborted) to `1` (donePass), `upsBatteryVoltage` +left `480` for a float reading that varies, and the runtime estimate no longer +sits on exactly `63`. The silence on `UpsSelfTestFailed` was deleted the same day +rather than left to expire in September, so that rule is live again. + +Two things outlast the fix. Stored metrics older than 2026-08-28 *are* the +fabricated values rather than measurements, so a dashboard or query whose range +crosses that date is reading fiction on one side of it. And scheduled self-tests +are not yet enabled on the card, which makes `1` a last-known result with +nothing refreshing it — `UpsBatteryUnproven` cannot catch a card that stops +testing, because it matches `6` (noTestsInitiated) and this one now reads `1`. +That is step 6 of +[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and it is +outstanding. ### Why SNMPv2c is still a weak point diff --git a/stacks/observability/grafana/dashboards/ups-power.json b/stacks/observability/grafana/dashboards/ups-power.json index f83df1e..f0c9cd3 100644 --- a/stacks/observability/grafana/dashboards/ups-power.json +++ b/stacks/observability/grafana/dashboards/ups-power.json @@ -1,7 +1,7 @@ { "uid": "homelab-ups", "title": "UPS & Power", - "description": "APC Smart-UPS (mjolnir) via UPS-MIB. A replacement pack was fitted on 2026-08-28 and has not yet passed a self-test, so runtime figures are not proven — see docs/runbooks/fit-the-ups-battery.md.", + "description": "APC Smart-UPS (mjolnir) via UPS-MIB. An APCRBC115 pack was fitted on 2026-08-28 and passed its self-test the same day, so the battery figures are measured — but anything before that date is fabricated, and scheduled self-tests are not yet enabled on the card. See docs/runbooks/fit-the-ups-battery.md.", "tags": [ "homelab", "power", @@ -258,7 +258,7 @@ }, { "type": "text", - "title": "Battery fitted 2026-08-28 — not yet proven", + "title": "Battery proven 2026-08-28 — the history before it is not", "gridPos": { "x": 0, "y": 6, @@ -267,7 +267,7 @@ }, "options": { "mode": "markdown", - "content": "**A pack is fitted. Nothing has yet confirmed the UPS can carry load.** An APCRBC115 cartridge went into `mjolnir` on 2026-08-28. Until a self-test passes, every battery figure below is unproven rather than measured: a management card that cannot see a fitted pack reports `batteryNormal`, 100% charge, 48.0 V and a runtime estimate exactly as it did when the bay was empty.\n\nThe test is comparison, not appearance. Charge still reading exactly `100` and runtime still at exactly `63` is the signature of a card that is still fabricating — compare against the pre-fit baseline in step 5 of `docs/runbooks/fit-the-ups-battery.md`.\n\n**Two steps are outstanding.** The `UpsSelfTestFailed` silence `54f1715c` runs until 2026-09-20 and is suppressing the one alert that can report a faulty or badly seated new pack — delete it *before* running the self-test, not after. Then enable scheduled self-tests on the card, so `upsTestResultsSummary` stays live evidence rather than a stale last result." + "content": "**The pack is fitted and proven.** An APCRBC115 went into `mjolnir` on 2026-08-28 and `upsTestResultsSummary` moved `4` (aborted) → `1` (donePass) at 22:45 UTC. `upsBatteryVoltage` left its fabricated `480` for a float reading that varies, and the runtime estimate moves instead of sitting on exactly `63` — the step 5 comparison a card cannot fake. Charge still reads exactly `100`, which is what a pack at float voltage looks like.\n\n**Anything on these panels before 2026-08-28 is fabricated.** With an empty bay the card invented a flat 100%, 63 minutes and 48.0 V. The trend panels below cross that boundary — widen the time range with that in mind.\n\n**Scheduled self-tests are not enabled yet.** `1` is the *last* result, not a fresh one; if the NMC never tests again this stops changing and nothing notices. `UpsBatteryUnproven` cannot catch it — that rule matches `6` (noTestsInitiated), and this card now reads `1`. Step 6 of `docs/runbooks/fit-the-ups-battery.md`." } }, { @@ -361,8 +361,8 @@ }, { "type": "gauge", - "title": "Battery charge (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Battery charge", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -497,8 +497,8 @@ }, { "type": "stat", - "title": "Runtime remaining (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Runtime remaining", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -567,8 +567,8 @@ }, { "type": "stat", - "title": "Battery status (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Battery status", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -791,8 +791,8 @@ }, { "type": "stat", - "title": "Battery temp (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Battery temp", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -873,8 +873,8 @@ }, { "type": "timeseries", - "title": "Battery charge and runtime (unproven — self-test pending)", - "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", + "title": "Battery charge and runtime", + "description": "Measured. The APCRBC115 pack was fitted to mjolnir on 2026-08-28 and passed its self-test at 22:45 UTC that day. Values before that date are fabricated — the management card reported them over an empty bay — so history either side of that boundary is two different things. See the note at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index ed9acc7..67b094a 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -1,16 +1,17 @@ --- # APC Smart-UPS (mjolnir, 10.0.99.10) via UPS-MIB. # -# READ THIS BEFORE TRUSTING ANY ALERT BELOW. +# These rules now read real hardware. An APCRBC115 pack was fitted on 2026-08-28 +# and passed its self-test the same day: upsTestResultsSummary went 4 (aborted) +# to 1 (donePass) at 22:45 UTC, upsBatteryVoltage left its fabricated 480 for a +# float reading that varies, and the runtime estimate moves rather than sitting +# on exactly 63. Every rule here is deliberately loud and routed to the `urgent` +# receiver by category=power in alertmanager.yaml. # -# An APCRBC115 pack was fitted on 2026-08-28. Until then the UPS had none and a -# mains loss was an immediate hard shutdown of the whole rack. Every rule here -# is deliberately loud and routed to the `urgent` receiver by category=power in -# alertmanager.yaml. +# WHY THE PRE-FIT BASELINE IS STILL RECORDED HERE. # -# A fitted pack is not yet a proven one, and here that distinction is not -# pedantry. The management card never reported the missing pack; it fabricated -# a healthy one. Observed on this UPS with no battery physically present: +# Until 2026-08-28 this UPS had no pack, and the management card never reported +# that. It fabricated a healthy one: # # upsBatteryStatus 2 (batteryNormal) # upsEstimatedChargeRemaining 100 (percent) @@ -18,64 +19,48 @@ # upsBatteryVoltage 480 (48.0 V) # upsAlarmsPresent 0 (no alarm raised) # -# A card that cannot see a pack which *is* fitted — one badly seated, or dead on -# arrival — reports those same five values. So the numbers looking healthy is -# not evidence; only their having moved off that pre-fit baseline is. Step 5 of -# docs/runbooks/fit-the-ups-battery.md is that comparison, and it has not been -# made yet. +# Those numbers stay for two reasons. Stored metrics older than 2026-08-28 *are* +# these values rather than measurements, so any query or dashboard whose range +# crosses that date is reading fiction on one side of it. And a card that cannot +# see a pack which *is* fitted — badly seated, or dead on arrival — emits this +# same set again, so it is the comparison to make the next time a pack changes. # -# Until it has, UpsBatteryLow, UpsChargeLow, UpsRuntimeCritical and -# UpsAlarmPresent sit comfortably inside their thresholds and still tell you -# nothing. They are kept because they become correct the moment the card is -# reading real hardware — not because they are protecting anything yet. A quiet -# `ups` group is not evidence of runtime. -# -# UpsOnBattery, UpsOverloaded and UpsBatteryTemperatureHigh read mains-side or -# load-side values and were unaffected throughout. UpsOnBattery in particular -# would still fire on a transfer — it would just be reporting an outage already -# in progress rather than warning of one. +# upsEstimatedChargeRemaining still reads exactly 100. On its own that is the +# fabrication signature; here it is consistent with a pack at float voltage, +# because voltage and runtime both moved and the self-test passed. Charge alone +# was never the evidence. # # THE ONE METRIC THAT DOES NOT LIE. # -# upsTestResultsSummary is the exception to everything above. It reports the -# result of the last diagnostics test, and the UPS will not fake one — without a -# usable pack it cannot complete a test, so it reports the failure honestly -# while charge, runtime, voltage and alarm count all read healthy. +# upsTestResultsSummary reports the result of the last diagnostics test, and the +# UPS will not fake one — without a usable pack it cannot complete a test, so it +# reports the failure honestly while charge, runtime, voltage and alarm count +# all read healthy. # # 1 donePass 2 doneWarning 3 doneError # 4 aborted 5 inProgress 6 noTestsInitiated # -# Observed before the fit: 4 (aborted), which the management card's own -# diagnostics page rendered as "Refused — internal fault". No self-test has been -# run since the pack went in, so that remains the last stored result and this -# metric has not yet had anything new to say. -# # UpsSelfTestFailed and UpsBatteryUnproven below are keyed on it, and they are # the only two rules in this file that can tell a UPS which works from one that -# merely looks like it does. -# -# WHAT IS STILL OUTSTANDING, AND WHY THIS HEADER IS NOT SHORTER YET. -# -# UpsSelfTestFailed is silenced in Alertmanager until 2026-09-20 -# (54f1715c-e57b-4322-8a6d-5435bc8e1bd8). That silence was correct while nothing -# could be done about a missing pack — it fired every 30 minutes about a known -# condition, which is how an urgent receiver stops being read. It stopped being -# correct the moment a pack was fitted, because it now suppresses the single -# rule that could report that pack being faulty or badly seated. Delete it -# before running the self-test, not after: run them in the other order and the -# one result most worth seeing is the one that is hidden. -# -# Still do not run self-tests by hand to keep the metric warm. A test transfers -# the load to battery, and on a pack that is weak that is the outage the alert +# merely looks like it does. The silence that held UpsSelfTestFailed until +# 2026-09-20 (54f1715c-e57b-4322-8a6d-5435bc8e1bd8) was deleted on 2026-08-28, +# so both rules are live. +# +# WHAT IS STILL OUTSTANDING. +# +# Scheduled self-tests are not enabled on the NMC. A value of 1 is the *last* +# result, not a fresh one: if the card stops testing the metric simply stops +# changing, and nothing here detects that. UpsBatteryUnproven cannot — it +# matches 6 (noTestsInitiated), and this card now reads 1. upsTestStartTime is +# sysUpTime-relative and does not survive an agent restart, so it is not a +# usable staleness signal either. Scheduled tests on the device are the control +# and this file cannot substitute for them; step 6 of +# docs/runbooks/fit-the-ups-battery.md. +# +# Do not run self-tests by hand to keep the metric warm. A test transfers the +# load to battery, and on a pack that is weak that is the outage the alert # exists to warn about. The pre-fit refusals were safe only because no transfer -# ever occurred. Enable the NMC's *scheduled* self-test instead, so this metric -# stays live evidence rather than a stale last-known result. -# -# Known limit: a value of 1 (donePass) is the last result, not a fresh one. If -# the NMC stops testing entirely the value simply stops changing, and nothing -# here detects that. upsTestStartTime is sysUpTime-relative and does not survive -# an agent restart, so it is not a usable staleness signal. Scheduled tests on -# the device are the control; this file cannot substitute for them. +# ever occurred. groups: - name: ups interval: 60s