diff --git a/README.md b/README.md index a44942c..48b012c 100644 --- a/README.md +++ b/README.md @@ -221,6 +221,10 @@ draw and hardware health.](docs/images/network-snmp.png) and runtime, under a banner explaining that every battery figure is fabricated because no battery pack is fitted.](docs/images/ups-power.png) +That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The +banner it shows has been rewritten since; re-shoot with `make screenshots` once +the self-test has passed and the panels read something worth showing. + ## What runs it The entire observability stack runs on a 2012 MacBook Pro with Ubuntu Server on @@ -251,8 +255,10 @@ Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and why it is in that order. -The current top items: replace the UPS battery and rack the shelf switch, which -are one purchase — the pack alone leaves the monitoring path half-protected +The current top items: prove the UPS battery and rack the shelf switch. The pack +went in on 2026-08-28 and has not yet passed a self-test, so the rack is not +demonstrably protected yet; the switch between the monitoring host and the +network still has no battery at all ([#93](https://github.com/Gerrrt/HomeLab/issues/93), [#110](https://github.com/Gerrrt/HomeLab/issues/110)); get the firewall backup off the machine it protects, and buy the spare that turns its restore runbook diff --git a/docs/hardware.md b/docs/hardware.md index cbc84ce..fe40d38 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -55,6 +55,7 @@ revisions of this repository treated `shiva` as the hypervisor itself. - 1U rackmount tray for the ProDesk Mini[^ProDeskRackmount] - Sliding rails for the ProLiant[^Sliderail] - 1U universal rack mount for the APC[^Rail] +- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28 - ViewSonic N1700W LCD, used as a rack console via the KVM - RJ45 Cat6 in-line couplers[^Couplers] - Cat6 patch cables[^Patchcables] diff --git a/docs/images/README.md b/docs/images/README.md index 23805bf..e562603 100644 --- a/docs/images/README.md +++ b/docs/images/README.md @@ -25,6 +25,15 @@ carries a `renderer` series in two legends: that is the `capture` profile container taking the screenshot, which exists only for the duration of a capture, which is why the inventory below it lists six containers and not seven. +`ups-power.png` is out of date in one specific way, and knowingly so. It was +shot before a battery pack was fitted to `mjolnir` on 2026-08-28, so it still +shows the old "No battery is installed in this UPS" banner and the +"(fabricated — no battery fitted)" panel titles. It stays in place rather than +being deleted — it is a real render of what the dashboard said that day, and the +root `README.md` says underneath it when it was taken and what has changed +since. Re-shoot it once the pack has passed a self-test, when the panels will +read measured values instead of unproven ones. + The Container inventory panel used to publish the absolute path of `compose.yaml` — and so a username — because it excluded fields by name and cAdvisor kept adding new ones. It now filters to an allowlist. That was caught diff --git a/docs/observability.md b/docs/observability.md index 900726d..65ee792 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -105,7 +105,7 @@ boot check. | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew, reboots | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware | -| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. No pack is fitted, so most of these read values the management card fabricates — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | +| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 but has not yet passed a self-test, so most of these still read values the card may be fabricating — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling, and the stack watching itself | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | diff --git a/docs/roadmap.md b/docs/roadmap.md index a5f8d16..730a85d 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -95,10 +95,12 @@ what left this one unfireable for months. [`restore-the-firewall.md`](runbooks/restore-the-firewall.md) stays a hypothesis until it has been restored onto a spare once. - **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.** - `mjolnir` has none, so a mains loss is an immediate hard shutdown of the rack. - The management card fabricates a healthy pack. **The `UpsSelfTestFailed` - silence must be deleted the moment the pack is fitted, not left to expire** — - the issue carries the command and the reasoning. + An APCRBC115 went into `mjolnir` on 2026-08-28. That is the hardware done, not + the issue: the management card reports a healthy pack whether or not it can see + one, so nothing yet separates a good fit from a bad one. **The `UpsSelfTestFailed` + silence is still in place until 2026-09-20, and now suppresses the only alert + that could tell you which this is** — delete it, then self-test, then enable + scheduled tests on the card. → [runbook](runbooks/fit-the-ups-battery.md) - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.** A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index c6bd0c2..e27ca0b 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -3,12 +3,32 @@ **One rack visit, in an order that matters — and one alert that has to be un-silenced by hand at the right moment.** -`mjolnir` has no battery pack fitted. A mains loss is an immediate hard shutdown -of the entire rack, and `docs/security.md` lists it under "Not currently -defended". The management card does not report the missing pack; it fabricates a -healthy one, so every alert keyed on charge, runtime or alarm count is quietly -dead. Only `UpsSelfTestFailed` and `UpsBatteryUnproven` in `ups.rules.yaml` can -see the real condition, because they read the one value the card will not fake. +> **Status — 2026-08-28: the pack is in, and nothing else is done.** +> +> The APCRBC115 was fitted. **Steps 3 to 6 are all outstanding**, and step 3 is +> the urgent one: `UpsSelfTestFailed` is still silenced until 2026-09-20, so +> right now the only alert that could report a faulty or badly seated new pack +> is suppressed — on a pack nobody has tested yet. +> +> The shelf and the switch move (step 2, items 1–4) were not done either, so +> [#110](https://github.com/Gerrrt/HomeLab/issues/110) is untouched and +> `prometheus` and `oracle` still go deaf on a mains cut. +> +> Step 1's baseline was not captured before the fit. It does not have to be +> re-derived: the pre-fit values are recorded in the table under step 1 and in +> the header of `ups.rules.yaml`, and comparing against those is what step 5 +> needs. + +`mjolnir` ran with no battery pack for the whole life of this stack. A mains +loss was an immediate hard shutdown of the entire rack. The management card +never reported the missing pack; it fabricated a healthy one, so every alert +keyed on charge, runtime or alarm count was quietly dead. Only +`UpsSelfTestFailed` and `UpsBatteryUnproven` in `ups.rules.yaml` can see the +real condition, because they read the one value the card will not fake. + +Fitting a pack does not by itself end any of that. A card that cannot see a pack +which *is* present reports the same fabricated values as one over an empty bay, +so the steps below are what turns a fitted battery into a proven one. The pack alone is not the whole fix. @@ -45,7 +65,11 @@ card's web UI for `mjolnir` at `10.0.99.10`. ## 1. Record the baseline before touching anything The point of this step is that the after-state is *provably* different rather -than assumed to be. Capture what the card claims today: +than assumed to be. If the pack is already in and this was not run first, use +the recorded pre-fit values in the table below as the baseline — they are the +observed readings from this UPS, and `ups.rules.yaml` carries the same five. + +Capture what the card claims: ```bash for m in upsBatteryStatus upsEstimatedChargeRemaining \ @@ -58,8 +82,8 @@ for m in upsBatteryStatus upsEstimatedChargeRemaining \ done ``` -Expected on a UPS with no pack fitted — every one of these is derived, not -measured: +Observed on this UPS with no pack fitted — every one of these was derived, +not measured: | Metric | Reads | Means | | --- | --- | --- | @@ -206,26 +230,44 @@ to settle before concluding anything. uplink to port 3 of the main switch, not the UPS. Nothing in this runbook touches VLAN configuration. -## Only when all of the above passes — flip the documents +## Flipping the documents — the first half is done, the second is not + +This happens in two passes, because "a pack is fitted" and "the UPS works" are +different claims and became true at different times. + +**Done on 2026-08-28**, when the pack went in: every file that asserted there is +no battery said something false from that moment, so they were moved to what was +actually true — a pack is fitted and unproven, with the outstanding silence and +self-test named. That is `ups-power.json`, `ups.rules.yaml`, `docs/security.md`, +`docs/observability.md`, `README.md`, `docs/roadmap.md` and `docs/hardware.md`. + +**Still to do, only once step 5 passes** — a separate commit made after +verification, not before: -Several files currently assert, correctly, that there is no battery. They become -wrong the moment this runbook succeeds, and they are a separate commit made -*after* verification, not before: +| File | What changes | +| --- | --- | +| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "Battery fitted — not yet proven" banner panel; strip "(unproven — self-test pending)" from the five panel titles and reset their descriptions | +| `stacks/observability/prometheus/rules/ups.rules.yaml` | Cut the header back to a short note. The rules themselves do not change | +| `docs/security.md` | The "Mains power loss" row, and the closing paragraphs of "The UPS reported a battery it did not have" | +| `docs/observability.md` | The `ups.rules.yaml` row of the rule-file table | +| `README.md` | The UPS screenshot and the note under it, and the "current top items" paragraph. Re-capture with `make screenshots` | +| `docs/roadmap.md` | Move #93 into Done. Existing entries stay as written; the roadmap is a record | + +And when the shelf is racked, which is a separate visit and separate issue: | File | What changes | | --- | --- | -| `stacks/observability/grafana/dashboards/ups-power.json` | Delete the "No battery is installed in this UPS" banner panel; strip "(fabricated — no battery fitted)" from the six panel titles and their descriptions | -| `stacks/observability/prometheus/rules/ups.rules.yaml` | Rewrite the header comment. The rules themselves do not change | -| `docs/security.md` | The "Mains power loss" row in the threat table, and the "The UPS reports a battery it does not have" section | -| `README.md` | The UPS dashboard screenshot alt text, and the "current top items" paragraph. Re-capture with `make screenshots` | | `docs/hardware.md` | The 1U shelf into the rack table at U4, and into Accessories | | `docs/network.md` | The TP-Link note — now racked and on UPS power | -| `docs/roadmap.md` | Move #93 and #110 into Done. Existing entries stay as written; the roadmap is a record | +| `docs/roadmap.md` | Move #110 into Done | One coupling to know about before you start editing: **deleting the banner panel changes the dashboard's panel count**, and `scripts/check_docs.py` asserts every -panel count quoted in prose against the live JSON. Move the numbers in the same -commit or CI will fail. Run `make validate` before pushing. - -Then close [#93](https://github.com/Gerrrt/HomeLab/issues/93) and -[#110](https://github.com/Gerrrt/HomeLab/issues/110). +panel count quoted in prose against the live JSON. The banner was retitled +rather than removed on 2026-08-28 for exactly that reason — the count did not +move. When you do delete it, 84 becomes 83 in both `README.md` and +`stacks/observability/README.md`, in the same commit, or CI will fail. Run +`make validate` before pushing. + +Then close [#93](https://github.com/Gerrrt/HomeLab/issues/93), and +[#110](https://github.com/Gerrrt/HomeLab/issues/110) when the shelf follows. diff --git a/docs/security.md b/docs/security.md index 7ea8885..5c65e23 100644 --- a/docs/security.md +++ b/docs/security.md @@ -15,7 +15,7 @@ What this network is actually built to survive: | A corporate laptop carrying something in from outside | Sits on VLAN 50 but has no management access | | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | Losing visibility of a failure | 35 alert rules, 30 days of metrics and logs | -| Mains power loss | **Not currently defended.** `mjolnir` has no battery installed — see below | +| Mains power loss | **Partly defended.** A pack was fitted to `mjolnir` on 2026-08-28 and has not yet proven it can carry the load — see below | What it explicitly does **not** defend against: a determined attacker with physical access to the rack, a supply-chain compromise in an upstream container @@ -155,26 +155,40 @@ strings at mode 664 in `~/.local/state/nvim/undodir/` were a real finding *because* the disk beneath them is readable. On an encrypted disk that is a much smaller problem. Neither fact is interesting alone. -### The UPS reports a battery it does not have +### The UPS reported a battery it did not have -`mjolnir` has no battery installed. Its Network Management Card nonetheless -reports 100% state of charge, 48.0 VDC, a battery temperature, an hour of -runtime, a 2030 replacement date, and `upsAlarmsPresent = 0`. Every one of those -values is derived rather than measured. +Until 2026-08-28 `mjolnir` had no battery installed. Its Network Management Card +nonetheless reported 100% state of charge, 48.0 VDC, a battery temperature, an +hour of runtime, a 2030 replacement date, and `upsAlarmsPresent = 0`. Every one +of those values was derived rather than measured. The single honest signal it emits is the self-test result. The management card -renders it as **Refused — internal fault**; over SNMP it is +rendered it as **Refused — internal fault**; over SNMP it is `upsTestResultsSummary = 4` (aborted), from the standard UPS-MIB the `apc_ups` module already walks. No extra OIDs were needed to see it. -Any alert rule keyed on charge, runtime or alarm count will therefore never -fire, no matter how bad things get. `UpsSelfTestFailed` and `UpsBatteryUnproven` -in `ups.rules.yaml` key on the self-test instead, and are the only two rules in +Any alert rule keyed on charge, runtime or alarm count therefore could not fire, +no matter how bad things got. `UpsSelfTestFailed` and `UpsBatteryUnproven` in +`ups.rules.yaml` key on the self-test instead, and are the only two rules in that file that can detect this condition. This is worth stating carefully: the monitoring did not fail, and neither did the rules. The device lied, and the rules trusted it. +**A pack was fitted on 2026-08-28, and that has not yet closed this.** A card +that cannot see a pack which *is* present — badly seated, or faulty out of the +box — emits the same five fabricated values as one sitting over an empty bay. So +a healthy-looking dashboard distinguishes nothing; only a passing self-test, and +charge and runtime that have moved off the pre-fit baseline, do. + +Neither has happened yet. No self-test has been run since the fit, so +`upsTestResultsSummary` still holds its pre-fit `4`. And `UpsSelfTestFailed` is +still silenced in Alertmanager until 2026-09-20 — a silence that was correct +while nothing could be done about a missing pack, and which now suppresses the +one rule that would report the new pack being bad. Deleting it is step 3 of +[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), it comes +before the self-test rather than after, and it is outstanding. + ### Why SNMPv2c is still a weak point The devices are polled with SNMPv2c, which transmits the community string in diff --git a/stacks/observability/grafana/dashboards/ups-power.json b/stacks/observability/grafana/dashboards/ups-power.json index 81c0ffe..f83df1e 100644 --- a/stacks/observability/grafana/dashboards/ups-power.json +++ b/stacks/observability/grafana/dashboards/ups-power.json @@ -1,7 +1,7 @@ { "uid": "homelab-ups", "title": "UPS & Power", - "description": "APC Smart-UPS (mjolnir) via UPS-MIB. Note: this UPS currently has no working battery, so runtime figures are not to be trusted until it is replaced — see docs/roadmap.md.", + "description": "APC Smart-UPS (mjolnir) via UPS-MIB. A replacement pack was fitted on 2026-08-28 and has not yet passed a self-test, so runtime figures are not proven — see docs/runbooks/fit-the-ups-battery.md.", "tags": [ "homelab", "power", @@ -258,7 +258,7 @@ }, { "type": "text", - "title": "No battery is installed in this UPS", + "title": "Battery fitted 2026-08-28 — not yet proven", "gridPos": { "x": 0, "y": 6, @@ -267,7 +267,7 @@ }, "options": { "mode": "markdown", - "content": "**Every battery figure on this dashboard is fabricated.** `mjolnir` has no battery pack fitted, yet its management card still reports `batteryNormal`, 100% charge, 48.0 V and a runtime estimate. It raises no alarm (`upsAlarmsPresent = 0`).\n\nA mains loss is therefore an **immediate hard shutdown of the rack**, with no warning from here. Treat charge, runtime, battery status and battery temperature below as decoration until a pack is fitted — every alert in `ups.rules.yaml` is dead for the same reason.\n\nDo **not** run a UPS self-test to get an honest reading: a self-test transfers the load to battery, and with none fitted that is the outage it is meant to warn about. Tracked in `docs/roadmap.md`." + "content": "**A pack is fitted. Nothing has yet confirmed the UPS can carry load.** An APCRBC115 cartridge went into `mjolnir` on 2026-08-28. Until a self-test passes, every battery figure below is unproven rather than measured: a management card that cannot see a fitted pack reports `batteryNormal`, 100% charge, 48.0 V and a runtime estimate exactly as it did when the bay was empty.\n\nThe test is comparison, not appearance. Charge still reading exactly `100` and runtime still at exactly `63` is the signature of a card that is still fabricating — compare against the pre-fit baseline in step 5 of `docs/runbooks/fit-the-ups-battery.md`.\n\n**Two steps are outstanding.** The `UpsSelfTestFailed` silence `54f1715c` runs until 2026-09-20 and is suppressing the one alert that can report a faulty or badly seated new pack — delete it *before* running the self-test, not after. Then enable scheduled self-tests on the card, so `upsTestResultsSummary` stays live evidence rather than a stale last result." } }, { @@ -361,8 +361,8 @@ }, { "type": "gauge", - "title": "Battery charge (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Battery charge (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -497,8 +497,8 @@ }, { "type": "stat", - "title": "Runtime remaining (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Runtime remaining (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -567,8 +567,8 @@ }, { "type": "stat", - "title": "Battery status (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Battery status (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -791,8 +791,8 @@ }, { "type": "stat", - "title": "Battery temp (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Battery temp (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" @@ -873,8 +873,8 @@ }, { "type": "timeseries", - "title": "Battery charge and runtime (fabricated — no battery fitted)", - "description": "No battery pack is fitted to mjolnir. The management card reports this value anyway; it does not reflect any hardware. See the banner at the top of this dashboard.", + "title": "Battery charge and runtime (unproven — self-test pending)", + "description": "A pack was fitted to mjolnir on 2026-08-28 and has not yet passed a self-test. The management card reports this value whether or not it can see the pack, so it is not yet evidence of anything — compare it against the pre-fit baseline in docs/runbooks/fit-the-ups-battery.md before trusting it. See the banner at the top of this dashboard.", "datasource": { "type": "prometheus", "uid": "prometheus" diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index c4dd572..ed9acc7 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -3,53 +3,73 @@ # # READ THIS BEFORE TRUSTING ANY ALERT BELOW. # -# The UPS has no battery pack fitted — see docs/roadmap.md. A mains loss is an -# immediate hard shutdown of the whole rack. Every rule here is deliberately -# loud and routed to the `urgent` receiver by category=power in +# An APCRBC115 pack was fitted on 2026-08-28. Until then the UPS had none and a +# mains loss was an immediate hard shutdown of the whole rack. Every rule here +# is deliberately loud and routed to the `urgent` receiver by category=power in # alertmanager.yaml. # -# None of them can fire for that condition. The management card does not report -# the missing pack; it fabricates a healthy one. Observed on a UPS with no -# battery physically present: +# A fitted pack is not yet a proven one, and here that distinction is not +# pedantry. The management card never reported the missing pack; it fabricated +# a healthy one. Observed on this UPS with no battery physically present: # # upsBatteryStatus 2 (batteryNormal) # upsEstimatedChargeRemaining 100 (percent) -# upsEstimatedMinutesRemaining 63 (minutes of runtime it does not have) +# upsEstimatedMinutesRemaining 63 (minutes of runtime it did not have) # upsBatteryVoltage 480 (48.0 V) # upsAlarmsPresent 0 (no alarm raised) # -# So UpsBatteryLow, UpsChargeLow, UpsRuntimeCritical and UpsAlarmPresent are -# all comfortably inside their thresholds and will stay there. These rules are -# kept because they become correct the moment a pack is fitted — not because -# they are protecting anything today. A quiet `ups` group is not evidence of -# runtime. +# A card that cannot see a pack which *is* fitted — one badly seated, or dead on +# arrival — reports those same five values. So the numbers looking healthy is +# not evidence; only their having moved off that pre-fit baseline is. Step 5 of +# docs/runbooks/fit-the-ups-battery.md is that comparison, and it has not been +# made yet. +# +# Until it has, UpsBatteryLow, UpsChargeLow, UpsRuntimeCritical and +# UpsAlarmPresent sit comfortably inside their thresholds and still tell you +# nothing. They are kept because they become correct the moment the card is +# reading real hardware — not because they are protecting anything yet. A quiet +# `ups` group is not evidence of runtime. # # UpsOnBattery, UpsOverloaded and UpsBatteryTemperatureHigh read mains-side or -# load-side values and are unaffected. UpsOnBattery in particular would still -# fire on a transfer — it would just be reporting an outage already in progress -# rather than warning of one. +# load-side values and were unaffected throughout. UpsOnBattery in particular +# would still fire on a transfer — it would just be reporting an outage already +# in progress rather than warning of one. # # THE ONE METRIC THAT DOES NOT LIE. # # upsTestResultsSummary is the exception to everything above. It reports the -# result of the last diagnostics test, and the UPS will not fake one — with no -# pack fitted it cannot complete a test, so it reports the failure honestly +# result of the last diagnostics test, and the UPS will not fake one — without a +# usable pack it cannot complete a test, so it reports the failure honestly # while charge, runtime, voltage and alarm count all read healthy. # # 1 donePass 2 doneWarning 3 doneError # 4 aborted 5 inProgress 6 noTestsInitiated # -# Observed on this UPS: 4 (aborted). The management card's own diagnostics page -# renders the same state as "Refused — internal fault". +# Observed before the fit: 4 (aborted), which the management card's own +# diagnostics page rendered as "Refused — internal fault". No self-test has been +# run since the pack went in, so that remains the last stored result and this +# metric has not yet had anything new to say. # # UpsSelfTestFailed and UpsBatteryUnproven below are keyed on it, and they are -# the only two rules in this file that can detect the current condition. +# the only two rules in this file that can tell a UPS which works from one that +# merely looks like it does. +# +# WHAT IS STILL OUTSTANDING, AND WHY THIS HEADER IS NOT SHORTER YET. +# +# UpsSelfTestFailed is silenced in Alertmanager until 2026-09-20 +# (54f1715c-e57b-4322-8a6d-5435bc8e1bd8). That silence was correct while nothing +# could be done about a missing pack — it fired every 30 minutes about a known +# condition, which is how an urgent receiver stops being read. It stopped being +# correct the moment a pack was fitted, because it now suppresses the single +# rule that could report that pack being faulty or badly seated. Delete it +# before running the self-test, not after: run them in the other order and the +# one result most worth seeing is the one that is hidden. # -# Still do not run a UPS self-test by hand to force a reading. A test transfers -# the load to battery; with a pack fitted but weak, that is the outage it exists -# to warn about. The refusal above was safe only because no transfer occurred. -# Once a pack is in, enable the NMC's *scheduled* self-test instead so this -# metric stays live evidence rather than a stale last-known result. +# Still do not run self-tests by hand to keep the metric warm. A test transfers +# the load to battery, and on a pack that is weak that is the outage the alert +# exists to warn about. The pre-fit refusals were safe only because no transfer +# ever occurred. Enable the NMC's *scheduled* self-test instead, so this metric +# stays live evidence rather than a stale last-known result. # # Known limit: a value of 1 (donePass) is the last result, not a fresh one. If # the NMC stops testing entirely the value simply stops changing, and nothing @@ -156,8 +176,9 @@ groups: 2=doneWarning 3=doneError 4=aborted. A UPS that cannot complete a self-test cannot be assumed to carry the load, whatever its charge and runtime metrics claim. Check the management card's diagnostics - page — "Refused / internal fault" here means no pack is fitted or - the pack is not seen. + page — "Refused / internal fault" means the pack is not seen, and + now that one is fitted that is a finding about the pack or its + seating rather than about an empty bay. - alert: UpsBatteryUnproven # 6 = noTestsInitiated. Not a safe resting state: it is the state in