Skip to content

docs(ups): record a pack that is fitted and not yet proven - #159

Merged
Gerrrt merged 1 commit into
mainfrom
claude/battery-installed-n06dxp
Aug 28, 2026
Merged

docs(ups): record a pack that is fitted and not yet proven#159
Gerrrt merged 1 commit into
mainfrom
claude/battery-installed-n06dxp

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Aug 28, 2026

Copy link
Copy Markdown
Owner

What changed

The documents and the UPS dashboard stop saying mjolnir has no battery, because as of 2026-08-28 it has one. They do not start saying the UPS works — they say a pack is fitted and unproven, and they name the two steps that would settle it.

Nothing about the running stack changes: no rule, expression, threshold, route or receiver is edited. The ups.rules.yaml rules are byte-for-byte the same, as that file was designed for.

Why

#93. The APCRBC115 went in today, so every file asserting an empty bay became false at that moment and had to move. But the runbook merged in #158 gates the full flip on verification, and verification has not happened — the pack was plugged in and nothing else.

That gap is not pedantry here. A management card that cannot see a pack which is present — badly seated, or faulty out of the box — emits exactly the five values it emitted over an empty bay: batteryNormal, 100% charge, 63 minutes, 48.0 V, upsAlarmsPresent = 0. A healthy-looking dashboard separates those two worlds not at all. Only a passing self-test, and charge/runtime that have moved off the pre-fit baseline, do.

The urgent part is not in this diff, because it cannot be. The UpsSelfTestFailed silence 54f1715c-e57b-4322-8a6d-5435bc8e1bd8 runs until 2026-09-20 and is now suppressing the one rule that could report a bad new pack. The runbook predicted this exact window; we are inside it. It lives in Alertmanager's state, not in the repo, so this PR names it everywhere it is relevant instead of fixing it:

curl -sS -X DELETE \
  http://localhost:9093/api/v2/silence/54f1715c-e57b-4322-8a6d-5435bc8e1bd8

Delete that before running the self-test, not after.

Detail worth flagging:

  • The dashboard banner is retitled, not deleted. It now reads "Battery fitted 2026-08-28 — not yet proven" and carries the remaining steps. Keeping the panel holds the count at 84, so the check_docs.py panel-count coupling the runbook warns about is not triggered by this commit — it is deferred to the one that deletes the banner.
  • The five (fabricated — no battery fitted) panel titles become (unproven — self-test pending).
  • ups.rules.yaml keeps its long header rather than being trimmed. The pre-fit readings stay in it deliberately: step 1's baseline capture was never run live, so those recorded values are the baseline step 5 compares against.
  • security.md keeps the finding and its lesson — "the device lied, and the rules trusted it" — and stops claiming the bay is empty. "Mains power loss" moves from not defended to partly defended, which is the honest reading of an untested pack.
  • The screenshot is left alone. It is a real render of what the dashboard said on 2026-08-22, and docs/images/README.md refuses mocked-up images; both READMEs now say what it predates and when to re-shoot it.
  • #110 is untouched — the shelf and the switch move did not happen, so prometheus and oracle still go deaf on a mains cut. The runbook's flip list is split three ways: done now, done after the self-test, done after the shelf.

Blast radius

Documentation and one Grafana dashboard's cosmetic fields (titles, descriptions, one text panel). No host, VLAN or service behaviour changes. Grafana re-provisions the dashboard from the file; the panel count and every PromQL expression are unchanged.

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

Verification

Run in this environment, which has no route to the lab:

$ python3 scripts/check_docs.py
docs OK — 35 Prometheus + 13 Loki rules, 5 dashboards, 84 panels, 6 assertions

$ python3 scripts/check_dashboards.py
5 dashboards OK (84 panels, 83 PromQL expressions)

$ yamllint stacks/observability/prometheus/rules/ups.rules.yaml
(clean)

promtool and markdownlint-cli2 are not installed here, so those two legs of make validate were not run — CI covers both. The rules file is unchanged below its header comment, and MD013 is disabled in .markdownlint-cli2.yaml.

  • make validate passes — partially: check_docs, check_dashboards and yamllint pass locally; promtool and markdownlint left to CI
  • Deployed to the lab and confirmed working — not done, and it is the point of the PR. Delete the silence, run one self-test, confirm upsTestResultsSummary = 1 from both the NMC and Prometheus, compare charge and runtime against the pre-fit baseline, then enable scheduled self-tests
  • Docs updated (docs/, service README, or docs/roadmap.md)

Generated by Claude Code

The APCRBC115 went into `mjolnir` today. Every file that asserted there is
no battery became false at that moment, so they move now rather than
waiting — but they move to *fitted and unproven*, not to *working*, because
nothing has yet demonstrated the second.

The distinction is the whole point of the previous commit's runbook. A
management card that cannot see a pack which is present — badly seated, or
faulty out of the box — emits the same five fabricated values as one sitting
over an empty bay: `batteryNormal`, 100%, 63 minutes, 48.0 V, no alarm. A
healthy-looking dashboard therefore separates nothing. Only a passing
self-test and a charge that has moved off the pre-fit baseline do, and
neither has been done.

What is outstanding is also what is dangerous. The `UpsSelfTestFailed`
silence 54f1715c-e57b-4322-8a6d-5435bc8e1bd8 runs until 2026-09-20 and is
now suppressing the one rule that could report a bad new pack — the runbook
predicted exactly this window, and we are in it. Deleting that silence
cannot be done from a commit, so every file that touches the subject names
it instead:

- the dashboard banner is retitled rather than deleted, so it now says a
  pack is fitted and untested and carries the two remaining steps. Keeping
  the panel means the count stays 84 and the check_docs.py coupling the
  runbook warned about is not triggered by this commit
- the five "(fabricated — no battery fitted)" panel titles become
  "(unproven — self-test pending)", which is what they now are
- `ups.rules.yaml` keeps its long header. The pre-fit readings stay because
  they are the baseline step 5 compares against, and step 1 was never run
  live; the rules themselves are untouched, as designed
- `security.md` keeps the finding and its lesson, and stops claiming the
  bay is empty. "Mains power loss" goes from not defended to partly
  defended, which is the honest reading of an untested pack

The screenshot is left as it is: it is a real render of what the dashboard
said on 2026-08-22, and `docs/images/README.md` refuses mocked-up images.
Both READMEs say what it predates and when to re-shoot it.

#110 is untouched — the shelf and the switch move did not happen, so the
monitoring path still has no battery behind it. The runbook's flip list is
split accordingly: done now, done after the self-test, done after the shelf.

Refs #93, #110

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015ipPWFSYkEX3MtP226HzR1
@Gerrrt
Gerrrt merged commit 4299dc0 into main Aug 28, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants