feat(alerts): a dead man's switch on the notification path

[Gerrrt]

What changed

One rule — Watchdog, expr: vector(1) — that fires unconditionally and
forever, plus a heartbeat receiver and the routing to make its absence
observable in two independent ways.

Stacked on #118. Base is feat/three-notification-channels, because both
touch alertmanager.yaml, render-config.sh, verify-key-backup.sh and the
secrets example. GitHub will retarget this to main when #118 merges.

One new SOPS key: ALERTMANAGER_HEARTBEAT_URL — a cron-monitor ping URL,
not an ntfy topic. See "Before deploying".

Why

Closes #67.

AlertmanagerNotificationsFailing catches delivery errors — a refused
connection, a 5xx. It cannot catch a webhook URL that is well-formed, reachable,
and pointed at nothing: a 200 into a deleted ntfy topic is a successful
notification by every measure Alertmanager has.

Not hypothetical. The webhook was the ntfy.example.invalid placeholder for the
entire life of the stack and nothing noticed, because the only symptom is that
alerts stop arriving — which is also what a healthy week looks like.

The repository already applies this reasoning elsewhere. security.rules.yaml has
FirewallLogsStopped because a firewall passing traffic for the whole house is
never silent for thirty minutes, and it spends ten lines explaining why there is
deliberately no SuricataStopped rule: "absence of alerts is indistinguishable
from absence of the service", and detecting that needs a heartbeat rather than a
threshold. The notification path was the one place the argument had never been
turned on itself. That comment now points at the worked example.

How it works

Watchdog's firing carries no information. Its absence is the entire signal,
and one continue: true — the only one in the routing tree — sends one rule to
two places on two cadences:

Route	Destination	Cadence	Catches
heartbeat	external cron-monitor ping	5m	Prometheus stopped evaluating, Alertmanager died, no outbound network
default	the real alert channel	24h	the alert channel itself is a 200 into nothing

Neither half substitutes for the other. The heartbeat proves delivery to a
different URL than real alerts use, so it cannot see a deleted topic. The daily
notification travels the identical URL your warnings travel, but nothing
machine-checks its absence — you do. The watcher lives off this host by
necessity: one here fails at the same moment as the thing it watches.

severity: none is load-bearing

Not a placeholder, and the rule comment says so at length:

1. It keeps the alert out of the three bare severity routes. With info it
   would be swallowed by the "null" receiver and the switch would be silently
   disarmed.
2. It keeps the alert out of both inhibit_rules. The second matches
   target_matchers: [severity =~ "warning|critical"] with equal: ["instance"]
   — and Alertmanager treats absent == absent as equal, so a Watchdog
   carrying a warning severity and no instance label would be silenced every
   single time InstanceDown fired.

A dead man's switch that goes quiet exactly when a host goes down is worse than
none: it reports the path healthy right up to the moment it stops, then looks
like the host's fault.

Blast radius

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

One new Prometheus rule (one extra series from vector(1); cardinality impact is
nil), one new Alertmanager receiver, one new SOPS key. The external watcher is
configured outside this repo.

Verification

make validate — all pass, one skip (gitleaks not installed locally).

  PASS  promtool check rules
  PASS  promtool test rules
  PASS  amtool check-config
  PASS  amtool config routes test (7 assertions)

The test has no paired quiet case, deliberately

containers.test.yaml pairs every firing case with a silent one, because a test
that only ever expects silence would have passed against the unfireable rule in
#63. Watchdog is the mirror image: vector(1) is unconditional by design, so
there is no input that should silence it and a quiet case would be asserting a
bug. The header comment says this rather than leaving it to be wondered about.

What is under test instead is the label set, which is the part that can break
silently. Confirmed by mutation — changing severity: none to severity: warning:

$ promtool test rules stacks/observability/prometheus/tests/watchdog.test.yaml
FAILED:
    alertname: Watchdog, time: 1m,
        exp:[ Labels:{alertname="Watchdog", category="monitoring",
              component="watchdog", severity="none"} ...
exit=1

The routing assertions gained a row for the same reason, and it is the only one
expecting two receivers:

$ amtool config routes test --verify.receivers=heartbeat,default \
    alertname=Watchdog severity=none category=monitoring
heartbeat,default
exit=0

That row is also what would catch a severity: info slip — it would resolve to
null, and the assertion fails.

render-config.sh

Exercised on the host with throwaway URLs supplied via the environment, so
nothing went near the encrypted file:

-- rendering 4 alertmanager receiver URL(s)
-- writing observability/.env
-- done — observability is ready to start

Rule counts confirmed by counting, not by assuming: 35 metric, 13 Loki.

• make validate passes
• Deployed to the lab and confirmed working — not yet, needs the key below
• Docs updated — new runbook
  verify-the-alert-path.md, a
  ### The dead man's switch section in docs/observability.md, and the
  live rule counts moved 34 → 35 / 47 → 48 in five places

docs/roadmap.md:182 is deliberately left at "34 rules" — it records what was
built at the time rather than a current count, and #72 owns that drift.

Before deploying

make secrets-edit     # set ALERTMANAGER_HEARTBEAT_URL
make up

The ping URL comes from something that lives off this host — healthchecks.io
(free tier is enough for one check), Cronitor, or an Uptime Kuma push monitor on
another machine. Set the check's period to 5m and grace to 15m; the runbook
explains why those are tied to repeat_interval and what breaks if you move one
without the other.

Point that check's own notification somewhere that is not this stack's
webhook. If both land on the same ntfy topic, a deleted topic takes out the alert
and the warning about the alert together.

Then, once, actually watch it trip — a dead man's switch nobody has seen go red
is indistinguishable from one that does not work:

docker stop alertmanager    # external check must go DOWN within the grace window
docker start alertmanager   # must return green within one repeat_interval

🤖 Generated with Claude Code

[Gerrrt]

Merged, not deployed — ALERTMANAGER_HEARTBEAT_URL has to be filled in first, and the external check registered off-host. Tracked in #120; the full procedure is in docs/runbooks/verify-the-alert-path.md.