feat(alerts): three notification channels, and security routes on its own

[Gerrrt]

What changed

urgent, default and security become three real destinations instead of
three names for one URL, and category: security gets its own routes at both
severities.

Matches	Receiver	First notification	Repeats
critical + category=power	urgent	immediately	30m
critical + category=security	security	immediately	1h
warning + category=security	security	30s	4h
critical (anything else)	urgent	10s	4h
warning (anything else)	default	30s	12h
info	null	never	—

Two new SOPS keys are required before make up will run —
ALERTMANAGER_URGENT_WEBHOOK_URL and ALERTMANAGER_SECURITY_WEBHOOK_URL. See
"Before deploying" at the bottom.

Why

Closes #66.

urgent and default both read url_file: .../webhook_url, so the routing tree
differentiated timing and not destination. Timing is not the part you
notice: per-topic sound and do-not-disturb settings live on the receiving end, so
only a separate destination can make "this one is urgent" audible. A UPS on
battery and a warning about scrape latency arrived in the same place, differing
only in how often they repeated.

And nothing routed on category except power, while ten rules carry
category: security — SshBruteForce, TerminalSegmentReachedInternalNetwork,
IoTAttemptedLateralMovement, SuricataHighPriorityAlert and the rest. Routed on
severity alone, the warning-severity half of that list went to the default
channel on a 12-hour repeat, indistinguishable from a disk filling up.
category: power having its own route was already the precedent.

Blast radius

• No change to network segmentation or firewall rules
• No new port published to a VLAN that could not already reach the service
• No credential added outside secrets/*.sops.yaml

Alertmanager only. Two new keys go in the encrypted secrets file; nothing else
in the stack is touched. Worth stating explicitly: until the new keys exist,
render-config.sh refuses to render anything at all, so a half-configured
deploy is not reachable.

Verification

make validate — all pass, one skip (gitleaks not installed locally).

The routing assertions are the point of this PR

amtool check-config proves the tree parses and that every route names a
receiver that exists. It cannot see route order — and the tree is
first-match-wins with no continue anywhere, so order is the entire design.

I mutated the config to confirm that gap is real, taking the security warning
route and demoting it below the bare severity = "warning" route:

$ amtool check-config am-broken.yaml
Checking 'am-broken.yaml'  SUCCESS

$ amtool config routes test --config.file=am-broken.yaml \
    --verify.receivers=security severity=warning category=security
default
WARNING: Expected receivers did not match resolved receivers.
exit=1

check-config passes; every warning-severity security alert silently goes to the
wrong channel. So scripts/validate.sh and .github/workflows/ci.yml now assert
each row of the table above with amtool config routes test --verify.receivers:

  PASS  amtool check-config
  PASS  amtool config routes test (6 assertions)

Both places in the same commit — make validate and CI have drifted before
(#68).

render-config.sh

The three URLs render from a list rather than three copies of the same four
lines, and every url_file: in alertmanager.yaml is cross-checked against that
list. That check matters because a missing url_file is not a config error:
Alertmanager reads it at notify time, so the stack starts, check-config passes,
and the first real alert is what discovers it.

Exercised end to end on the host with throwaway URLs supplied via the
environment, so nothing went near the encrypted file:

-- rendering 3 alertmanager receiver URL(s)
-- writing observability/.env
-- done — observability is ready to start

Three files, 0600, no trailing newline (a newline inside a URL is a delivery
error, not a warning). And with a deliberate typo in one url_file:

error: alertmanager.yaml reads typo_url, which nothing in AM_CHANNELS renders
exit=1

The cross-check also caught a bug in its own first draft — a bare
secrets/[a-z_]+ grep matched secrets/observability.sops.yaml in the file's
own header comment and reported the comment as a missing channel. It is anchored
on url_file: now, and the comment says why.

Not deployed

• make validate passes
• Deployed to the lab and confirmed working — deliberately not, see below
• Docs updated — docs/observability.md gains a ### Routing section with
  the table; secrets/README.md, the compose.yaml mount comment and the
  secrets example all catch up

Before deploying

make secrets-edit     # add ALERTMANAGER_URGENT_WEBHOOK_URL and ALERTMANAGER_SECURITY_WEBHOOK_URL
make up

With ntfy, three topics on one server is the whole setup — the topic is the last
path segment and needs no creation step. Then confirm all three arrive
separately:

docker exec alertmanager ls /etc/alertmanager/secrets/
amtool alert add --alertmanager.url=http://localhost:9093 \
  alertname=RouteTest severity=warning category=security

Pointing all three at the same topic works and breaks nothing — but then the
receiver names are lying again, which is what this PR fixed, so the secrets
example says to record that decision in docs/observability.md if you make it.

🤖 Generated with Claude Code

[Gerrrt]

Merged, not deployed — the two new SOPS keys have to be filled in first. Tracked with the exact commands and the current host state in #120.