#118 and #119 are merged but not deployed. Both add SOPS keys that only you
can fill in, so make up will refuse to run until they exist. This is the
intended behaviour — render-config.sh dies naming the missing keys rather than
starting a stack with half a routing tree — but it means the change is sitting
in main doing nothing.
The three keys
| Key |
What it is |
ALERTMANAGER_URGENT_WEBHOOK_URL |
ntfy topic for criticals |
ALERTMANAGER_SECURITY_WEBHOOK_URL |
ntfy topic for category=security |
ALERTMANAGER_HEARTBEAT_URL |
cron-monitor ping URL, not an ntfy topic |
For the first two, three topics on one ntfy server is the whole setup — the topic
is the last path segment and needs no creation step. The point of separate topics
is the per-topic sound and do-not-disturb settings on the phone; pointing them all
at the same topic works and breaks nothing, but then the receiver names are lying
again, which is what #66 was about.
The third comes from something that lives off this host — healthchecks.io
(free tier covers one check), Cronitor, or an Uptime Kuma push monitor on another
machine. A watcher on the monitoring host fails at the same moment as the thing
it is watching.
Then verify
docs/runbooks/verify-the-alert-path.md is the full procedure. The short version:
docker exec alertmanager ls /etc/alertmanager/secrets/ # four files
amtool alert add --alertmanager.url=http://localhost:9093 \
alertname=RouteTest severity=warning category=security # lands on the security topic
Set the external check's period to 5m and grace to 15m — both are tied to
repeat_interval on the heartbeat route, and the runbook says what breaks if you
move one without the other. Point that check's own notification somewhere that is
not this stack's webhook; if both land on the same topic, a deleted topic
takes out the alert and the warning about the alert together.
Then, once, actually watch it trip. A dead man's switch nobody has seen go red is
indistinguishable from one that does not work:
docker stop alertmanager # external check must go DOWN inside the grace window
docker start alertmanager # must return green within one repeat_interval
Until then — the current state, precisely
The running Alertmanager still holds the old config in memory (verified via
/api/v2/status: three receivers, default and urgent both on webhook_url),
because it only re-reads on reload or restart and neither has happened. Every
alert is being delivered normally right now.
alertmanager/.rendered/ holds only webhook_url, unchanged. Nothing points at
a placeholder — throwaway URLs used to exercise render-config.sh were removed
after testing.
The one gap: alertmanager.yaml is a bind mount, so an unexpected restart
before the keys are filled in would load the new five-receiver config and find
urgent_url, security_url and heartbeat_url missing. Warnings on the
default route would still deliver; criticals and security alerts would not.
Uptime is currently 3.85 days with RestartCount=0 and restart: unless-stopped,
so this needs a reboot or a manual restart to happen — but do not make reload
or restart the stack before running make secrets-edit.
Filed from the session that merged #117, #118 and #119.
#118 and #119 are merged but not deployed. Both add SOPS keys that only you
can fill in, so
make upwill refuse to run until they exist. This is theintended behaviour —
render-config.shdies naming the missing keys rather thanstarting a stack with half a routing tree — but it means the change is sitting
in
maindoing nothing.The three keys
ALERTMANAGER_URGENT_WEBHOOK_URLALERTMANAGER_SECURITY_WEBHOOK_URLcategory=securityALERTMANAGER_HEARTBEAT_URLFor the first two, three topics on one ntfy server is the whole setup — the topic
is the last path segment and needs no creation step. The point of separate topics
is the per-topic sound and do-not-disturb settings on the phone; pointing them all
at the same topic works and breaks nothing, but then the receiver names are lying
again, which is what #66 was about.
The third comes from something that lives off this host — healthchecks.io
(free tier covers one check), Cronitor, or an Uptime Kuma push monitor on another
machine. A watcher on the monitoring host fails at the same moment as the thing
it is watching.
Then verify
docs/runbooks/verify-the-alert-path.mdis the full procedure. The short version:Set the external check's period to 5m and grace to 15m — both are tied to
repeat_intervalon the heartbeat route, and the runbook says what breaks if youmove one without the other. Point that check's own notification somewhere that is
not this stack's webhook; if both land on the same topic, a deleted topic
takes out the alert and the warning about the alert together.
Then, once, actually watch it trip. A dead man's switch nobody has seen go red is
indistinguishable from one that does not work:
Until then — the current state, precisely
The running Alertmanager still holds the old config in memory (verified via
/api/v2/status: three receivers,defaultandurgentboth onwebhook_url),because it only re-reads on reload or restart and neither has happened. Every
alert is being delivered normally right now.
alertmanager/.rendered/holds onlywebhook_url, unchanged. Nothing points ata placeholder — throwaway URLs used to exercise
render-config.shwere removedafter testing.
The one gap:
alertmanager.yamlis a bind mount, so an unexpected restartbefore the keys are filled in would load the new five-receiver config and find
urgent_url,security_urlandheartbeat_urlmissing. Warnings on thedefaultroute would still deliver; criticals and security alerts would not.Uptime is currently 3.85 days with
RestartCount=0andrestart: unless-stopped,so this needs a reboot or a manual restart to happen — but do not
make reloador restart the stack before running
make secrets-edit.Filed from the session that merged #117, #118 and #119.