fix(backup): quiesce, timestamp, encrypt and verify the volume backup

[Gerrrt]

Closes #64.

make backup had five defects in seven inline lines, plus two the issue did not
name. It had also never been run on the monitoring host — backups/ did not
exist.

What was wrong

#	Defect	Fix
1	Fixed filename; tar truncates at open(2), so a failed run destroyed the last good backup	Timestamped set directories; MANIFEST written last marks a set complete; pruning runs only after the new set verifies
2	Nothing verified the result — tar's exit status was the whole of the quality control	Decrypt, read the whole stream as a tar, require the volume's own discriminating sentinel
3	alloy-data missing from the hardcoded list	The volume list, mount points and services to stop are derived from compose.yaml
4	prometheus-data tarred while Prometheus was writing it	docker compose stop by default, with a trap that restarts on every exit path; --hot opts out and marks the set unproven
5	No restore target, no runbook	make restore and docs/runbooks/restore-the-stack.md
6	backups/ bind-mounted into a root container, so archives were root:root and unrotatable	tar writes to stdout, age writes the file as the operator; backups/ never enters a container
7	docker run -v missing_vol:/data creates an empty volume — a wrong prefix yields five plausible 45-byte archives and exit 0	Project name derived from name: in compose.yaml, then every volume inspected up front

Also: .PHONY: backup sat above backup-firewall:, so backup-firewall was
never phony and backup's declaration was seven lines from its target.

Decisions worth reviewing

Quiesce by default. Rejected --web.enable-admin-api for the TSDB snapshot
endpoint: it also exposes delete-series on a port bound to 0.0.0.0:9090. A
~90s nightly gap fits inside the dead-man's-switch heartbeat and is the only
variant the runbook can honestly claim.

age, not sops. sops holds the whole document in memory and base64s it
into YAML — free for a 6 KB config.xml, a gigabyte of RSS for a 1 GB TSDB.
Recipient still read from .sops.yaml, so there is still exactly one key.
grafana.db carries the admin password hash, every API token and every
datasource credential, so this is not optional.

Debian archiver, pinned as a profiled compose service. BusyBox tar cannot
report file changed as we read it, and that exit status is the only mechanical
evidence a --hot run has that it wrote a torn archive. It earned the choice
immediately — a --hot run reported three of five volumes torn, where the old
recipe exited 0 and called it a backup. Pinning it in a script instead would be
a pin Dependabot never sees and make pin-digests never touches.

The sentinel is load-bearing. age -r has no associated data, so
loki-data.tar.gz.age and prometheus-data.tar.gz.age are interchangeable as
far as age is concerned. The sentinel is the only thing binding a filename to
its content, so an archive carrying another volume's sentinel is fatal in both
modes.

What was actually tested

Not just "it ran". On 2026-08-29, each archive was extracted into a scratch
volume under a throwaway project name and the pinned image started against it:

• Prometheus reported every block healthy, replayed its WAL in 1.26s without
  error, and answered instant queries for series months older than the backup.
• Grafana connected to the restored database, found the schema already
  current (710 migrations skipped) and restored its plugin cache.
• Loki returned label values and real firewall log lines from an hour that
  closed before the backup was taken.
• Ownership survived: 65534, 10001 and 472 — what Prometheus, Loki and
  Grafana need in order to write.

Also verified: Ctrl-C and SIGTERM mid-archive both leave six services running
(exit 130 / 143); a corrupted byte fails the AEAD check; swapped archives are
caught by the sentinel; KEEP=1 prunes correctly while keeping the newest
quiesced set and leaving incomplete sets alone.

A whole-stack restore has still never been performed, and the runbook's
closing section says so, names what the cheap check does and does not prove, and
specifies the experiment that would close the gap.

Out of scope

• Scheduling (Nothing schedules any backup, digest check or key verification #77) and getting the sets off the host they protect (Get the firewall backup off prometheus, and buy a spare ProDesk to rehearse the restore #92) both
  stay open; the closing banner points at them.
• stop_grace_period on prometheus/alertmanager — -t 60 fixes the backup
  path, but make down still SIGKILLs at 10s. Stack-wide lifecycle change,
  its own PR.
• backup-firewall.sh uses relative paths and only works from the repo root.

One drive-by: CI's "no decrypted artefact is tracked" loop omitted backups/
while validate.sh checked it, and backup-firewall.sh:21-22 claims both do.
Now both do.

🤖 Generated with Claude Code

[Gerrrt]

The whole-stack restore has now been done

The runbook called §3 a hypothesis. It isn't one any more. A set was restored
into a scratch compose project and all six services were brought up on it, with
§4 run against the result including the negative assertions.

What it established

Claim	Evidence
The TSDB restores and serves	Every block healthy, WAL replayed in ~1s, count(up) returned 12 at stamp−1d and stamp−7d
The restore actually happened	result:[] at stamp+30m, +1h and +3h — a genuine gap — while the same stack scraped 12 targets live
grafana.db is intact	Table-for-table match against the live DB; admin row created=2026-08-17, twelve days before the restore
Loki serves the restored store	Range query over an hour that closed before the stamp returned suricata/filterlog lines from morpheus
Alertmanager state came back	nflog and silences present with their original 06:41 mtimes
Ownership survives	65534 / 10001 / 472

Two things per-volume testing had not found

Grafana will not start without internet. GF_INSTALL_PLUGINS makes the
background installer contact grafana.com on every start and a failure there is
fatal — it crash-loops, even though both plugins are already in the restored
volume. On a host that has lost its uplink, which is an ordinary disaster, the
restore succeeds and Grafana still does not come up. Now a > [!CAUTION] on the
rebuilt-host path. Worth its own issue: GF_INSTALL_PLUGINS is also deprecated
in favour of GF_PLUGINS_PREINSTALL, and I did not want to change stack startup
behaviour inside a backup PR.

Alloy replays, and my §6b was wrong to call that a failure. Restoring
alloy-data puts the log positions back, so Alloy re-reads from them and
re-ships lines carrying their original timestamps — ~4,600 duplicates inside
the hour before the stamp within four minutes of start-up, then steady across
three samples. The restored lines themselves never moved. §6b now measures the
replay and names the count that must not move, instead of asserting one that
always does.

§4's Grafana check also changed: under unified storage the provisioned
dashboards aren't in the legacy dashboard table, and this lab has no
annotations, so both checks I'd suggested were vacuous. It now reads the admin's
created column straight out of grafana.db — no credential needed, and it
doesn't depend on the password differing.

Two fixes that fell out of running it

• Both scripts now honour COMPOSE_PROJECT_NAME, as compose itself does.
  This was a real bug: deriving the prefix only from the file's name: key meant
  that with COMPOSE_PROJECT_NAME set, backup would archive one project's
  volumes while compose ran another's — and restore would then overwrite the
  wrong ones. It's also what makes the rehearsal possible.
• restore-volumes.sh no longer writes an empty pre-restore snapshot, or claims
  to hold "the only copy" of what it replaced, when the target volumes don't
  exist yet.

Safety of the rehearsal

The rehearsal network is internal: true. That is the argument, not a detail —
without it the rehearsal Alertmanager would send to the real notification
channels and ping the real dead-man's-switch heartbeat, and the rehearsal
snmp-exporter would poll production devices. The live stack was untouched
throughout and is still up. The runbook now carries the overlay and the
reasoning.

Still not proven

Restoring in place over the live project and running make up — which
renders config and hot-reloads, and takes no overlay. Smaller than the gap this
closed, but not zero.