fix(backup): quiesce, timestamp, encrypt and verify the volume backup

.github/workflows/ci.yml

@@ -266,7 +266,7 @@ jobs:
       - name: Assert no decrypted artefact is tracked
         run: |
           fail=0
-          for pattern in '.env' '.rendered/' '.purge-secrets.txt' 'certificates/'; do
+          for pattern in '.env' '.rendered/' '.purge-secrets.txt' 'certificates/' 'backups/'; do
             if git ls-files | grep -E "(^|/)${pattern//./\\.}" | grep -v '\.env\.example'; then
               echo "::error::tracked file matching '${pattern}' — it must be gitignored"
               fail=1

.gitignore

@@ -33,12 +33,18 @@ certificates/
 stacks/*/snmp-exporter/mibs/
 
 # ---- Backups ----
-# `make backup` writes stack volumes here; `make backup-firewall` writes the
-# SOPS-encrypted pfSense config here. Neither belongs in git. The firewall
-# config in particular carries the WAN address, the full rule set and user
+# `make backup` writes age-encrypted volume archives to backups/volumes/<stamp>/,
+# and `make restore` writes a pre-restore snapshot of what it is about to
+# replace to backups/volumes/.pre-restore-<stamp>/. `make backup-firewall`
+# writes the SOPS-encrypted pfSense config to backups/firewall/. None of it
+# belongs in git.
+#
+# The firewall config carries the WAN address, the full rule set and user
 # password hashes — docs/security.md says those are deliberately unpublished,
 # and encryption does not change that a public repository is the wrong place
-# for them. See scripts/backup-firewall.sh.
+# for them. grafana-data is the same argument: it holds the admin password
+# hash, every API token and every datasource credential. See
+# scripts/backup-firewall.sh.
 backups/
 
 # ---- Runtime state produced by the stack ----

Makefile

@@ -262,21 +262,36 @@ screenshots: ## Render the dashboards to docs/images/ (stack must be up)
 	@# what to look for.
 	./scripts/capture-screenshots.sh $(STACK)
 
-.PHONY: backup
+.PHONY: backup-firewall
 backup-firewall: ## Pull morpheus's pfSense config and encrypt it to ./backups/
 	@# The single largest unmitigated failure in the estate is morpheus dying
 	@# with no config export. Output is gitignored and never committed — see
 	@# the header of scripts/backup-firewall.sh for why.
 	./scripts/backup-firewall.sh $(ARGS)
 
-backup: ## Back up the stack's volumes to ./backups/
-	@mkdir -p backups
-	@for v in prometheus-data loki-data grafana-data alertmanager-data; do \
-		printf 'backing up %s\n' "$$v"; \
-		docker run --rm -v $(STACK)_$$v:/data -v "$(PWD)/backups:/backup" \
-			alpine tar czf "/backup/$$v.tar.gz" -C /data . ; \
-	done
-	@printf '\033[0;32mwrote backups/\033[0m\n'
+.PHONY: backup
+backup: ## Quiesce the stack, archive its volumes to ./backups/ and verify
+	@# Thin on purpose. This target used to BE the implementation, and every
+	@# defect in #64 followed from that: one fixed output filename that tar
+	@# truncated at open, so the only way to lose a backup was to take one; a
+	@# hardcoded volume list that had silently skipped alloy-data since Alloy
+	@# was added; an unpinned `alpine`; a hot copy of an open TSDB; and no
+	@# verification beyond tar's exit status.
+	@#
+	@# The volume list and the services to stop are now derived from
+	@# compose.yaml, so a sixth volume cannot be forgotten. STACK goes in the
+	@# environment rather than positionally: the script's arguments are flags.
+	STACK=$(STACK) ./scripts/backup-volumes.sh $(ARGS)
+
+.PHONY: restore
+restore: ## Restore the stack's volumes from a backup set (ARGS="--from <stamp>")
+	@# Deliberately a separate script from `backup`. One script that both writes
+	@# archives and overwrites live volumes is one mistyped flag from an outage,
+	@# and scripts/backup-firewall.sh — the model for both — is non-destructive
+	@# throughout. The volume inventory is not duplicated: restore-volumes.sh
+	@# reads `backup-volumes.sh --inventory`, the way every SNMP tool reads
+	@# scripts/snmp-targets.sh.
+	STACK=$(STACK) ./scripts/restore-volumes.sh $(ARGS)
 
 .PHONY: purge-history-dry-run
 purge-history-dry-run: ## Preview the git-history secret purge (safe)

README.md

@@ -65,9 +65,9 @@ incident.
 - **Supply chain pinned by digest.** Every image carries both a tag and a
   `sha256:` digest, so a moved tag cannot change what deploys. CI enforces it;
   `make pin-digests` re-resolves them from the registry.
-- **Documented decisions and runbooks.** Eight ADRs covering what was chosen and
-  what was rejected — including the costs accepted knowingly; nine runbooks for
-  the operations that are easy to get wrong at 1am.
+- **Documented decisions and runbooks.** Eleven ADRs covering what was chosen
+  and what was rejected — including the costs accepted knowingly; twelve
+  runbooks for the operations that are easy to get wrong at 1am.
 
 ## Architecture
 
@@ -164,7 +164,8 @@ rack; a dashed border means egress only. Full topology and data flow in
 │   ├── observability.md  security.md  roadmap.md
 │   ├── adr/                  # 11 architecture decision records
 │   └── runbooks/             # deploy, add device, rotate creds, certs, key backup,
-│                             #   purge, restore the firewall, ship firewall logs,
+│                             #   purge, restore the firewall, restore the stack,
+│                             #   ship firewall logs, verify the alert path,
 │                             #   enable suricata, fit the UPS battery
 └── Makefile                  # make help
 ```
@@ -195,7 +196,8 @@ $ make help
   secrets-edit     Edit the encrypted secrets in $EDITOR
   secrets-verify-backup  Check a backup age key decrypts the secrets
   validate         Run every check CI runs
-  backup           Back up the stack's volumes to ./backups/
+  backup           Quiesce the stack, archive its volumes to ./backups/ and verify
+  restore          Restore the stack's volumes from a backup set
   ...
 ```
 

docs/observability.md

@@ -236,9 +236,11 @@ make ps                    # container status
 make logs SERVICE=grafana  # tail one service
 make reload                # hot-reload Prometheus, Alertmanager, snmp-exporter
 make validate              # everything CI runs
-make backup                # tar the data volumes into ./backups/
+make backup                # quiesce, archive, encrypt and verify the data volumes
+make restore ARGS=--list   # the backup sets that exist
 make down                  # stop, keep data
-make nuke                  # stop, destroy data (prompts)
+make nuke                  # stop, destroy data (prompts) — recoverable, see
+                           #   docs/runbooks/restore-the-stack.md
 ```
 
 Prometheus and Alertmanager are started with lifecycle endpoints enabled, so

docs/roadmap.md

@@ -93,7 +93,11 @@ what left this one unfireable for months.
   off `prometheus`, and buy a spare ProDesk.** A backup on the same shelf as the
   thing it protects is not a backup, and
   [`restore-the-firewall.md`](runbooks/restore-the-firewall.md) stays a
-  hypothesis until it has been restored onto a spare once.
+  hypothesis until it has been restored onto a spare once. The volume sets
+  `make backup` writes have exactly the same defect: they sit on the host they
+  protect. Unlike the firewall, they have now been restored — the whole stack
+  was brought up on a restored set on 2026-08-29 and verified. Getting a copy
+  off this host is the part that is still missing.
 - **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.**
   An APCRBC115 went into `mjolnir` on 2026-08-28 and passed its self-test the
   same day: `upsTestResultsSummary` `4` → `1`, `upsBatteryVoltage` off its
@@ -141,7 +145,10 @@ what left this one unfireable for months.
 - **[#77](https://github.com/Gerrrt/HomeLab/issues/77) Schedule something.**
   Nothing runs `make backup`, `make backup-firewall`, `make check-digests` or
   `make secrets-verify-backup` on a timer. The last of those is the only proof
-  the secrets are recoverable.
+  the secrets are recoverable, and `make restore ARGS=--dry-run` is now the
+  only proof the volumes are — `make backup ARGS='--verify-only --all'` is the
+  cheap version to put on the timer, because it is what catches bit-rot in a
+  set nobody has touched for a week.
 
 ## Decided but not built
 

docs/runbooks/deploy-stack.md

@@ -86,7 +86,8 @@ make up
 ```
 
 Data volumes survive `make down` and `make up`. Only `make nuke` destroys them,
-and it prompts.
+it prompts, and it is recoverable from a backup set — see
+[`restore-the-stack.md`](restore-the-stack.md).
 
 ## Troubleshooting
 
@@ -103,9 +104,19 @@ and it prompts.
 ## Backups
 
 ```bash
-make backup      # tars each data volume into ./backups/
+make backup                                 # quiesce, archive, encrypt, verify
+make backup ARGS=--list                     # the sets that exist
+make restore ARGS="--dry-run --from latest" # prove the newest one is readable
 ```
 
-Prometheus and Loki data is reproducible-ish (it re-accumulates), but Grafana's
-volume holds annotations and users. The dashboards themselves are in git, so a
-lost Grafana volume is an inconvenience rather than a loss.
+`make backup` stops the services for the length of the copy — about ninety
+seconds — because a copy of a live store is not a backup. It writes one
+timestamped, age-encrypted, verified archive per volume into
+`backups/volumes/<STAMP>/`, keeps the seven newest complete sets, and prunes
+only after the new one has verified.
+
+Prometheus and Loki data re-accumulates, so losing it is a hole in the record
+rather than a loss of function. `grafana-data` is the exception and the reason
+this matters: the dashboards are in git, but the users, the annotations, the
+admin password and every UI edit that was never exported exist only in that
+volume. Restoring any of it is [`restore-the-stack.md`](restore-the-stack.md).