feat(ops): scheduled Postgres backups to restic/S3 with a tested restore

[tibroc]

Closes #89.

docs/04 §5 asks for "regular Postgres backups + a tested restore"; there was no backup automation anywhere in the repo, so durability rested entirely on the pgdata volume never being lost. This adds the org's standard pattern — pg_dump → restic → S3, with a restic forget --prune retention policy — and rewrites the restore runbook around it.

What's here

A new opt-in backup service in compose.prod.yaml (profile backup) running deploy/backup/backup.sh:

pg_dump -Fc → /tmp/wolke-<utc>.dump → restic backup --tag wolke-db → restic forget --prune → sleep

Deliberately boring: one pinned image (docker.io/library/postgres:17-alpine) and a POSIX shell loop. No cron daemon, no scheduler, no second published image. The postgres:*-alpine base is the load-bearing choice — pg_dump then always matches the server's major version, which an older client refuses to do. restic is apk added at container start rather than baked into a separately built image; see Trade-offs below.

Optional, for real

Without --profile backup the service does not exist:

$ podman-compose --env-file <no backup vars> -f compose.prod.yaml config     → ['app', 'caddy', 'postgres']
$ podman-compose --env-file <no backup vars> -f compose.prod.yaml --profile backup config
                                                                            → ['app', 'backup', 'caddy', 'postgres']

Note the second line: with the profile on but nothing configured, compose still parses. That's on purpose — compose interpolates the whole file before it applies profiles, so the ${VAR:?…} fail-closed form used for SESSION_SECRET and friends would abort compose up for every deployment that never enables backups. The backup variables therefore use :-, and the fail-closed check moved into backup.sh, which refuses to start and says why.

It never fails quietly

Every failure path logs a distinct ERROR and exits non-zero, so the container crash-loops under restart: unless-stopped instead of silently skipping backups. backup cycle ok is logged only when pg_dump, restic backup and restic forget --prune all succeeded. All six paths were exercised (transcripts below).

Nothing institution-specific

RESTIC_REPOSITORY, RESTIC_PASSWORD, AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION, BACKUP_INTERVAL_SECONDS (86400), BACKUP_RUN_ON_START (true), BACKUP_KEEP_DAILY/WEEKLY/MONTHLY (7/4/6), BACKUP_TAG (wolke-db), BACKUP_INIT_REPO (true), BACKUP_PROBE_TIMEOUT (45), BACKUP_RESTIC_VERSION — all env, documented with defaults in .env.example, pointed at from config.example.yaml, and tabulated in README → Backups.

Networking

backend (to reach Postgres) plus a new egress network for the S3 endpoint. Deliberately not edge — that's the subnet the app trusts for X-Forwarded-* headers, and a backup job has no business being in it. Postgres stays internal-only.

Runbook

docs/runbooks/restore-postgres.md is rewritten around list snapshots → restic restore → pg_restore, all inside the backup container (it already has the right pg_restore major version, the repository credentials, and network reach to both S3 and Postgres). The bare-dump procedure is kept as Fallback: restore from a bare dump file. The stale "there is no automated backup job in this repo" banner is gone, and docs/04 §5 now records what shipped.

---

Verification

Stood the production stack up locally with MinIO as the S3 bucket, took a real backup, destroyed the pgdata volume, and restored following the runbook.

Honest caveats about the rehearsal, up front:

1. Run with podman-compose 1.x + podman 5.8.4, not docker compose (this workstation has no docker). Same flags.
2. Against a copy of compose.prod.yaml differing in exactly one line — edge's subnet moved from 172.28.0.0/16 to 172.31.0.0/16, because another project on this machine already owns 172.28/16. diff confirmed that was the only difference. The backup service, the egress network and every backup path are byte-identical to what's committed.
3. Two mounts needed :ro,z added locally for SELinux (rootless podman on Fedora). Pre-existing: compose.prod.yaml already carries :ro,z on the Caddyfile mount but not on ./config.yaml. Not changed here — flagging it as a separate small papercut for podman/SELinux deployers.
4. A mock IdP and a published app port were added via an override file so the app could start and /readyz be reachable without Caddy.

1. A real backup

2026-08-28T13:06:02Z INFO  no repository at s3:http://minio:9000/wolke-backups yet — initialising it
created restic repository 79ac8b75 at s3:http://minio:9000/wolke-backups
2026-08-28T13:06:02Z INFO  repository initialised
2026-08-28T13:06:02Z INFO  pg_dump -Fc wolke@postgres/wolke → /tmp/wolke-20260828T130602Z.dump
2026-08-28T13:06:02Z INFO  dump ok (33972 bytes)
2026-08-28T13:06:02Z INFO  restic backup --tag wolke-db
Added to the repository: 34.131 KiB (9.089 KiB stored)
snapshot 3257a349 saved
2026-08-28T13:06:03Z INFO  restic forget --prune (keep daily:7 weekly:4 monthly:6)
Applying Policy: keep 7 daily, 4 weekly, 6 monthly snapshots
2026-08-28T13:06:04Z INFO  backup cycle ok

State at backup time (seeded catalog + a marker user with a favorite):

       t       | count
---------------+-------
 services      |     7
 categories    |     9
 role_defaults |    12
 favorites     |     1
 users         |     1

2. Destroy the database volume

$ podman rm -f wolkeverify_postgres_1
$ podman volume rm wolkeverify_pgdata
wolkeverify_pgdata
$ podman volume ls | grep wolkeverify
wolkeverify_miniodata          # the backup repo — untouched
$ ... up -d postgres && psql -U wolke -d wolke -c '\dt'
Did not find any relations.

3. Restore, following the runbook verbatim

Step 2 of the runbook — list snapshots:

ID        Time                 Host        Tags        Paths                             Size
---------------------------------------------------------------------------------------------------
3257a349  2026-08-28 13:06:02  wolke       wolke-db    /tmp/wolke-20260828T130602Z.dump  33.176 KiB
1 snapshots

Step 3 — restore + pg_restore, copy-pasted from the runbook:

restoring snapshot 3257a349 of [/tmp/wolke-20260828T130602Z.dump] at 2026-08-28 13:06:02 by root@wolke to /tmp/restore
Summary: Restored 2 files/dirs (33.176 KiB) in 0:00
restoring /tmp/restore/tmp/wolke-20260828T130602Z.dump (33972 bytes)
DROP DATABASE
CREATE DATABASE

Step 4 — start the app:

{"level":"INFO","msg":"migrations applied","count":0}
{"level":"INFO","msg":"database connected"}
{"level":"INFO","msg":"server listening","addr":":8080"}

4. Verify the restore actually worked

$ curl -fsS http://localhost:18080/readyz
{"status":"ready"}

Row counts — identical to pre-destruction:

       t       | count
---------------+-------
 services      |     7
 categories    |     9
 role_defaults |    12
 favorites     |     1
 users         |     1

Marker row survived, is_admin intact:

         oidc_sub          |   display_name    | is_admin
---------------------------+-------------------+----------
 restore-rehearsal-subject | Restore Rehearsal | t

Catalog contents:

BigBlueButton / Identitätsmanagement / MyShare / Stud.IP / VPN / Webmail / WLAN an der UOS

Runbook step 6 — backup service resumed, next cycle clean:

2026-08-28T13:07:58Z INFO  repository opened
2026-08-28T13:07:59Z INFO  dump ok (33948 bytes)
2026-08-28T13:08:00Z INFO  backup cycle ok

5. Every failure mode, exercised

Scenario	Actual log output
Enabled, unconfigured	ERROR missing required environment: RESTIC_REPOSITORY RESTIC_PASSWORD PGHOST PGUSER PGPASSWORD PGDATABASE / the backup service is enabled but not configured — refusing to start
S3 endpoint unreachable	ERROR cannot reach the backup repository: … / check RESTIC_REPOSITORY and that the S3 endpoint is reachable from the egress network
Wrong S3 credentials	ERROR the backup repository rejected our credentials: …Stat: The Access Key Id you provided does not exist in our records. / check AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_DEFAULT_REGION
Wrong RESTIC_PASSWORD	ERROR RESTIC_PASSWORD does not open this repository / refusing to continue — a wrong password must never silently start a second repository
Postgres unreachable	ERROR pg_dump failed — is postgres reachable and the password correct?
No route to Alpine mirrors	ERROR could not install restic (no network to the Alpine mirrors?)

All six then log backup cycle FAILED — no snapshot was written; exiting so the restart policy surfaces this and exit 1.

One finding worth calling out: restic retries backend errors with exponential backoff for ~15 minutes before giving up, so a typo'd bucket name would have sat silent for a quarter of an hour. The repository probe is therefore wrapped in timeout ${BACKUP_PROBE_TIMEOUT:-45}, and the failure classes are told apart by pattern so each gets an actionable message.

shellcheck -s sh deploy/backup/backup.sh (v0.10.0): clean, no findings.

---

Trade-offs worth a reviewer's opinion

apk add restic at container start, rather than a purpose-built image. The alternative is a Dockerfile.backup (postgres:17-alpine + COPY --from=restic/restic) published by CI to a second ghcr package. That would keep read_only: true and drop the runtime network dependency, at the cost of a new workflow, a second image to version, and an operator requirement to have built it. The chosen path keeps the moving parts in the repo to one shell script and one pinned upstream image, and the failure is loud and at start-up. Two consequences a reviewer should weigh:

• The backup container is the only service without read_only: true (apk writes across /usr, /lib, /etc). no-new-privileges and cap_drop: ALL are kept.
• It needs outbound access to the Alpine mirrors at start-up, not just to S3. BACKUP_RESTIC_VERSION pins the package if reproducibility matters.

Happy to switch to a CI-built image if you'd rather have the hardening back.

postgres:17-alpine is pinned to the major, not a patch tag — matching how postgres:17 is pinned for the server in the same file. Pinning the dumper tighter than the server would create a mismatch to maintain.

Auto-init defaults to on. Convenient, but a typo'd RESTIC_REPOSITORY would silently start a second empty repository rather than erroring. BACKUP_INIT_REPO=false opts out; happy to flip the default if you'd rather it be explicit.

Not wired into compose.yaml (the staging/build-from-source stack) — the issue and docs/04 scope this to production. Easy to add if you want staging backed up too.

Left unmerged for supervisor review.

[tibroc]

Supervisor review — looks good; merge is deliberately left to @timb for manual review.

Highlights verified: fail-closed script-level env validation (with the correct compose interpolation-vs-profiles rationale), the bounded repo probe with per-failure-class messages incl. refusing to init on a wrong password, empty-dump guard, tag-scoped retention, egress network separated from edge (X-Forwarded-* trust), cap_drop+no-new-privileges, and the MinIO destroy-and-restore rehearsal.

Two advisory follow-ups, non-blocking:

1. The dump + restic cache live in an unsized tmpfs (/tmp) — fine today, but click_events/usage growth makes an explicit size (e.g. tmpfs: /tmp:size=512m) or a small named volume worth a line.
2. No periodic restic check — consider every Nth cycle or a documented quarterly operator task, since snapshots are otherwise only proven readable at restore time.

The pre-existing SELinux mount finding is now #94 (M4).