Fail PostgreSQL backups before publishing incomplete dumps

[MrStarktastic]

The PostgreSQL backup job can report success when pg_dumpall fails because POSIX sh returns the gzip pipeline status. That publishes an empty archive and then prunes older backups.

Run dumping and compression separately, require nonempty SQL with the cluster-dump completion marker, validate gzip before atomic publication, and clean private temporary files on failure. Retention runs only after successful publication. Pin the client to PostgreSQL18.6 and a verified digest, matching the production server major. Add a regression check to CI.

Validation:

• Four regression tests cover seven success/failure scenarios; the original script failed five of them. Configured pre-commit, embedded-script shellcheck, pinned-runtime shell parsing and actual-value Helm render passed.
• Homepage coverage, Jellyfin compatibility and README image-reference checks passed.
• The rendered script generated a new31.6MiB production dump into disposable storage. Strict full restore in the exact production image with networking disabled recovered all nine databases,542 user tables and658,606 rows, with no warnings. This is not a value-by-value comparison with changing production.
• Independent review found no Critical/Important issues. Production backup storage had roughly368GiB free; this dump needed about181MiB additional raw+compressed staging. SIGKILL/node loss may leave private temporary files, which need inspection before manual cleanup.

All temporary pods, restore containers and private dumps/logs were removed. Production PostgreSQL stayed2/2Ready with zero restarts. The scheduled CronJob and retained backups, including the known invalid archive, remain unchanged until merge/Argo sync. After rollout, run and inspect one fresh CronJob-derived backup. Same-NAS archives still need independent backup protection.

Evidence: docs/superpowers/reports/2026-09-21-postgres-backup-reliability.md.

[github-actions]

Static Validation

✅ YAML Lint: Passed
✅ Kubeconform: Passed

Summary

./infrastructure/system/crowdsec/manifests/prometheus-rules.yaml - PrometheusRule crowdsec-alerts is valid
./infrastructure/controllers/authentik/manifests/ldap-outpost-service.yaml - Service ak-outpost-ldap-outpost is valid
./infrastructure/controllers/authentik/manifests/ldap-outpost-deployment.yaml - Deployment ak-outpost-ldap-outpost is valid
./infrastructure/system/intel-device-operator/manifests/gpu-device-plugin.yaml - intel-gpu GpuDevicePlugin skipped
Summary: 85 resources found in 78 files - Valid: 84, Invalid: 0, Errors: 0, Skipped: 1

✅ Homepage Coverage: Passed
✅ Alert Policy: Passed

---

Argo CD Diff Preview

Summary:

Modified (1):
± base-configs (+17|-9)

base-configs (bootstrap/appsets/config-apps.yaml)

CronJob: databases/pg-backup

             - name: PGPASSWORD
               valueFrom:
                 secretKeyRef:
                   key: postgres-password
                   name: postgres-admin-secret
-            image: postgres:alpine
+            image: postgres:18.6-alpine@sha256:6c538e7206ea40ff740ef27883529390a690b6ead6ba96b44c67a9f7c638e8fd
             name: pg-backup
             volumeMounts:
             - mountPath: /backup
               name: backup-storage
             - mountPath: /scripts

ConfigMap: databases/pg-backup-script

 apiVersion: v1
 data:
   backup.sh: |
     #!/bin/sh
-    set -e
+    set -eu
+    umask 077
 
     BACKUP_DIR=/backup
     RETENTION_DAYS=7
     TIMESTAMP=$(date -u +%Y-%m-%d_%H%M%S)
     DUMP_FILE="${BACKUP_DIR}/pg_dumpall_${TIMESTAMP}.sql.gz"
 
     echo "Starting PostgreSQL backup at ${TIMESTAMP}"
 
+    # Separate commands preserve producer failures on POSIX sh (no pipefail).
+    SQL_FILE="${DUMP_FILE}.sql.tmp"
+    trap 'rm -f "$SQL_FILE" "${DUMP_FILE}.tmp"' EXIT
+    trap 'exit 1' HUP INT TERM
+
     echo "Running pg_dumpall..."
-    if pg_dumpall --clean --if-exists | gzip > "${DUMP_FILE}.tmp"; then
-      mv "${DUMP_FILE}.tmp" "${DUMP_FILE}"
-      SIZE=$(du -h "${DUMP_FILE}" | cut -f1)
-      echo "  ✓ Backup saved: ${DUMP_FILE} (${SIZE})"
-    else
-      echo "  ✗ pg_dumpall failed!"
-      rm -f "${DUMP_FILE}.tmp"
+    pg_dumpall --clean --if-exists > "$SQL_FILE"
+    if [ ! -s "$SQL_FILE" ] || ! grep -q '^-- PostgreSQL database cluster dump complete$' "$SQL_FILE"; then
+      echo "  ✗ Empty or incomplete PostgreSQL dump" >&2
       exit 1
     fi
+    gzip -c "$SQL_FILE" > "${DUMP_FILE}.tmp"
+    gzip -t "${DUMP_FILE}.tmp"
+    mv "${DUMP_FILE}.tmp" "$DUMP_FILE"
+    rm -f "$SQL_FILE"
+    SIZE=$(du -h "$DUMP_FILE" | cut -f1)
+    echo "  ✓ Backup saved: ${DUMP_FILE} (${SIZE})"
 
     echo "Cleaning backups older than ${RETENTION_DAYS} days..."
     find "${BACKUP_DIR}" -name "pg_dumpall_*.sql.gz" -mtime +${RETENTION_DAYS} -print -delete
 
     echo "Current backups:"

Stats:
[Applications: 2], [Full Run: 55s], [Rendering: 2s], [Cluster: 20s], [Argo CD: 33s]