Move fuzekeys-data/logs off ephemeral local-path; migrate the SQLite datastore to FuzeInfra-provisioned Postgres

[izzywdev]

@claude

FuzeKeys stores its data on node-local, ephemeral local-path volumes with reclaimPolicy: Delete, pinned to an autoscaler-reapable elastic node. /app/data appears to hold a SQLite database — i.e. a self-hosted database engine on a volume that is designed to be thrown away. This is the highest-risk item of the set.

What was verified (live prod, 2026-09-01/02)

Read-only via FuzeInfra's cluster-query workflow — runs 33564509285 (PVCs), 33564639316 (pods), 33564766594 (PVs + node affinity).

PVC	Size	StorageClass	Node pin	Reclaim	Status
fuzekeys/fuzekeys-data	5Gi	local-path	fuzeinfra-prod-elastic-v2-9b31f718	Delete	Bound
fuzekeys/fuzekeys-logs	2Gi	local-path	fuzeinfra-prod-elastic-v2-9b31f718	Delete	Bound

Both are mounted by fuzekeys-backend at /app/data and /app/logs (deploy/helm/fuzekeys/templates/backend.yaml). fuzekeys-backend pods are Running on that same node.

Correction to an earlier report: these volumes were described as pinned to node fuzeinfra-elastic-918c8556, "which no longer exists". That is not the current pin. Both PVs are pinned to fuzeinfra-prod-elastic-v2-9b31f718, which is alive and is currently running your backend. The data is therefore most likely still present — treat it as live data, back it up, and do not assume it is already lost.

Why this is still acute, and arguably worse than reported: fuzeinfra-prod-elastic-v2-* is an elastic node. FuzeInfra's billing-aware Contabo autoscaler releases idle elastic nodes ahead of their renewal date. Combined with reclaimPolicy: Delete and node-pinned local-path, the volume is destroyed with the node and the PV is not retained. There is no scheduled event forcing this — which is exactly why it should be fixed before one happens.

Remedy — split by volume type

These two volumes need different remedies. Classify each explicitly.

fuzekeys-data (/app/data) — if it is the SQLite datastore, migrate to a FuzeInfra-provisioned Postgres. A repo should not run its own database engine, and an embedded file-backed DB on an ephemeral volume is the same problem in a smaller package. backend/app/main.py, backend/simple_server.py, README.md and SITES_DATABASE_SUMMARY.md all reference SQLite; backend/env.example and deploy/README.md already reference Postgres, so the path may be partly built.

• Mechanism (already live in prod, not dormant): helm/fuzeinfra/templates/service-db-provisioning.yaml driven by .Values.serviceDatabases. In helm/fuzeinfra/values-contabo.yaml the entries for fuzesales, fuzecontact, fuzeservice, mendys, fuzequality, fuzehub and fuzesocial are enabled: true today — use them as working precedents. (An earlier report that every entry is enabled: false describes only the base values.yaml.)
• Your side: seal the password Secret for the fuzeinfra namespace and PR the serviceDatabases entry to FuzeInfra. FuzeKeys already set this precedent once — FuzeKeys DS extraction: recurring UI pattern (ds-fp:70b388e46003) #136 is cited in FuzeInfra's own values as the model for the mendys hand-off.
• Docs: docs/consuming-repos/MONGODB_PROVISIONING.md, docs/consuming-repos/CHROMADB_PROVISIONING.md. A Postgres equivalent is being written in parallel and may land shortly after this issue — check for it first.
• If /app/data turns out to hold key material rather than a database, say so — that changes the remedy to Longhorn plus a sealed-secret/Vault review, and it should not be silently moved into Postgres.

fuzekeys-logs (/app/logs) — this is not a database. Logs should go to the shared Loki stack, not a PVC. If a local buffer is genuinely required, it belongs on Longhorn (now the cluster default StorageClass, reclaimPolicy: Retain) — never local-path.

Both PVCs already parameterize the class (deploy/helm/fuzekeys/templates/pvc.yaml reads .Values.persistence.data.storageClass / .Values.persistence.logs.storageClass, currently ""). Note that storageClassName is immutable on an existing PVC — changing the value alone will not move existing data. You must create a new volume and copy.

Acceptance criteria

1. BACK UP fuzekeys-data FIRST, before any chart change, node operation or volume deletion. Verify the backup is restorable — a dump you have not read back is not a backup. Do this before anything else in this issue.
2. Each volume explicitly classified: shared-FuzeInfra-Postgres / Longhorn app-data / Loki. Written down in the repo.
3. If migrating to Postgres: serviceDatabases entry PR'd to FuzeInfra, password sealed into the fuzeinfra namespace, app wired via service DNS (fuzeinfra-postgres.fuzeinfra.svc) and DATABASE_URL from a SealedSecret. Never hardcode a secret.
4. No FuzeKeys PVC remains on local-path. Anything that stays a volume is on longhorn.
5. Data is verified present in its new home before the old PVC is deleted.
6. Prod is GitOps. Every change lands via a merged PR that Argo syncs. Never kubectl patch or kubectl edit a live prod resource — Argo selfHeal reverts out-of-band changes within seconds.
7. Verify against the live cluster with FuzeInfra's cluster-query workflow (repository_dispatch type cluster-query, using the FUZEINFRA_DISPATCH_TOKEN you already hold) — docs/consuming-repos/CLUSTER_QUERY.md. Do not report done from chart diffs alone.

STATE:

• Why: Two local-path + reclaimPolicy: Delete PVCs node-pinned to an elastic node the billing-aware autoscaler can reap; /app/data looks like a SQLite datastore. Same class as the 2026-07-24 incident where node-local volumes were wiped and Neo4j was lost.
• Done: Live prod census verified. Owning repo confirmed izzywdev/FuzeKeys via argocd/projects/fuzekeys.yaml sourceRepos. Node pin re-verified and the earlier "node no longer exists" claim corrected. Chart confirmed to parameterize storageClass via values.
• Remains: All acceptance criteria. No backup has been taken and nothing has been changed in this repo or in the cluster.
• Decisions made: Logs are not a database — Loki or Longhorn, not local-path. Longhorn is correct only for genuinely app-owned non-database volumes.
• Open: Whether /app/data is a SQLite DB or key material. That classification decides the remedy and is yours to make.
• Filed by: platform-governance (FuzeInfra), as cross-repo @claude delegation. Nothing in this repo was edited.

[claude]

Claude finished @izzywdev's task in 0s —— View job

---

I'll analyze this and get back to you.

[izzywdev]

@claude

One change to the acceptance criteria, and it changes what you do first.

Add a new criterion 1: a verified backup, before anything else

As written, the closest this issue gets to protecting the data is criterion 5 — "Data is verified present in its new home before the old PVC is deleted." That is not sufficient. It only guards the deletion step; it leaves the window between now and cutover completely uncovered, and it presumes the migration reaches a "new home" at all. If the node is reaped mid-migration, criterion 5 was never reached and there is no copy anywhere.

Please insert as criterion 1, ahead of the classification work:

1. A verified, restorable backup of fuzekeys-data (and fuzekeys-logs if it holds anything of value) exists off that volume, on durable storage. Read it back and confirm it is complete — a dump nobody has restored is not a backup. Nothing else in this issue starts until this is done.

Renumber the existing criteria after it. The established off-cluster target is the FuzeInfra Contabo Object Storage backup sink — the same sink the nightly DB backups use (helm/fuzeinfra/templates/backup-cronjobs.yaml, bucket/keys via terraform/contabo/object-storage.tf). Note the two standing gotchas: Contabo S3 credentials come from the Contabo API, not the panel, and every API call needs an x-request-id header that is a valid UUID.

Why this is the first action and not a formality

The issue body already establishes the two facts that make it urgent, and they point the same way:

• The pin is fuzeinfra-prod-elastic-v2-9b31f718, which is alive and currently running your backend. So the data is most likely still there. This is not a post-mortem — it is a window.
• That is an elastic node under a billing-aware autoscaler that releases idle elastic nodes ahead of renewal, combined with node-pinned local-path and reclaimPolicy: Delete. When it goes, the volume goes with it and the PV is not retained. There is no scheduled event forcing this, and no warning when it happens.

This is the same shape as the two losses already on record in this sweep: mendys-wp's 40Gi MariaDB volume (directory verified empty after its node was reinstalled) and Neo4j in the 2026-07-24 incident. In both cases the migration plan was correct and simply arrived after the node did. FuzeKeys is the one in this set that still has a live node and therefore still has a choice.

Take the backup even if you intend to migrate immediately, and take it before you classify /app/data — the classification question (SQLite datastore vs. key material) does not need to be settled first, and settling it is not free.

Unchanged

• The classification of /app/data is still yours to make and still decides the remedy: SQLite datastore → FuzeInfra-provisioned Postgres via serviceDatabases; key material → Longhorn plus a sealed-secret/Vault review, not silently poured into Postgres. Note that docs/consuming-repos/POSTGRES_PROVISIONING.md has since landed on FuzeInfra main (PR #827) — read it rather than hand-rolling the handshake.
• fuzekeys-logs is not a database. Logs belong in the shared Loki stack. If some log data must persist as files, it goes to object storage or Longhorn — never local-path.
• No FuzeKeys PVC remains on local-path.
• Prod is GitOps. Every change lands as a merged PR that Argo syncs. Never kubectl patch or kubectl edit a live prod resource.

STATE:

• Why: The criteria as written protect only the deletion step, leaving live data on a reapable volume unprotected for the whole duration of the migration.
• Done: Re-read the criteria and confirmed no backup-first requirement existed. Confirmed from the FuzeInfra side that the Contabo Object Storage sink is the established off-cluster target, and that docs/consuming-repos/POSTGRES_PROVISIONING.md has landed on main.
• Remains: New criterion 1 (verified backup) plus all previously listed criteria. No backup has been taken. Nothing in this repo or the cluster has been changed.
• Decisions made: Backup precedes classification, not the other way round.
• Open: Whether /app/data is a SQLite datastore or key material.
• Blocked on: The prod k3s API was refusing connections on 6443 as of 2026-09-02, so the backup cannot be taken until the cluster returns. This does not reduce the priority — it is the first action the moment access is restored. Source-level preparation is not blocked.
• Comment by: platform-governance (FuzeInfra). Nothing in this repo was edited.

[claude]

Claude finished @izzywdev's task in 0s —— View job

---

I'll analyze this and get back to you.