storage: PVCs on local-path are on disposable disk — migrate to longhorn

[izzywdev]

@fuze this repo has PersistentVolumeClaims on the local-path StorageClass, which is unsafe on this cluster.

Affected PVCs: fuzekeys-data, fuzekeys-logs (namespace fuzekeys)

fuzekeys-data is already bound to a node that no longer exists (fuzeinfra-elastic-918c8556). That data is gone, and it is why fuzekeys-backend has been unschedulable — its PV points at a deleted machine. Separately, fuzekeys-backend declares replicas: 2 with strategy: RollingUpdate against two ReadWriteOnce claims, which can never reach ready state even with a healthy volume: only one pod can mount an RWO PVC, and RollingUpdate deadlocks against it. Both need fixing.

Why this is urgent

local-path provisions node-local disk and pins the PV to that node by nodeAffinity. This cluster's elastic nodes are created and reaped by an autoscaler, so a volume on local-path dies with its node. Its reclaimPolicy is also Delete, not Retain.

This is not hypothetical. Measured on 2026-09-01, two PVs are already bound to nodes that no longer exist:

fuzekeys/fuzekeys-data               ->  fuzeinfra-elastic-918c8556   (node gone)
fuzeinfra/data-fuzeinfra-postgres-0  ->  fuzeinfra-elastic-57543f68   (node gone)

It is the same failure as the 2026-07-24 incident, in which local-path volumes were destroyed by node reinstalls and Neo4j was lost.

Why it happened — it was not your mistake

local-path was the cluster default StorageClass. Any chart that did not explicitly set storageClassName got ephemeral disk silently. global.storageClass: longhorn only governs FuzeInfra's own umbrella chart; consuming repos deploy their own Helm releases and never inherited it.

FuzeInfra has now made longhorn the default (FuzeInfra#814), verified by probe: a PVC with no storageClassName binds to longhorn, replicated across 3 durable nodes with reclaimPolicy: Retain. So newly created PVCs are safe from here.

What is still needed from this repo

Existing PVCs do not migrate automatically — storageClassName is immutable on a bound claim. Please:

1. Set storageClassName: longhorn explicitly in this repo's Helm chart for the PVCs listed above, so it is declared rather than inherited.
2. Plan the migration of the existing volume(s): back up, delete the PVC, let it be recreated on longhorn, restore. Sequence it per-service; do not do it blind.
3. If any of these hold data you cannot lose, treat it as at risk right now until migrated — it is on a disk that disappears if that node is replaced.

Not urgent for stateless/scratch volumes; state clearly which of yours are which.

[izzywdev]

Recovery done on the live cluster — your volumes were the true dead-node case

Ran the dead-volume recovery today. Findings here rather than a duplicate issue.

Evidence

FuzeKeys is the one repo in this class where the original diagnosis was exactly right — the PVs were pinned to a node that no longer exists:

pvc-036a8dd5...  fuzekeys/fuzekeys-data  -> nodeAffinity fuzeinfra-elastic-918c8556
pvc-c3fa39cc...  fuzekeys/fuzekeys-logs  -> nodeAffinity fuzeinfra-elastic-918c8556

fuzeinfra-elastic-918c8556 is absent from kubectl get nodes (the live elastic nodes are all named fuzeinfra-prod-elastic-v2-* — different generation). The scheduler said so plainly:

0/10 nodes are available: ... 4 node(s) didn't match PersistentVolume's node affinity

That is unrecoverable by definition: the disk was destroyed with the node. fuzekeys-backend had been Pending for 10 days.

Action taken and result

Deleted both PVCs and the pending pods. Argo recreated them and the backend is up:

fuzekeys-backend-78d7757f57-l6dvx   1/1   Running
fuzekeys-backend-78d7757f57-nlz8v   1/1   Running

The part that still needs a change in THIS repo

Unlike FuzeAgent, your chart explicitly sets the storage class — argocd-controller owns .spec.storageClassName on both claims, so Argo re-applied local-path and the recreated volumes went straight back onto disposable node-local disk:

fuzekeys-data   Bound   pvc-b6683cd5-...   5Gi   local-path
fuzekeys-logs   Bound   pvc-4d083e76-...   2Gi   local-path

The workload is recovered but the underlying fault is not fixed — these will die again the next time their node is reaped or reinstalled. This issue should stay open.

Two changes needed:

1. Drop the explicit storageClassName: local-path (or set it to longhorn). Longhorn is the cluster default since FuzeInfra #814, so simply omitting the field is the cleanest fix and keeps the chart portable to kind/AWS.
2. Add a durable-node pin to pods mounting a PVC. longhorn-manager is CrashLoopBackOff on all 5 elastic nodes, so Longhorn registers node objects only for the 3 durable nodes; a pod scheduled onto an elastic node can never attach a Longhorn volume. All 3 durable nodes carry PreferNoSchedule, so the scheduler actively prefers the elastic ones:

nodeSelector:
  node.longhorn.io/create-default-disk: "true"

Note fuzekeys-logs is a logs volume — worth asking whether it needs to be a PVC at all rather than shipping to Loki.

FuzeInfra is making the equivalent fix for its own charts in izzywdev/FuzeInfra#817. I did not edit your chart.

Posted by the FuzeInfra devops recovery pass. The data on both volumes was already gone with the node.