Description
When controller concurrency is enabled, periodic backend reconciliation
replaces existing backend objects with newly initialized backend objects.
The replacement backend has an empty backend-local volume map. As a result,
tridentctl get backend reports VOLUMES=0, while tridentctl get volume
continues to return all managed volumes with valid backend UUIDs.
Environment
- Trident: 26.06.0
- Driver: ontap-san
- Controller concurrency: enabled
- Persistence: Kubernetes CRDs
The issue was also observed with Trident 26.02.1.
Observed behavior
All backends remain online and normal:
netapp-qos-01-02 online 0
netapp-qos-04-02 online 0
netapp-qos-03-01 online 0
netapp-qos-03-02 online 0
netapp-qos-04-01 online 0
netapp-qos-01-01 online 0
At the same time, the global volume store remains populated and all volumes
have valid backend UUIDs.
Disabling controller concurrency and restarting the controller restores the
backend volume counts.
Suspected cause
ConcurrentTridentOrchestrator.reconcileBackendState replaces the backend
through upsertBackend, but the replacement backend does not retain the
original backend-local volume associations.
Proposed fix
Copy the existing backend-local volume associations from the original backend
to the replacement backend before node access reconciliation and cache
replacement.
I have prepared a focused fix and regression tests and plan to submit a PR.
Description
When controller concurrency is enabled, periodic backend reconciliation
replaces existing backend objects with newly initialized backend objects.
The replacement backend has an empty backend-local volume map. As a result,
tridentctl get backendreportsVOLUMES=0, whiletridentctl get volumecontinues to return all managed volumes with valid backend UUIDs.
Environment
The issue was also observed with Trident 26.02.1.
Observed behavior
All backends remain online and normal:
At the same time, the global volume store remains populated and all volumes
have valid backend UUIDs.
Disabling controller concurrency and restarting the controller restores the
backend volume counts.
Suspected cause
ConcurrentTridentOrchestrator.reconcileBackendStatereplaces the backendthrough
upsertBackend, but the replacement backend does not retain theoriginal backend-local volume associations.
Proposed fix
Copy the existing backend-local volume associations from the original backend
to the replacement backend before node access reconciliation and cache
replacement.
I have prepared a focused fix and regression tests and plan to submit a PR.