Skip to content

Concurrent backend reconciliation clears backend volume associations #1166

Description

@s-starostin

Description

When controller concurrency is enabled, periodic backend reconciliation
replaces existing backend objects with newly initialized backend objects.

The replacement backend has an empty backend-local volume map. As a result,
tridentctl get backend reports VOLUMES=0, while tridentctl get volume
continues to return all managed volumes with valid backend UUIDs.

Environment

  • Trident: 26.06.0
  • Driver: ontap-san
  • Controller concurrency: enabled
  • Persistence: Kubernetes CRDs

The issue was also observed with Trident 26.02.1.

Observed behavior

All backends remain online and normal:

netapp-qos-01-02  online  0
netapp-qos-04-02  online  0
netapp-qos-03-01  online  0
netapp-qos-03-02  online  0
netapp-qos-04-01  online  0
netapp-qos-01-01  online  0

At the same time, the global volume store remains populated and all volumes
have valid backend UUIDs.

Disabling controller concurrency and restarting the controller restores the
backend volume counts.

Suspected cause

ConcurrentTridentOrchestrator.reconcileBackendState replaces the backend
through upsertBackend, but the replacement backend does not retain the
original backend-local volume associations.

Proposed fix

Copy the existing backend-local volume associations from the original backend
to the replacement backend before node access reconciliation and cache
replacement.

I have prepared a focused fix and regression tests and plan to submit a PR.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions