Stale VolumeAttachments after StatefulSet scale-down on ontap-san

[stratogit]

Describe the bug

When scaling an iSCSI (ontap-san) StatefulSet down and up with time in between, some VolumeAttachments remain in attached: true state on the original node. When pods are rescheduled to different nodes on the scale-up, they fail with Multi-Attach error because the RWO volumes are still exclusively attached to the previous node.

This was not observed with Trident v26.02.x. nor any older trident version since 25. The issue is new in v26.06.0.

The suspected root cause is commit 42d54ab3 ("Remove ghost multipath devices"). In PrepareDeviceForRemoval, when VerifyMultipathDevice returns isGhost == true, the function returns early ("", nil) — skipping removeSCSIDevice. During a rapid scale-down, GetMultipathDeviceDisks reads /sys/block/<dm>/slaves/ which can be transiently empty as paths are being removed during the iSCSI logout sequence itself, causing a false-positive ghost detection.

In v26.02.1, the ghost detection result was discarded (_, err := client.devices.VerifyMultipathDevice(...)) and removeSCSIDevice was always called regardless.

Code path

1. nodeUnstageISCSIVolume → PrepareDeviceForRemoval
2. PrepareDeviceForRemoval → VerifyMultipathDevice → compareWithPublishedDevicePath
3. compareWithPublishedDevicePath → GetMultipathDeviceDisks reads /sys/block/<dm>/slaves/
4. During iSCSI logout/cleanup, slaves can transiently disappear → isProbablyGhostDevice = true
5. PrepareDeviceForRemoval returns ("", nil) early — skips removeSCSIDevice
6. Unstage appears to succeed but volume is never fully detached from the node
7. VolumeAttachment persists → Multi-Attach error on reschedule

Relevant code (utils/iscsi/iscsi.go in v26.06.0)

// In PrepareDeviceForRemoval:
if publishInfo.IscsiTargetPortal != "" && deviceInfo.MultipathDevice != "" {
    isGhost, err := client.devices.VerifyMultipathDevice(ctx, publishInfo, allPublishInfos, deviceInfo)
    if err != nil {
        return "", err
    }

    if isGhost {
        if err := client.devices.RemoveGhostMultipathDevice(ctx,
            deviceInfo.MultipathDevice, publishInfo.IscsiLunSerial); err != nil {
            // ... just logs a warning
        }
        return "", nil  // <-- EARLY RETURN, skips removeSCSIDevice
    }
}

Compared with v26.02.1

// In PrepareDeviceForRemoval (v26.02.1):
if publishInfo.IscsiTargetPortal != "" && deviceInfo.MultipathDevice != "" {
    _, err := client.devices.VerifyMultipathDevice(ctx, publishInfo, allPublishInfos, deviceInfo)
    if err != nil {
        return "", err
    }
    // No early return — always proceeds to removeSCSIDevice
}

Ghost detection logic (utils/devices/devices.go)

func (c *Client) compareWithPublishedDevicePath(...) (bool, error) {
    // ...
    devices, err := c.GetMultipathDeviceDisks(ctx, deviceInfo.MultipathDevice)
    // ...
    isProbablyGhostDevice = len(devices) == 0  // <-- false positive during active teardown
    // ...
}

GetMultipathDeviceDisks reads /sys/block/<dm>/slaves/ and only counts sd* entries. During the unstage sequence when SCSI paths are being removed by concurrent operations, this directory can briefly appear empty, triggering false ghost detection.

Environment

• Trident version: v26.06.0 (v26.06.0)
• Container runtime: containerd 2.3.1
• Kubernetes version: v1.36.1
• OS: SUSE Linux Enterprise Server 15 SP7
• Kernel: 6.4.0
• NetApp backend types: ONTAP SAN (iSCSI), ontap-san StorageClass
• Multipath: active (dm-multipath)
• Fresh cluster, no prior restarts or disruptions

To Reproduce

1. Deploy a StatefulSet with 30 replicas using ontap-san (RWO iSCSI block) StorageClass
2. Wait for all pods to be running and volumes attached (all healthy)
3. Scale the StatefulSet to 0 replicas
4. Wait 10 seconds
5. Scale the StatefulSet back to 30 replicas
6. Observe that 2 or more out of 30 pods are stuck in ContainerCreating with Multi-Attach error

The issue reproduces when:

• Multiple volumes are being unstaged concurrently (30 or more pods scaling down at once)
• The volumes are RWO iSCSI block (ontap-san) with multipath
• The scale-up happens before all VolumeAttachments are fully cleaned up
• NAS/NFS volumes are NOT affected (only block/iSCSI)

Expected behavior

NodeUnstageVolume should fully clean up iSCSI devices, sessions, and multipath mappings regardless of ghost detection status. VolumeAttachments should be deleted within a reasonable time after unstage, allowing volumes to be re-attached to different nodes without Multi-Attach errors.

Additional context

Pod events showing the failure

Warning  FailedAttachVolume  attachdetach-controller  Multi-Attach error for volume
         "pvc-47658c7a-b534-46f0-82f2-2266698f81b8" Volume is already exclusively
         attached to one node and can't be attached to another

Stale VolumeAttachments

Both volumes show attached: true to the original node even though no pods on that node reference them:

NAME                   ATTACHER                PV                                      NODE           ATTACHED
csi-ef268c38...        csi.trident.netapp.io   pvc-47658c7a-b534-46f0-82f2-226669...   pool3-node5    true
csi-26079276...        csi.trident.netapp.io   pvc-9cf7f841-a22d-46e4-b77f-fc4913...   pool3-node5    true

CSI attacher GRPC response confirms stale publish

Trident's ListVolumes response still reports the volumes as published to the original node:

{"published_node_ids":["pool3-node5"],"volume":{"volume_id":"pvc-47658c7a-...","protocol":"block"}}
{"published_node_ids":["pool3-node5"],"volume":{"volume_id":"pvc-9cf7f841-...","protocol":"block"}}

Impact

• 2 out of 30 pods permanently stuck in ContainerCreating
• StatefulSet never reaches full replica count
• Manual intervention required to delete stale VolumeAttachments