Recovery: drain handling for GPU reset

[tkatila]

No description provided.

[Copilot]

🟡 Changes recommended

A drain timeout can be enforced using the default timeout while incorrectly reporting “after 0s” in the failure message when spec.drain.timeoutSeconds is non-positive (defensive path inconsistency).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds a pre-reset node drain phase to the GPURecoveryPlan controller so PCIe reset recoveries (slot/SBR/AMC) don’t run under active workloads and while DRA ResourceClaims still reserve the device, while keeping reflashes out of this drain path.

Changes:

• Introduces a draining state machine phase before reset Jobs, including node tainting, pod eviction, and ResourceClaim-based “GPU still reserved” gating.
• Adds taint cleanup logic (including on finalizer delete) so nodes are not left unschedulable.
• Expands/updates envtest coverage for drain behavior and adds operator tolerations for the recovery taint (kustomize + Helm).

File summaries

File	Description
internal/controller/gpurecoveryplan_helpers.go	Adds drain helpers (capping lists, drain deadline/failure helpers, taint builder) and de-duplicates repeated status messages.
internal/controller/gpurecoveryplan_controller.go	Adds drain phase to reconciliation, node taint reconciliation/cleanup, ResourceClaim reservation checks, and RBAC updates.
internal/controller/gpurecoveryplan_controller_test.go	Adds extensive envtest coverage for the draining state machine, taint cleanup, PDB behavior, and claim-reservation gating.
internal/controller/gpurecoveryplan_const.go	Adds drain-related constants (taint key, list cap, default timeout).
config/manager/manager.yaml	Ensures the operator Deployment tolerates the recovery taint it applies.
charts/gpu-base-operator/values.yaml	Adds configurable extra tolerations for the operator pod (Helm values).
charts/gpu-base-operator/templates/manager.yaml	Adds built-in recovery taint toleration plus optional extra tolerations via Helm values.
charts/gpu-base-operator/README.md	Documents the new operator.tolerations chart value.

Review details

• Files reviewed: 8/8 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[pfl]

Maybe just an 'if' here if no other cases will be added later?