fix(operator): reconcile PVC storage growth from Cluster persistence sizes - #1701
fix(operator): reconcile PVC storage growth from Cluster persistence sizes#1701Dav-14 wants to merge 2 commits into
Conversation
…sizes Growing spec.persistence.<volume>.size on an existing Cluster was silently ignored: volumeClaimTemplatesChanged only compares template names (the PVC<->hostPath switch case), templates are immutable on a live StatefulSet, and nothing expanded the existing PVCs. The reconciler now detects per-template size drift, patches the backing PVCs (grow-only, requires an expandable storage class), and recreates the StatefulSet with orphan propagation so future replicas are provisioned at the new size. Shrink requests are ignored and surfaced with a VolumeShrinkIgnored warning event since PVC storage cannot shrink.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## release/v3.0 #1701 +/- ##
================================================
+ Coverage 75.71% 75.98% +0.26%
================================================
Files 453 453
Lines 48366 48378 +12
================================================
+ Hits 36622 36761 +139
+ Misses 8466 8347 -119
+ Partials 3278 3270 -8
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
Growing
spec.persistence.<volume>.sizeon an existingClusterwas silently ignored:volumeClaimTemplatesChangedonly compares template names — it was written for the PVC↔hostPath switch, so a pure size change never triggers the orphan-recreate path.The spec would say
250Gi, reality stayed at10Gi, with no event, condition, or log surfacing the drift (observed on a dev env: Cluster CR updated to 250Gi by the Formance operator, StatefulSet and PVCs untouched).Fix
On each StatefulSet reconcile pass, after the existing name-change check:
volumeClaimTemplateSizeChanges), the backing PVCs are patched to the new size ordinal-by-ordinal (expandClusterPVCs, grow-only,VolumeExpanded/VolumeExpansionFailedevents), then the StatefulSet is recreated with orphan propagation — same mechanism as the hostPath switch — so future replicas are provisioned at the new size.VolumeShrinkIgnoredwarning event on the Cluster.Notes:
allowVolumeExpansion: true; failures surface as warning events and requeue with backoff.patchonpersistentvolumeclaims; no RBAC change.Tests
TestVolumeClaimTemplateSizeChanges(grow / shrink / mixed / one-sided names).TestReconcile_DataVolumeSizeGrow— bound PVCs at 1Gi + expandable StorageClass; growing the Cluster to 2Gi patches all 3 PVCs and recreates the StatefulSet with the 2Gi template (the test plays the garbage collector for the orphan finalizer, which envtest doesn't run).TestReconcile_DataVolumeSizeShrinkIgnored— shrink emitsVolumeShrinkIgnored, StatefulSet untouched (same UID, template keeps its size).release/v3.0(TestReconcile_EvenReplicas,TestReconcile_IngressEnabledNoHosts,TestReconcile_IngressGrpcEnabledNoHosts) — identical on a clean checkout, unrelated to this change.just pre-commit(generate, tidy, build) clean — no generated diff.