Skip to content

fix(operator): reconcile PVC storage growth from Cluster persistence sizes - #1701

Draft
Dav-14 wants to merge 2 commits into
release/v3.0from
fix/operator-pvc-resize
Draft

fix(operator): reconcile PVC storage growth from Cluster persistence sizes#1701
Dav-14 wants to merge 2 commits into
release/v3.0from
fix/operator-pvc-resize

Conversation

@Dav-14

@Dav-14 Dav-14 commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Growing spec.persistence.<volume>.size on an existing Cluster was silently ignored:

  • volumeClaimTemplatesChanged only compares template names — it was written for the PVC↔hostPath switch, so a pure size change never triggers the orphan-recreate path.
  • VolumeClaimTemplates are immutable on a live StatefulSet, and no code path expanded the existing PVCs.

The spec would say 250Gi, reality stayed at 10Gi, with no event, condition, or log surfacing the drift (observed on a dev env: Cluster CR updated to 250Gi by the Formance operator, StatefulSet and PVCs untouched).

Fix

On each StatefulSet reconcile pass, after the existing name-change check:

  • Grow: per-template size drift is detected (volumeClaimTemplateSizeChanges), the backing PVCs are patched to the new size ordinal-by-ordinal (expandClusterPVCs, grow-only, VolumeExpanded/VolumeExpansionFailed events), then the StatefulSet is recreated with orphan propagation — same mechanism as the hostPath switch — so future replicas are provisioned at the new size.
  • Shrink: ignored (PVC storage is grow-only in Kubernetes) and surfaced with a VolumeShrinkIgnored warning event on the Cluster.

Notes:

  • PVC expansion requires the storage class to have allowVolumeExpansion: true; failures surface as warning events and requeue with backoff.
  • PVCs not yet provisioned are skipped — the recreated template sizes them at creation.
  • RBAC already grants patch on persistentvolumeclaims; no RBAC change.
  • Scope: PVCs up to the current replica count. Retained PVCs from a previous scale-down beyond that are not resized.

Tests

  • Unit: TestVolumeClaimTemplateSizeChanges (grow / shrink / mixed / one-sided names).
  • envtest integration:
    • TestReconcile_DataVolumeSizeGrow — bound PVCs at 1Gi + expandable StorageClass; growing the Cluster to 2Gi patches all 3 PVCs and recreates the StatefulSet with the 2Gi template (the test plays the garbage collector for the orphan finalizer, which envtest doesn't run).
    • TestReconcile_DataVolumeSizeShrinkIgnored — shrink emits VolumeShrinkIgnored, StatefulSet untouched (same UID, template keeps its size).
  • Full suite: only pre-existing failures on release/v3.0 (TestReconcile_EvenReplicas, TestReconcile_IngressEnabledNoHosts, TestReconcile_IngressGrpcEnabledNoHosts) — identical on a clean checkout, unrelated to this change.
  • just pre-commit (generate, tidy, build) clean — no generated diff.

…sizes

Growing spec.persistence.<volume>.size on an existing Cluster was silently
ignored: volumeClaimTemplatesChanged only compares template names (the
PVC<->hostPath switch case), templates are immutable on a live StatefulSet,
and nothing expanded the existing PVCs.

The reconciler now detects per-template size drift, patches the backing
PVCs (grow-only, requires an expandable storage class), and recreates the
StatefulSet with orphan propagation so future replicas are provisioned at
the new size. Shrink requests are ignored and surfaced with a
VolumeShrinkIgnored warning event since PVC storage cannot shrink.
@coderabbitai

coderabbitai Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c87d1d74-6b03-4c9d-aae9-109cd9ed6102

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codecov

codecov Bot commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.98%. Comparing base (bb0297c) to head (e279731).
⚠️ Report is 3 commits behind head on release/v3.0.

Additional details and impacted files
@@               Coverage Diff                @@
##           release/v3.0    #1701      +/-   ##
================================================
+ Coverage         75.71%   75.98%   +0.26%     
================================================
  Files               453      453              
  Lines             48366    48378      +12     
================================================
+ Hits              36622    36761     +139     
+ Misses             8466     8347     -119     
+ Partials           3278     3270       -8     
Flag Coverage Δ
e2e 75.98% <ø> (+0.26%) ⬆️
scenario 75.98% <ø> (+0.26%) ⬆️
unit 75.98% <ø> (+0.26%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

1 participant