Skip to content

fix: reconcile starvation around configcheck pods and pipeline event debouncing#261

Merged
aa1ex merged 4 commits into
kaasops:mainfrom
aa1ex:fix/configcheck-skip-terminating-namespace
Jul 16, 2026
Merged

fix: reconcile starvation around configcheck pods and pipeline event debouncing#261
aa1ex merged 4 commits into
kaasops:mainfrom
aa1ex:fix/configcheck-skip-terminating-namespace

Conversation

@aa1ex

@aa1ex aa1ex commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Problem

The e2e suite flakes with Timed out after 120s waiting for an agent config secret (seen on reruns of #253/#257/#258 and reproducible on a v1.32.11 kind cluster). Tracing it surfaced four independent defects, all starving the serial (MaxConcurrentReconciles=1) reconcile workers:

  1. A configcheck launched into a terminating namespace can never create its pod/secret (unable to create new content in namespace ... because it is being terminated), so getCheckResult holds the worker for the full ConfigCheckTimeout (300s); one CI run showed 30 failed reconciles plus two back-to-back 300s stalls.
  2. Reconciling a CR whose target namespace is gone keeps erroring and requeueing forever — leaked cluster-scoped ClusterVectorAggregators from e2e kept pointing resourceNamespace at deleted namespaces, spamming failed to create or update Secret: namespaces ... not found.
  3. The pipeline→agent debounce (reconcileWithDelay) resets its ticker on every incoming event, so events arriving faster than the 10s delay postpone the flush indefinitely and agents stop being notified at all.
  4. A configcheck pod that can never start (e.g. CR env referencing a missing secret → CreateContainerConfigError; the configcheck pod inherits the CR's env) stays Pending forever, and getCheckResult only reacted to Modified events with Succeeded/Failed phases — every reconcile burned the full 300s. The same watch loop also ignored the initial synthetic Added events and Deleted events, and treated a closed watch channel as success, letting an unvalidated config through.

Fixes (one commit per defect)

  • configcheck.Run returns a new ErrConfigcheckSkipped sentinel when the target namespace is terminating or gone; all call sites treat it as a no-op skip.
  • Vector / VectorAggregator / ClusterVectorAggregator reconcilers skip early when the target namespace (for CVA — spec.resourceNamespace) is terminating or gone, via a shared k8s.NamespaceIsTerminating helper; the podmonitor e2e now deletes its cluster-scoped aggregators in AfterAll.
  • reconcileWithDelay coalesces events and flushes every delay instead of resetting the ticker per event, so notifications survive continuous load.
  • getCheckResult fails fast with the container's waiting reason when the pod is unstartable (CreateContainerConfigError, InvalidImageName) — the CR gets a Failed status carrying the actual cause within seconds; it also handles Added, errors on Deleted, and errors on a closed watch channel instead of reporting success.

Verification

  • TDD unit tests per fix (configcheck/namespace_test.go, configcheck/watch_test.go, utils/k8s/namespace_test.go, cmd/manager/main_test.go); make test green.
  • kind v1.32.11 (CI matrix pin): before the fixes the full suite failed on the first run with this flake; with all four fixes it passed 4/4 consecutive full-suite runs with zero configcheck timeouts, and full-suite time dropped from ~23 to ~11.5 minutes (two 300s configcheck stalls per run gone).
  • Live check: a VectorAggregator whose env references a missing secret reaches configCheckResult: false with reason configcheck pod cannot start, CreateContainerConfigError: secret "no-such-secret" not found in ~25s; before the fix its configcheck pod sat in CreateContainerConfigError until the 300s timeout on every reconcile.

Behavior note: an unstartable configcheck pod is now reported as a validation failure (Failed status with reason) instead of a silent timeout-and-requeue loop; fixing the CR or creating the missing secret re-triggers reconciliation as usual.

Follow-up intentionally left out: raising MaxConcurrentReconciles for the Vector controllers — serialization amplifies any single stall and deserves its own discussion.

@aa1ex
aa1ex merged commit 12f09c1 into kaasops:main Jul 16, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant