[Feature] Pin a constant Kafka cluster ID to survive re-installs and upgrades

[GiladShapira94]

📝 Description

Adds an optional mechanism to pin a constant Kafka cluster ID so that re-installs and upgrades reuse the data on the existing PersistentVolumeClaims instead of failing with InconsistentClusterIdException.

In KRaft mode the Strimzi operator otherwise generates a fresh random cluster ID on every (re)install, which no longer matches the ID persisted in the retained PVC (meta.properties).

Because clusterId is a status-only field in the Strimzi CRDs, it cannot be set through a normal Helm/kubectl apply, so the Kafka and KafkaNodePool CRs are now created with reconciliation paused and a post-install hook Job patches status.clusterId on both resources before resuming reconciliation.

This affects only the kafka component and is gated on the new kafka.clusterId value.

---

🛠️ Changes Made

• Added kafka.clusterId to charts/mlrun-ce/values.yaml, defaulting to the fixed value "ByHirbmSVDCwP7YDBt3V2A" (a 22-char base64url-encoded 16-byte UUID); set to "" to keep the previous random-ID-per-install behaviour.

• Added kafka.clusterIdInit block to values.yaml exposing the hook Job's image (alpine/kubectl:1.33.4) and CPU/memory requests+limits.

• New template charts/mlrun-ce/templates/kafka/kafka-clusterid-init.yaml — creates a ServiceAccount, Role, RoleBinding, and post-install/post-upgrade Job ({{ .Values.kafka.name }}-clusterid-init) that waits for the CRs, patches status.clusterId on kafka/<name> and kafkanodepool/<name>-pool via the /status subresource (idempotent), then removes the pause annotation to resume reconciliation. Guarded by {{- if and .Values.kafka.enabled .Values.kafka.clusterId }}.

• templates/kafka/kafka-cluster.yaml and templates/kafka/kafka-nodepool.yaml — conditionally add strimzi.io/pause-reconciliation: "true" (only when kafka.clusterId is set) so the operator does not format storage with a random ID before the init Job pins the constant ID.

• The RBAC grants get/list/watch/patch/update on kafkas/kafkanodepools and get/patch/update on their /status subresources, scoped to .Release.Namespace.

• Hook ordering: CRs at hook-weight: "5", init Job at hook-weight: "10" with hook-delete-policy: before-hook-creation,hook-succeeded and a 600s timeout.

---

✅ Checklist

• I have tested the changes in this PR
• I confirmed whether my changes require a change in documentation and if so, I created another PR in MLRun for the relevant documentation.
• I confirmed whether my changes require changes in QA tests, for example: credentials changes, resources naming change and if so, I updated the relevant Jira ticket for QA.
• I increased the Chart version in charts/mlrun-ce/Chart.yaml.
• I confirmed that the installation works both on a local Docker Desktop environment and on a real cluster when using the required prerequisites.
  • If installation issues were found, I updated the relevant Jira ticket with the issue and steps to reproduce, or updated the prerequisites documentation if the issue is related to missing or outdated prerequisites.
• If needed, update https://github.com/mlrun/ce/blob/development/charts/mlrun-ce/README.md with the relevant installation instructions and version Matrix.
• If needed, update the following values files for multi namespace support:
  • Admin values
  • User values Node Port
  • User values ClusterIP

---

🧪 Testing

Suggested (not yet confirmed from the diff): make helm-lint; helm template with kafka.enabled=true to confirm the init Job, RBAC, and pause annotations render only when kafka.clusterId is non-empty; and helm template ... | kubectl apply --dry-run=client -f - for schema validation.

A real-cluster run is required to verify the full flow: fresh install pins the ID, a re-install reuses the retained PVC without InconsistentClusterIdException, and the init Job resumes reconciliation successfully.

---

🔗 References

• Ticket link: CEML-724
• External links: https://strimzi.io/docs/operators/latest/deploying.html#assembly-pausing-reconciliation-str

---

🚨 Breaking Changes?

• Yes (explain below)

• No

• No new or renamed value keys are removed, and no Secret/ConfigMap names or NodePorts change — but the default behaviour changes: kafka.clusterId now ships with a fixed non-empty default instead of the operator generating a random ID per install.

• For an existing Kafka deployment whose PVCs already hold data under a previously-generated random cluster ID, upgrading to this chart will cause the init Job to pin status.clusterId to the new fixed value and resume reconciliation, which can trigger InconsistentClusterIdException against the retained meta.properties.

• Mitigation for such installs: set kafka.clusterId to the cluster's existing ID, or set kafka.clusterId: "" to preserve the prior random-ID behaviour.

---

🔍️ Additional Notes

The clusterId default is hardcoded to a single shared value across all installs; consider whether each deployment should generate/pin its own unique ID to avoid multiple clusters sharing the same identifier.

The init Job overwrites any pre-existing status.clusterId (it only logs a WARNING), so upgrades on clusters with existing data need care.

[royischoss]

LGTM 3 small comments

[royischoss]

maybe worthy to provide a Notes.txt msg if this job failed or succeed

[GiladShapira94]

As we spoke, the note just prints a message based on the user value. I can add a message that the installation uses a constant cluster ID, but I think it will confuse users more than help them. wdyt?
also if the job failed the entire installation shold fail

[royischoss]

RBAC wiring is correct. Please scope the Role with resourceNames to {{ .Values.kafka.name }} and {{ .Values.kafka.name }}-pool, drop unused update/list/watch verbs, and note in the PR that the SA retains patch permissions for the life of the release.

means: any pod using this SA can patch any Kafka / KafkaNodePool CR in the namespace — not just the one this chart creates.

The Job script only touches:

kafka/{{ .Values.kafka.name }} → e.g. kafka-stream
kafkanodepool/{{ .Values.kafka.name }}-pool → e.g. kafka-stream-pool
Adding resourceNames would narrow that to those two objects only:

• apiGroups: [kafka.strimzi.io]
  resources: [kafkas]
  resourceNames: [kafka-stream]
  verbs: [get, patch]
• apiGroups: [kafka.strimzi.io]
  resources: [kafkanodepools]
  resourceNames: [kafka-stream-pool]
  verbs: [get, patch]

Not a must but hardening the Role

[GiladShapira94]

Fixed

[royischoss]

LGTM