CAC-260: Report SGLang HiCache readiness from engine Pods

api/v1alpha1/cachebackend_types.go

@@ -643,9 +643,10 @@ type CacheBackendIntegrationSpec struct {
 	// default — the field carries no meaningful "wait forever" or "fail
 	// immediately" semantics.
 	//
-	// The first SGLangHiCache implementation publishes no Ready condition, so
-	// this field is inert for that engine-local backend until its separate
-	// readiness contract is implemented.
+	// SGLangHiCache readiness is derived from engine Pod injection receipts,
+	// convergence with the current adapter configuration, and Kubernetes Pod
+	// readiness rather than KV events, so this field is inert for that
+	// engine-local backend.
 	//
 	// The value is a Go duration string (e.g. "90s", "5m", "1h"). The CRD
 	// schema types it as a string; a malformed value is rejected when

config/crd/bases/inferencecache.io_cachebackends.yaml

@@ -574,9 +574,10 @@ spec:
                       default — the field carries no meaningful "wait forever" or "fail
                       immediately" semantics.
 
-                      The first SGLangHiCache implementation publishes no Ready condition, so
-                      this field is inert for that engine-local backend until its separate
-                      readiness contract is implemented.
+                      SGLangHiCache readiness is derived from engine Pod injection receipts,
+                      convergence with the current adapter configuration, and Kubernetes Pod
+                      readiness rather than KV events, so this field is inert for that
+                      engine-local backend.
 
                       The value is a Go duration string (e.g. "90s", "5m", "1h"). The CRD
                       schema types it as a string; a malformed value is rejected when

config/samples/README.md

@@ -54,9 +54,12 @@ they go `Ready` as soon as admission accepts the endpoint. See the
 [quickstart](../../docs/quickstart.md).
 
 `SGLangHiCache` is endpoint-free and has no endpoint publication race. Its
-first implementation intentionally publishes no `Ready` condition; the
-matching Pod's injection annotations are the available wiring signal until the
-separate HiCache readiness contract ships.
+`Ready` condition is derived from matching engine Pods: all participating Pods
+must carry the current CacheBackend name, UID, and generation receipt, already
+contain the configuration that the current HiCache adapter would inject, and
+be Kubernetes Ready. This confirms that the current configuration reached
+serving Pods; it does not claim a successful HiCache host-tier write/read round
+trip.
 
 `recipe-multi-tenant.yaml` spans two namespaces, so it carries a
 `# verify-samples: skip` marker — server-side dry-run can't create the

docs/concepts/cachebackend-engine-binding.md

@@ -48,14 +48,19 @@ The match is evaluated **once at pod CREATE** by the mutating webhook. The wirin
 > **Native SGLang HiCache exception.** `type: SGLangHiCache` is engine-local:
 > the controller creates no backend workload or endpoint, and the webhook does
 > not wait for `status.endpoint`. It injects the typed `spec.hiCache` launch
-> flags directly into matching SGLang Pods. The LMCache lifecycle below remains
-> the endpoint-bearing path.
+> flags directly into matching SGLang Pods. Its readiness is engine-side:
+> every participating Pod must carry the current CacheBackend name, UID, and
+> generation receipt, already contain the current adapter configuration
+> including `spec.integration.engineOverrides`, and be Kubernetes Ready. This
+> reports configuration rollout and serving availability, not a proven HiCache
+> host-tier read/write. The LMCache lifecycle below remains the endpoint-bearing
+> path.
 
 ## Lifecycle
 
 1. **Apply the CacheBackend.** `kubectl apply -f cachebackend.yaml`. The reconciler creates the managed lmcache-server Deployment + Service and publishes the resolved address in `status.endpoint`.
 2. **Deploy the engine.** Apply an engine Deployment whose pod template labels include every key/value in `spec.engineSelector.matchLabels`. New pods from that Deployment hit the mutating webhook at admission time.
-3. **The webhook claims matching pods (precondition: status.endpoint published).** When the matched CacheBackend has `status.endpoint` populated by the time admission runs, the webhook injects LMCache env vars, the `--kv-transfer-config` CLI arg, and stamps TWO annotations: `inferencecache.io/injected-by: <ns>/<name>` (operator-readable identity, visible in `kubectl describe pod`) and `inferencecache.io/injected-by-uid: <cache.UID>` (the matched CR's `metadata.uid` — an apiserver-assigned identifier that the events controller cross-checks against the live CR's UID before emitting. The check catches casual copy-paste of an injected pod's annotations into a fresh template, but it isn't a security boundary: UIDs are readable metadata, so a pod creator with `get` RBAC on CacheBackends can stamp the pair correctly). When the controller is started with `--kvevent-subscriber-image` set (empty by default) AND the matched CacheBackend has a model id configured, the kvevent-subscriber sidecar is also appended; otherwise the engine is wired without the sidecar. A controller watching pods then validates the UID annotation against the live CR's UID and records a `Normal InjectedByCacheBackend` event on the now-persisted pod, visible in `kubectl describe pod`. (The event is emitted from a controller rather than the webhook because the apiserver does not assign `metadata.uid` until after mutating admission — an event recorded from the webhook would have `involvedObject.uid=""` and would not surface under describe. The UID annotation is what lets the controller distinguish a real webhook injection from a user-supplied `injected-by` annotation when the webhook is unreachable under `failurePolicy=Ignore`.) **If the CacheBackend's `status.endpoint` is empty at admission time** (the operator applied the engine Deployment before the reconciler had a chance to publish it), the webhook fail-opens — the pod is admitted unwired, no annotations are stamped, no Event is emitted. Because admission is CREATE-only, this is permanent for that pod; recovery is `kubectl rollout restart deploy/<engine>` so new pods re-enter admission. Note that `status.matchedEnginePods` still counts the unwired pod (the selector matches its labels), so `Matched > 0` does **not** guarantee the pod was injected — the per-pod annotations and Event are the authoritative wiring signals.
+3. **The webhook claims matching pods (precondition: status.endpoint published).** When the matched CacheBackend has `status.endpoint` populated by the time admission runs, the webhook injects LMCache env vars, the `--kv-transfer-config` CLI arg, and stamps three annotations: `inferencecache.io/injected-by: <ns>/<name>` (operator-readable identity), `inferencecache.io/injected-by-uid: <cache.UID>` (the apiserver-assigned identity of the matched CR), and `inferencecache.io/injected-generation: <cache.metadata.generation>` (the spec generation validated and rendered at admission time). The events controller cross-checks the UID against the live CR before emitting; the generation receipt lets lifecycle controllers distinguish current pods from pods carrying an older configuration. These annotations are operational evidence, not a security boundary: UID and generation are readable metadata. When the controller is started with `--kvevent-subscriber-image` set (empty by default) AND the matched CacheBackend has a model id configured, the kvevent-subscriber sidecar is also appended; otherwise the engine is wired without the sidecar. A controller watching pods then validates the UID annotation against the live CR's UID and records a `Normal InjectedByCacheBackend` event on the now-persisted pod, visible in `kubectl describe pod`. (The event is emitted from a controller rather than the webhook because the apiserver does not assign `metadata.uid` until after mutating admission — an event recorded from the webhook would have `involvedObject.uid=""` and would not surface under describe.) **If the CacheBackend's `status.endpoint` is empty at admission time** (the operator applied the engine Deployment before the reconciler had a chance to publish it), the webhook fail-opens — the pod is admitted unwired, no injection annotations are stamped, no Event is emitted. Because admission is CREATE-only, this is permanent for that pod; recovery is `kubectl rollout restart deploy/<engine>` so new pods re-enter admission. Note that `status.matchedEnginePods` still counts the unwired pod (the selector matches its labels), so `Matched > 0` does **not** guarantee the pod was injected — the per-pod annotations and Event are the authoritative wiring signals.
 4. **KV events flow (when the sidecar is configured).** When the subscriber sidecar is present, it streams the engine's KV-cache events to the policy server's index, which surfaces them in `CacheBackend.status` (the index-participation status fields). Without the sidecar, the binding is still observable — `status.matchedEnginePods` and the per-pod annotation/Event still materialize from step 3 — but no KV events flow and the index-participation fields stay unset.
 5. **Observe and debug.** `kubectl get cachebackend` shows the `Matched` column — the snapshot count of pods whose labels currently match. `kubectl describe pod <engine-pod>` shows which CacheBackend (if any) claimed it.