LMCache MP mode

[fredericsun]

Summary

Migrate LMCache integrations from the legacy IP-based data plane to the typed multiprocess data plane.

This PR:

• Adds the typed LMCache multiprocess API and shared MP server renderer.
• Supports both vLLM and SGLang while preserving their engine-specific configuration surfaces.
• Supports:
  • PodLocal: one MP server per engine Pod.
  • NodeLocal: one engine-demanded MP server per active node, shared by matching engines on that node.
• Preserves inference-system-owned engine scheduling. NodeLocal servers follow scheduled engines using exact-node affinity.
• Gates engine startup until a healthy, ownership-verified same-node server is available.
• Uses Downward API node addressing and never exposes a load-balanced ClusterIP as a CUDA MP endpoint.
• Adds namespace-unique inferencecache.io/cache-domain matching to prevent ambiguous CacheBackend ownership.
• Adds per-engine same-node coverage and desired/ready server counts to CacheBackend status.
• Declares NodeLocal host ports so Kubernetes detects same-node conflicts.
• Adds UID-scoped --shm-name values to isolate NodeLocal LMCache shared-memory pools.
• Adds typed idle retention so a node-local server can survive temporary engine restarts and is removed after demand expires.
• Removes the legacy LMCache IP wire, LMCacheConnectorV1, lm:// configuration, and LMCacheServer provider path.
• Updates CRDs, generated code, RBAC, samples, documentation, unit/envtest coverage, and install smoke tests.

The migration was validated locally and on SJC dev with vLLM and SGLang TP=1 workloads, including same-node sharing, multi-node placement, L1 store/reset/retrieve, Redis outage/recovery, host-port conflict, ownership/status coverage, UID-scoped shared-memory isolation, and runtime-specific engine metrics.

Linked issues

N/A

Checklist

Vendor-neutral naming (required — see CONTRIBUTING.md)

• No oci / oracle / *.oci.com / oraclecloud.com in any API group, CRD group, proto package, gRPC service/package, Kubernetes namespace, image registry, Helm chart, or Go module path.
• Any cloud-specific integration lives in an isolated, optional adapter (pkg/adapters/.../) — never in core controllers, CRD types, the proto contract, or default config.
• No Oracle/OCI domain or namespace in sample manifests, README, or default values.
• Pre-commit naming guard passed (make install-hooks once, then it runs on every commit).

Quality

• Every human-authored commit includes a matching DCO Signed-off-by: trailer (git commit --signoff).
• make reuse-lint passes (SPDX headers and licensing metadata are complete).
• make build and make test pass locally.
• make lint clean (gofmt + go vet).
• make manifests generate produces no drift (generated code committed).
• New/changed behavior has unit tests.
• Operator-facing change (CRD columns/fields, .status, CLI, gRPC/HTTP, install bundle/RBAC, samples)? If so, the install-smoke gate asserts it (see CONTRIBUTING.md).
• CI is green.

Local make ci, including the pre-push CI gate and race tests, passes. The CI checkbox should be completed after the remote workflow finishes.

Contracts (only if touching CRDs or proto)

• Change matches the tech spec (or the spec is updated in the same PR).
• Backward compatibility considered for v1alpha1 consumers (engines, gateway clients).
• If proto/ changed, docs/design/grpc-contract.md is updated to match (the pre-commit hook enforces this).
• If CRD API types (api/v1alpha1/*_types.go) or the proto contract changed, the documentation is updated to match — the docs site (site/) and/or the design docs (docs/). CI enforces this (make verify-docs-sync); add the no-docs-needed label to waive a genuinely doc-exempt change.

[github-actions]

Codex review

Files reviewed

• .github/workflows/: c2-reconciler-canary.yml [reviewed], c6-engine-wiring-canary.yml [reviewed], default-install-smoke.yml [reviewed]
• /: README.md [reviewed]
• api/v1alpha1/: cachebackend_effective_test.go [reviewed], cachebackend_types.go [reviewed], cachebackend_types_test.go [reviewed], zz_generated.deepcopy.go [skipped — generated]
• cmd/controller/: main.go [reviewed]
• cmd/inferencecache/: doctor.go [reviewed], doctor_unit_test.go [reviewed]
• config/crd/bases/: inferencecache.io_cachebackends.yaml [skipped — generated]
• config/manager/: manager.yaml [reviewed]
• config/observability/: kustomization.yaml [reviewed], lmcache-podmonitor.yaml [reviewed], podmonitor.yaml [reviewed]
• config/overlays/gpu-validation/: kustomization.yaml [reviewed]
• config/rbac/: role.yaml [skipped — generated]
• config/samples/: README.md [reviewed], _test/cachebackend-invalid-scale-to-zero-no-min.yaml [reviewed], cache_v1alpha1_cachebackend.yaml [reviewed], cachebackend-cpu-override.yaml [reviewed], cachebackend-events-only.yaml [reviewed], cachebackend-external.yaml [reviewed], cachebackend-lmcache-cpu.yaml [reviewed], cachebackend-lmcache.yaml [reviewed], cachebackend-mooncake.yaml [reviewed], cachebackend-sglang-hicache.yaml [reviewed], cachebackend-sglang-host-only.yaml [reviewed], cachebackend-sglang-nodelocal-host-only.yaml [reviewed], cachebackend-sglang-podlocal-external-redis.yaml [reviewed], cachebackend-sglang-podlocal-host-only.yaml [reviewed], cachebackend-sglang-podlocal-managed-redis.yaml [reviewed], cachebackend-sglang.yaml [reviewed], cachebackend-vllm-nodelocal-host-only.yaml [reviewed], cachebackend-vllm-podlocal-external-redis.yaml [reviewed], cachebackend-vllm-podlocal-host-only.yaml [reviewed], cachebackend-vllm-podlocal-managed-redis.yaml [reviewed], cachebackend-with-engine.yaml [reviewed], cachebackend-with-override.yaml [reviewed], recipe-cpu-dev.yaml [reviewed], recipe-external-cache.yaml [reviewed], recipe-gpu-production.yaml [reviewed], recipe-multi-tenant.yaml [reviewed], recipe-tuning.yaml [reviewed]
• docs/cli/: doctor.md [reviewed]
• docs/concepts/: cachebackend-engine-binding.md [reviewed], cachebackend-engine-overrides.md [reviewed]
• docs/design/: cachebackend-api.md [reviewed], crd-contract.md [reviewed], kvevent-subscriber-wiring.md [reviewed], lmcache-multiprocess-migration-roadmap.md [reviewed], lmcache-server-persistence.md [reviewed], sglang-lmcache-mp-mode.md [reviewed]
• docs/observability/: alerts.md [reviewed]
• docs/: quickstart.md [reviewed]
• docs/reference-stack/: GPU-RUNBOOK.md [reviewed], README.md [reviewed], VERSIONS.md [reviewed], helm/values-reference.yaml [reviewed], manifests/deployment.yaml [reviewed], manifests/sglang-lmcache/README.md [reviewed], manifests/sglang-lmcache/deployment.yaml [reviewed], scripts/canary_c2_reconcile.sh [reviewed], scripts/canary_c6_engine_wiring.sh [reviewed], scripts/default_install_smoke.sh [reviewed]
• docs/reference/: metrics.md [reviewed]
• internal/adapters/builtin/: registry.go [reviewed], registry_test.go [reviewed]
• internal/adapters/builtin/runtime/: lmcache_mp_nodelocal.go [reviewed], lmcache_mp_nodelocal_test.go [reviewed], lmcache_mp_renderer.go [reviewed], lmcache_mp_renderer_test.go [reviewed], lmcachecheck.go [reviewed], lmcachecheck_script_test.go [reviewed], lmcachecheck_test.go [reviewed], runtime_helpers.go [reviewed], runtime_helpers_unit_test.go [reviewed], sglang_hicache.go [reviewed], sglang_hicache_test.go [reviewed], sglang_lmcache.go [reviewed], sglang_lmcache_test.go [reviewed], sglang_lmcache_wire.go [reviewed], test_helpers_test.go [reviewed], vllm_lmcache.go [reviewed], vllm_lmcache_mp.go [reviewed], vllm_lmcache_mp_test.go [reviewed], vllm_lmcache_test.go [reviewed], vllm_lmcache_wire.go [reviewed], vllm_lmcache_wire_test.go [reviewed]
• internal/adapters/builtin/storage/: effective_config.go [reviewed], lmcache_server.go [reviewed], mooncake.go [reviewed], redis.go [reviewed], redis_test.go [reviewed], registry.go [reviewed], registry_test.go [reviewed]
• internal/cli/doctor/: checks/cachebackend.go [reviewed], checks/checks.go [reviewed], checks/checks_test.go [reviewed], checks/podaudit.go [reviewed], finding.go [reviewed]
• internal/controller/: cachebackend_autoscaling_test.go [reviewed], cachebackend_dispatch.go [reviewed], cachebackend_events_only_integration_test.go [reviewed], cachebackend_events_test.go [reviewed], cachebackend_hostnetwork_test.go [reviewed], cachebackend_kernelcheck.go [reviewed], cachebackend_kvevent_gate_test.go [reviewed], cachebackend_lmcache_mp_status.go [reviewed], cachebackend_lmcache_mp_status_test.go [reviewed], cachebackend_lmcache_nodelocal.go [reviewed], cachebackend_managed.go [reviewed], cachebackend_managed_test.go [reviewed], cachebackend_matched_pods_test.go [reviewed], cachebackend_mooncake_hostnetwork_integration_test.go [reviewed], cachebackend_mp_lifecycle_test.go [reviewed], cachebackend_nodelocal_integration_test.go [reviewed], cachebackend_probe.go [reviewed], cachebackend_probe_integration_test.go [reviewed], cachebackend_reconciler.go [reviewed], cachebackend_reconciler_test.go [reviewed], cachebackend_resources_integration_test.go [reviewed], cachebackend_schema_trim_integration_test.go [reviewed], cachebackend_server_restart.go [reviewed], cachebackend_server_restart_integration_test.go [reviewed], cachebackend_server_restart_test.go [reviewed], cachebackend_serverless.go [reviewed], cachebackend_serverless_test.go [reviewed], cachebackend_status.go [reviewed], cachebackend_status_test.go [reviewed], cachebackend_t2degraded_test.go [reviewed], cachebackend_workload.go [reviewed], cachebackend_workload_test.go [reviewed], cacheindex_controller.go [reviewed], cacheindex_controller_test.go [reviewed], contract_coverage_sweep_test.go [reviewed], envtest_helpers_test.go [reviewed], integration_test.go [reviewed]
• internal/enginebinding/: metadata.go [reviewed], runtime.go [reviewed]
• internal/subscriber/: events.go [reviewed], events_test.go [reviewed]
• internal/webhook/pod/: doc.go [reviewed], envtest_integration_test.go [reviewed], podinjector.go [reviewed], podinjector_test.go [reviewed]
• internal/webhook/v1alpha1/: cachebackend_defaulter.go [reviewed], cachebackend_defaulter_envtest_test.go [reviewed], cachebackend_defaulter_test.go [reviewed], cachebackend_integration_validation.go [reviewed], cachebackend_integration_validation_test.go [reviewed], cachebackend_lmcache_mp_validation.go [reviewed], cachebackend_lmcache_mp_validation_test.go [reviewed], cachebackend_override_validation_test.go [reviewed], cachebackend_storage_validation.go [reviewed], cachebackend_storage_validation_test.go [reviewed], cachebackend_validator.go [reviewed], cachebackend_validator_test.go [reviewed], doc.go [reviewed]
• pkg/adapters/backend/: backend.go [reviewed], backend_test.go [reviewed], endpoint.go [reviewed]
• pkg/adapters/runtime/: adapter.go [reviewed], adapter_test.go [reviewed]
• site/content/en/docs/administration/: _index.md [reviewed], observability-and-alerts.md [reviewed]
• site/content/en/docs/concepts/: _index.md [reviewed], architecture.md [reviewed], cachebackend.md [reviewed], pdtopology.md [reviewed]
• site/content/en/docs/installation/: _index.md [reviewed]
• site/content/en/docs/overview/: _index.md [reviewed]
• site/content/en/docs/reference/: cli-doctor.md [reviewed], crd-api.md [reviewed], metrics.md [reviewed]
• site/content/en/docs/tasks/: bind-an-engine.md [reviewed], deploy-a-cache-backend.md [reviewed]
• test/fixtures/sglang-lmcache/: Dockerfile [reviewed]

Findings

Blocking

• config/overlays/gpu-validation/kustomization.yaml:15 — The shipped overlay hard-codes an ocir.io controller image, with additional OCIR defaults at lines 18 and 30. This violates the explicit vendor-neutral registry rule; cloud-specific configuration may not live in core config.
• docs/reference-stack/GPU-RUNBOOK.md:83 — The operator-facing runbook adds an OCI/Oracle-specific GPU-shape section. The hard rule permits cloud-specific material only in an isolated optional adapter, not core documentation.
• docs/design/lmcache-multiprocess-migration-roadmap.md:654 — The design record embeds a tenant-specific OCIR engine image. This introduces prohibited vendor-specific registry identity into a core design document.
• docs/reference-stack/VERSIONS.md:32 — The published compatibility evidence embeds a tenant-specific OCIR image. This is an operator-facing vendor-specific registry reference and conflicts with the PR’s checked vendor-neutrality claim.
• api/v1alpha1/cachebackend_types.go:41 — The existing v1alpha1 remote-storage enum is narrowed to Redis and removes the published LMCacheServer and Mooncake values. Existing persisted or declarative objects using those values become invalid after the CRD upgrade, violating the required backward-compatible v1alpha1 contract.
• api/v1alpha1/cachebackend_types.go:316 — LMCacheEngineSpec replaces the existing hostMemory, workerImage, workerPort, and remoteSerde wire shape with required topology-specific fields instead of retaining/deprecating the old fields. Existing clients can no longer round-trip or update their objects through the new schema.
• api/v1alpha1/cachebackend_types.go:475 — CacheBackendSpec removes published fields including deploymentKind, replicas, autoscaling, template, and engineHostNetwork. This is a breaking in-place CRD change; documentation of a pre-launch carve-out does not satisfy the review requirement that v1alpha1 remain backward-compatible.
• api/v1alpha1/cachebackend_types.go:852 — CacheBackendStatus removes status.endpoint and status.observedServerInstance rather than preserving deprecated fields. Existing controllers, scripts, and clients compiled against or reading those status paths break immediately on upgrade.

Should-fix

• docs/design/sglang-lmcache-mp-mode.md:132 — The changed design document still describes the removed ResolveCacheServer contract, legacy replica/autoscaling validation, vLLM lm://, status.endpoint, workerImage, and the old worker command through lines 219. It does not describe the implementation in this PR and therefore fails contract/spec synchronization.
• docs/design/lmcache-server-persistence.md:58 — The consequences section still identifies deleted source files and Mooncake as the active recommended provider despite the historical banner. Update or clearly quarantine the entire obsolete implementation section so readers are not directed to APIs and packages removed by this diff.

Nit

None.

Per-category coverage

1. Vendor-neutral naming: findings above at: config/overlays/gpu-validation/kustomization.yaml:15, docs/reference-stack/GPU-RUNBOOK.md:83, docs/design/lmcache-multiprocess-migration-roadmap.md:654, docs/reference-stack/VERSIONS.md:32
2. Contract & spec / docs sync: findings above at: api/v1alpha1/cachebackend_types.go:41, api/v1alpha1/cachebackend_types.go:316, api/v1alpha1/cachebackend_types.go:475, api/v1alpha1/cachebackend_types.go:852, docs/design/sglang-lmcache-mp-mode.md:132, docs/design/lmcache-server-persistence.md:58
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): Clean

Verdict

changes-requested

[github-actions]

Codex review

Files reviewed

• .github/workflows/: [reviewed] c2-reconciler-canary.yml, c6-engine-wiring-canary.yml, default-install-smoke.yml
• Root: [reviewed] Makefile, README.md
• api/v1alpha1/: [reviewed] cachebackend_effective_test.go, cachebackend_types.go, cachebackend_types_test.go; [skipped — generated] zz_generated.deepcopy.go
• cmd/controller/: [reviewed] main.go
• cmd/inferencecache/: [reviewed] doctor.go, doctor_unit_test.go
• config/crd/bases/: [skipped — generated] inferencecache.io_cachebackends.yaml
• config/manager/: [reviewed] manager.yaml
• config/observability/: [reviewed] kustomization.yaml, lmcache-podmonitor.yaml, podmonitor.yaml
• config/overlays/gpu-validation/: [reviewed] kustomization.yaml
• config/rbac/: [skipped — generated] role.yaml
• config/samples/: [reviewed] README.md, _test/cachebackend-invalid-scale-to-zero-no-min.yaml, cache_v1alpha1_cachebackend.yaml, cachebackend-cpu-override.yaml, cachebackend-events-only.yaml, cachebackend-external.yaml, cachebackend-lmcache-cpu.yaml, cachebackend-lmcache.yaml, cachebackend-mooncake.yaml, cachebackend-sglang-hicache.yaml, cachebackend-sglang-host-only.yaml, cachebackend-sglang-nodelocal-host-only.yaml, cachebackend-sglang-podlocal-external-redis.yaml, cachebackend-sglang-podlocal-host-only.yaml, cachebackend-sglang-podlocal-managed-redis.yaml, cachebackend-sglang.yaml, cachebackend-vllm-nodelocal-host-only.yaml, cachebackend-vllm-podlocal-external-redis.yaml, cachebackend-vllm-podlocal-host-only.yaml, cachebackend-vllm-podlocal-managed-redis.yaml, cachebackend-with-engine.yaml, cachebackend-with-override.yaml, recipe-cpu-dev.yaml, recipe-external-cache.yaml, recipe-gpu-production.yaml, recipe-multi-tenant.yaml, recipe-tuning.yaml
• docs/cli/: [reviewed] doctor.md
• docs/concepts/: [reviewed] cachebackend-engine-binding.md, cachebackend-engine-overrides.md
• docs/design/: [reviewed] cachebackend-api.md, crd-contract.md, kvevent-subscriber-wiring.md, lmcache-multiprocess-migration-roadmap.md, lmcache-server-persistence.md, sglang-lmcache-mp-mode.md
• docs/observability/: [reviewed] alerts.md
• docs/: [reviewed] quickstart.md
• docs/reference-stack/: [reviewed] GPU-RUNBOOK.md, README.md, VERSIONS.md
• docs/reference-stack/helm/: [reviewed] values-reference.yaml
• docs/reference-stack/manifests/: [reviewed] deployment.yaml
• docs/reference-stack/manifests/sglang-lmcache/: [reviewed] README.md, deployment.yaml
• docs/reference-stack/scripts/: [reviewed] canary_c2_reconcile.sh, canary_c6_engine_wiring.sh, default_install_smoke.sh
• docs/reference/: [reviewed] metrics.md
• hack/verify-samples/: [reviewed] admission_test.go
• internal/adapters/builtin/: [reviewed] registry.go, registry_test.go
• internal/adapters/builtin/runtime/: [reviewed] lmcache_mp_nodelocal.go, lmcache_mp_nodelocal_test.go, lmcache_mp_renderer.go, lmcache_mp_renderer_test.go, lmcachecheck.go, lmcachecheck_script_test.go, lmcachecheck_test.go, runtime_helpers.go, runtime_helpers_unit_test.go, sglang_hicache.go, sglang_hicache_test.go, sglang_lmcache.go, sglang_lmcache_test.go, sglang_lmcache_wire.go, test_helpers_test.go, vllm_lmcache.go, vllm_lmcache_mp.go, vllm_lmcache_mp_test.go, vllm_lmcache_test.go, vllm_lmcache_wire.go, vllm_lmcache_wire_test.go
• internal/adapters/builtin/storage/: [reviewed] effective_config.go, lmcache_server.go, mooncake.go, redis.go, redis_test.go, registry.go, registry_test.go
• internal/cli/doctor/checks/: [reviewed] cachebackend.go, checks.go, checks_test.go, podaudit.go
• internal/cli/doctor/: [reviewed] finding.go
• internal/controller/: [reviewed] cachebackend_autoscaling_test.go, cachebackend_dispatch.go, cachebackend_events_only_integration_test.go, cachebackend_events_test.go, cachebackend_hostnetwork_test.go, cachebackend_kernelcheck.go, cachebackend_kvevent_gate_test.go, cachebackend_lmcache_mp_status.go, cachebackend_lmcache_mp_status_test.go, cachebackend_lmcache_nodelocal.go, cachebackend_managed.go, cachebackend_managed_test.go, cachebackend_matched_pods_test.go, cachebackend_mooncake_hostnetwork_integration_test.go, cachebackend_mp_lifecycle_test.go, cachebackend_nodelocal_integration_test.go, cachebackend_probe.go, cachebackend_probe_integration_test.go, cachebackend_reconciler.go, cachebackend_reconciler_test.go, cachebackend_resources_integration_test.go, cachebackend_schema_trim_integration_test.go, cachebackend_server_restart.go, cachebackend_server_restart_integration_test.go, cachebackend_server_restart_test.go, cachebackend_serverless.go, cachebackend_serverless_test.go, cachebackend_status.go, cachebackend_status_test.go, cachebackend_t2degraded_test.go, cachebackend_workload.go, cachebackend_workload_test.go, cacheindex_controller.go, cacheindex_controller_test.go, contract_coverage_sweep_test.go, envtest_helpers_test.go, integration_test.go
• internal/enginebinding/: [reviewed] metadata.go, runtime.go
• internal/subscriber/: [reviewed] events.go, events_test.go
• internal/webhook/pod/: [reviewed] doc.go, envtest_integration_test.go, podinjector.go, podinjector_test.go
• internal/webhook/v1alpha1/: [reviewed] cachebackend_defaulter.go, cachebackend_defaulter_envtest_test.go, cachebackend_defaulter_test.go, cachebackend_integration_validation.go, cachebackend_integration_validation_test.go, cachebackend_lmcache_mp_validation.go, cachebackend_lmcache_mp_validation_test.go, cachebackend_override_validation_test.go, cachebackend_storage_validation.go, cachebackend_storage_validation_test.go, cachebackend_validator.go, cachebackend_validator_test.go, doc.go
• pkg/adapters/backend/: [reviewed] backend.go, backend_test.go, endpoint.go
• pkg/adapters/runtime/: [reviewed] adapter.go, adapter_test.go
• site/content/en/docs/administration/: [reviewed] _index.md, observability-and-alerts.md
• site/content/en/docs/concepts/: [reviewed] _index.md, architecture.md, cachebackend.md, pdtopology.md
• site/content/en/docs/installation/: [reviewed] _index.md
• site/content/en/docs/overview/: [reviewed] _index.md
• site/content/en/docs/reference/: [reviewed] cli-doctor.md, crd-api.md, metrics.md
• site/content/en/docs/tasks/: [reviewed] bind-an-engine.md, deploy-a-cache-backend.md
• test/fixtures/sglang-lmcache/: [reviewed] Dockerfile

Findings

Blocking

• api/v1alpha1/cachebackend_types.go:41 — The existing v1alpha1 contract removes LMCacheServer and Mooncake enum values along with multiple published spec/status fields instead of retaining or deprecating them. Existing CRs and clients consequently fail validation or lose fields through schema pruning, violating the review requirement that v1alpha1 remain backward-compatible.

Should-fix

• internal/adapters/builtin/runtime/lmcache_mp_nodelocal.go:178 — Every NodeLocal cache domain mounts the host’s entire /dev/shm; the UID-derived object name prevents accidental collisions but provides no isolation from another same-node workload able to enumerate or modify POSIX shared-memory objects. Verify and enforce an actual security boundary before describing these pools as isolated.
• internal/adapters/builtin/runtime/lmcache_mp_nodelocal.go:178 — NodeLocal L1 capacity uses an unbounded hostPath and has no node-level aggregate /dev/shm capacity check. Multiple valid CacheBackends can therefore schedule successfully while collectively exceeding the host tmpfs, causing runtime allocation failures rather than an admission or scheduling failure.

Nit

None.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: api/v1alpha1/cachebackend_types.go:41
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: internal/adapters/builtin/runtime/lmcache_mp_nodelocal.go:178

Verdict

changes-requested

[codecov]

Codecov Report

❌ Patch coverage is 94.14374% with 66 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...l/adapters/builtin/runtime/lmcache_mp_nodelocal.go	90.23%	19 Missing and 19 partials ⚠️
internal/controller/cachebackend_dispatch.go	27.27%	4 Missing and 4 partials ⚠️
...al/adapters/builtin/runtime/lmcache_mp_renderer.go	97.30%	3 Missing and 3 partials ⚠️
...nternal/adapters/builtin/runtime/sglang_lmcache.go	94.23%	3 Missing and 3 partials ⚠️
...ternal/adapters/builtin/runtime/vllm_lmcache_mp.go	96.93%	3 Missing and 3 partials ⚠️
...ternal/adapters/builtin/runtime/runtime_helpers.go	98.54%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[github-actions]

Codex review

Files reviewed

.github/workflows

• [reviewed] c2-reconciler-canary.yml
• [reviewed] c6-engine-wiring-canary.yml
• [reviewed] default-install-smoke.yml

Root

• [reviewed] Makefile
• [reviewed] README.md

api/v1alpha1

• [reviewed] cachebackend_effective_test.go
• [reviewed] cachebackend_types.go
• [reviewed] cachebackend_types_test.go
• [skipped — generated] zz_generated.deepcopy.go

cmd/controller

• [reviewed] main.go

cmd/inferencecache

• [reviewed] doctor.go
• [reviewed] doctor_unit_test.go

config/crd

• [skipped — generated] bases/inferencecache.io_cachebackends.yaml

config/manager

• [reviewed] manager.yaml

config/observability

• [reviewed] kustomization.yaml
• [reviewed] lmcache-podmonitor.yaml
• [reviewed] podmonitor.yaml

config/overlays/gpu-validation

• [reviewed] kustomization.yaml

config/rbac

• [skipped — generated] role.yaml

config/samples

• [reviewed] README.md
• [reviewed] _test/cachebackend-invalid-scale-to-zero-no-min.yaml
• [reviewed] cache_v1alpha1_cachebackend.yaml
• [reviewed] cachebackend-cpu-override.yaml
• [reviewed] cachebackend-events-only.yaml
• [reviewed] cachebackend-external.yaml
• [reviewed] cachebackend-lmcache-cpu.yaml
• [reviewed] cachebackend-lmcache.yaml
• [reviewed] cachebackend-mooncake.yaml
• [reviewed] cachebackend-sglang-hicache.yaml
• [reviewed] cachebackend-sglang-host-only.yaml
• [reviewed] cachebackend-sglang-nodelocal-host-only.yaml
• [reviewed] cachebackend-sglang-podlocal-external-redis.yaml
• [reviewed] cachebackend-sglang-podlocal-host-only.yaml
• [reviewed] cachebackend-sglang-podlocal-managed-redis.yaml
• [reviewed] cachebackend-sglang.yaml
• [reviewed] cachebackend-vllm-nodelocal-host-only.yaml
• [reviewed] cachebackend-vllm-podlocal-external-redis.yaml
• [reviewed] cachebackend-vllm-podlocal-host-only.yaml
• [reviewed] cachebackend-vllm-podlocal-managed-redis.yaml
• [reviewed] cachebackend-with-engine.yaml
• [reviewed] cachebackend-with-override.yaml
• [reviewed] recipe-cpu-dev.yaml
• [reviewed] recipe-external-cache.yaml
• [reviewed] recipe-gpu-production.yaml
• [reviewed] recipe-multi-tenant.yaml
• [reviewed] recipe-tuning.yaml

docs/cli

• [reviewed] doctor.md

docs/concepts

• [reviewed] cachebackend-engine-binding.md
• [reviewed] cachebackend-engine-overrides.md

docs/design

• [reviewed] cachebackend-api.md
• [reviewed] crd-contract.md
• [reviewed] kvevent-subscriber-wiring.md
• [reviewed] lmcache-multiprocess-migration-roadmap.md
• [reviewed] lmcache-server-persistence.md
• [reviewed] sglang-lmcache-mp-mode.md

docs/observability

• [reviewed] alerts.md

docs

• [reviewed] quickstart.md

docs/reference-stack

• [reviewed] GPU-RUNBOOK.md
• [reviewed] README.md
• [reviewed] VERSIONS.md
• [reviewed] helm/values-reference.yaml
• [reviewed] manifests/deployment.yaml
• [reviewed] manifests/sglang-lmcache/README.md
• [reviewed] manifests/sglang-lmcache/deployment.yaml
• [reviewed] scripts/canary_c2_reconcile.sh
• [reviewed] scripts/canary_c6_engine_wiring.sh
• [reviewed] scripts/default_install_smoke.sh

docs/reference

• [reviewed] metrics.md

hack/verify-samples

• [reviewed] admission_test.go

internal/adapters/builtin

• [reviewed] registry.go
• [reviewed] registry_test.go

internal/adapters/builtin/runtime

• [reviewed] lmcache_mp_nodelocal.go
• [reviewed] lmcache_mp_nodelocal_test.go
• [reviewed] lmcache_mp_renderer.go
• [reviewed] lmcache_mp_renderer_test.go
• [reviewed] lmcachecheck.go
• [reviewed] lmcachecheck_script_test.go
• [reviewed] lmcachecheck_test.go
• [reviewed] runtime_helpers.go
• [reviewed] runtime_helpers_unit_test.go
• [reviewed] sglang_hicache.go
• [reviewed] sglang_hicache_test.go
• [reviewed] sglang_lmcache.go
• [reviewed] sglang_lmcache_test.go
• [reviewed] sglang_lmcache_wire.go
• [reviewed] test_helpers_test.go
• [reviewed] vllm_lmcache.go
• [reviewed] vllm_lmcache_mp.go
• [reviewed] vllm_lmcache_mp_test.go
• [reviewed] vllm_lmcache_test.go
• [reviewed] vllm_lmcache_wire.go
• [reviewed] vllm_lmcache_wire_test.go

internal/adapters/builtin/storage

• [reviewed] effective_config.go
• [reviewed] lmcache_server.go
• [reviewed] mooncake.go
• [reviewed] redis.go
• [reviewed] redis_test.go
• [reviewed] registry.go
• [reviewed] registry_test.go

internal/cli/doctor

• [reviewed] checks/cachebackend.go
• [reviewed] checks/checks.go
• [reviewed] checks/checks_test.go
• [reviewed] checks/podaudit.go
• [reviewed] finding.go

internal/controller

• [reviewed] cachebackend_autoscaling_test.go
• [reviewed] cachebackend_dispatch.go
• [reviewed] cachebackend_events_only_integration_test.go
• [reviewed] cachebackend_events_test.go
• [reviewed] cachebackend_hostnetwork_test.go
• [reviewed] cachebackend_kernelcheck.go
• [reviewed] cachebackend_kvevent_gate_test.go
• [reviewed] cachebackend_lmcache_mp_status.go
• [reviewed] cachebackend_lmcache_mp_status_test.go
• [reviewed] cachebackend_lmcache_nodelocal.go
• [reviewed] cachebackend_managed.go
• [reviewed] cachebackend_managed_test.go
• [reviewed] cachebackend_matched_pods_test.go
• [reviewed] cachebackend_mooncake_hostnetwork_integration_test.go
• [reviewed] cachebackend_mp_lifecycle_test.go
• [reviewed] cachebackend_nodelocal_integration_test.go
• [reviewed] cachebackend_probe.go
• [reviewed] cachebackend_probe_integration_test.go
• [reviewed] cachebackend_reconciler.go
• [reviewed] cachebackend_reconciler_test.go
• [reviewed] cachebackend_resources_integration_test.go
• [reviewed] cachebackend_schema_trim_integration_test.go
• [reviewed] cachebackend_server_restart.go
• [reviewed] cachebackend_server_restart_integration_test.go
• [reviewed] cachebackend_server_restart_test.go
• [reviewed] cachebackend_serverless.go
• [reviewed] cachebackend_serverless_test.go
• [reviewed] cachebackend_status.go
• [reviewed] cachebackend_status_test.go
• [reviewed] cachebackend_t2degraded_test.go
• [reviewed] cachebackend_workload.go
• [reviewed] cachebackend_workload_test.go
• [reviewed] cacheindex_controller.go
• [reviewed] cacheindex_controller_test.go
• [reviewed] contract_coverage_sweep_test.go
• [reviewed] envtest_helpers_test.go
• [reviewed] integration_test.go

internal/enginebinding

• [reviewed] metadata.go
• [reviewed] runtime.go

internal/subscriber

• [reviewed] events.go
• [reviewed] events_test.go

internal/webhook/pod

• [reviewed] doc.go
• [reviewed] envtest_integration_test.go
• [reviewed] podinjector.go
• [reviewed] podinjector_test.go

internal/webhook/v1alpha1

• [reviewed] cachebackend_defaulter.go
• [reviewed] cachebackend_defaulter_envtest_test.go
• [reviewed] cachebackend_defaulter_test.go
• [reviewed] cachebackend_integration_validation.go
• [reviewed] cachebackend_integration_validation_test.go
• [reviewed] cachebackend_lmcache_mp_validation.go
• [reviewed] cachebackend_lmcache_mp_validation_test.go
• [reviewed] cachebackend_override_validation_test.go
• [reviewed] cachebackend_storage_validation.go
• [reviewed] cachebackend_storage_validation_test.go
• [reviewed] cachebackend_validator.go
• [reviewed] cachebackend_validator_test.go
• [reviewed] doc.go

pkg/adapters/backend

• [reviewed] backend.go
• [reviewed] backend_test.go
• [reviewed] endpoint.go

pkg/adapters/runtime

• [reviewed] adapter.go
• [reviewed] adapter_test.go

site/content/en/docs/administration

• [reviewed] _index.md
• [reviewed] observability-and-alerts.md

site/content/en/docs/concepts

• [reviewed] _index.md
• [reviewed] architecture.md
• [reviewed] cachebackend.md
• [reviewed] pdtopology.md

site/content/en/docs/installation

• [reviewed] _index.md

site/content/en/docs/overview

• [reviewed] _index.md

site/content/en/docs/reference

• [reviewed] cli-doctor.md
• [reviewed] crd-api.md
• [reviewed] metrics.md

site/content/en/docs/tasks

• [reviewed] bind-an-engine.md
• [reviewed] deploy-a-cache-backend.md

test/fixtures/sglang-lmcache

• [reviewed] Dockerfile

Findings

Blocking

• api/v1alpha1/cachebackend_types.go:41 — The published v1alpha1 provider enum is narrowed to only Redis, removing LMCacheServer and Mooncake. Existing persisted manifests using either value will fail schema validation on updates, violating the required backward-compatible contract.

• api/v1alpha1/cachebackend_types.go:475 — CacheBackendSpec removes published fields including deploymentKind, replicas, autoscaling, and template without introducing a new API version or retaining deprecated compatibility fields. Existing v1alpha1 objects lose accepted configuration and cannot safely round-trip through the new CRD.

• api/v1alpha1/cachebackend_types.go:489 — LMCacheEngineSpec has replaced the prior hostMemory, workerImage, workerPort, and remoteSerde surface with mandatory typed topology objects in place. This is another wire-breaking change to the same served/storage version, rather than an additive migration.

• api/v1alpha1/cachebackend_types.go:509 — engineSelector is restricted from the previously published label-selector map to exactly one inferencecache.io/cache-domain label. Previously valid selectors become invalid on update, so this ownership change needs a versioned migration or compatibility path.

Should-fix

None.

Nit

• internal/controller/cachebackend_reconciler.go:30 — The reconciler documentation still says Pods are deliberately not watched, while SetupWithManager now installs a Pod watch at line 260. Update the comment and APIReader field documentation so the implementation rationale is not self-contradictory.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: api/v1alpha1/cachebackend_types.go:41, api/v1alpha1/cachebackend_types.go:475, api/v1alpha1/cachebackend_types.go:489, api/v1alpha1/cachebackend_types.go:509
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: internal/controller/cachebackend_reconciler.go:30

Verdict

changes-requested

[github-actions]

Codex review

Files reviewed

.github/workflows

• [reviewed] c2-reconciler-canary.yml, c6-engine-wiring-canary.yml, default-install-smoke.yml

Repository root

• [reviewed] Makefile, README.md

api/v1alpha1

• [reviewed] cachebackend_effective_test.go, cachebackend_types.go, cachebackend_types_test.go
• [skipped — generated] zz_generated.deepcopy.go

cmd/controller

• [reviewed] main.go

cmd/inferencecache

• [reviewed] doctor.go, doctor_unit_test.go

config/crd/bases

• [skipped — generated] inferencecache.io_cachebackends.yaml

config/manager

• [reviewed] manager.yaml

config/observability

• [reviewed] kustomization.yaml, lmcache-podmonitor.yaml, podmonitor.yaml

config/overlays/gpu-validation

• [reviewed] kustomization.yaml

config/rbac

• [skipped — generated] role.yaml

config/samples

• [reviewed] README.md
• [reviewed] _test/cachebackend-invalid-scale-to-zero-no-min.yaml
• [reviewed] cache_v1alpha1_cachebackend.yaml, cachebackend-cpu-override.yaml, cachebackend-events-only.yaml, cachebackend-external.yaml
• [reviewed] cachebackend-lmcache-cpu.yaml, cachebackend-lmcache.yaml, cachebackend-mooncake.yaml
• [reviewed] cachebackend-sglang-hicache.yaml, cachebackend-sglang-host-only.yaml, cachebackend-sglang-nodelocal-host-only.yaml
• [reviewed] cachebackend-sglang-podlocal-external-redis.yaml, cachebackend-sglang-podlocal-host-only.yaml, cachebackend-sglang-podlocal-managed-redis.yaml, cachebackend-sglang.yaml
• [reviewed] cachebackend-vllm-nodelocal-host-only.yaml, cachebackend-vllm-podlocal-external-redis.yaml, cachebackend-vllm-podlocal-host-only.yaml, cachebackend-vllm-podlocal-managed-redis.yaml
• [reviewed] cachebackend-with-engine.yaml, cachebackend-with-override.yaml
• [reviewed] recipe-cpu-dev.yaml, recipe-external-cache.yaml, recipe-gpu-production.yaml, recipe-multi-tenant.yaml, recipe-tuning.yaml

docs/cli

• [reviewed] doctor.md

docs/concepts

• [reviewed] cachebackend-engine-binding.md, cachebackend-engine-overrides.md

docs/design

• [reviewed] cachebackend-api.md, crd-contract.md, kvevent-subscriber-wiring.md
• [reviewed] lmcache-multiprocess-migration-roadmap.md, lmcache-server-persistence.md, sglang-lmcache-mp-mode.md

docs/observability

• [reviewed] alerts.md

docs

• [reviewed] quickstart.md

docs/reference-stack

• [reviewed] GPU-RUNBOOK.md, README.md, VERSIONS.md

docs/reference-stack/helm

• [reviewed] values-reference.yaml

docs/reference-stack/manifests

• [reviewed] deployment.yaml

docs/reference-stack/manifests/sglang-lmcache

• [reviewed] README.md, deployment.yaml

docs/reference-stack/scripts

• [reviewed] canary_c2_reconcile.sh, canary_c6_engine_wiring.sh, default_install_smoke.sh

docs/reference

• [reviewed] metrics.md

hack/verify-samples

• [reviewed] admission_test.go

internal/adapters/builtin

• [reviewed] registry.go, registry_test.go

internal/adapters/builtin/runtime

• [reviewed] lmcache_mp_nodelocal.go, lmcache_mp_nodelocal_test.go, lmcache_mp_renderer.go, lmcache_mp_renderer_test.go
• [reviewed] lmcachecheck.go, lmcachecheck_script_test.go, lmcachecheck_test.go
• [reviewed] runtime_helpers.go, runtime_helpers_unit_test.go
• [reviewed] sglang_hicache.go, sglang_hicache_test.go, sglang_lmcache.go, sglang_lmcache_test.go, sglang_lmcache_wire.go
• [reviewed] test_helpers_test.go
• [reviewed] vllm_lmcache.go, vllm_lmcache_mp.go, vllm_lmcache_mp_test.go, vllm_lmcache_test.go, vllm_lmcache_wire.go, vllm_lmcache_wire_test.go

internal/adapters/builtin/storage

• [reviewed] effective_config.go, effective_config_test.go
• [reviewed] lmcache_server.go, mooncake.go
• [reviewed] redis.go, redis_test.go, registry.go, registry_test.go

internal/cli/doctor/checks

• [reviewed] cachebackend.go, checks.go, checks_test.go, podaudit.go

internal/cli/doctor

• [reviewed] finding.go

internal/controller

• [reviewed] cachebackend_autoscaling_test.go, cachebackend_dispatch.go
• [reviewed] cachebackend_events_only_integration_test.go, cachebackend_events_test.go
• [reviewed] cachebackend_hostnetwork_test.go, cachebackend_kernelcheck.go, cachebackend_kvevent_gate_test.go
• [reviewed] cachebackend_lmcache_mp_status.go, cachebackend_lmcache_mp_status_test.go, cachebackend_lmcache_nodelocal.go
• [reviewed] cachebackend_managed.go, cachebackend_managed_test.go, cachebackend_matched_pods_test.go
• [reviewed] cachebackend_mooncake_hostnetwork_integration_test.go, cachebackend_mp_lifecycle_test.go, cachebackend_nodelocal_integration_test.go
• [reviewed] cachebackend_probe.go, cachebackend_probe_integration_test.go
• [reviewed] cachebackend_reconciler.go, cachebackend_reconciler_test.go
• [reviewed] cachebackend_resources_integration_test.go, cachebackend_schema_trim_integration_test.go
• [reviewed] cachebackend_server_restart.go, cachebackend_server_restart_integration_test.go, cachebackend_server_restart_test.go
• [reviewed] cachebackend_serverless.go, cachebackend_serverless_test.go
• [reviewed] cachebackend_status.go, cachebackend_status_test.go, cachebackend_t2degraded_test.go
• [reviewed] cachebackend_workload.go, cachebackend_workload_test.go
• [reviewed] cacheindex_controller.go, cacheindex_controller_test.go, contract_coverage_sweep_test.go, envtest_helpers_test.go, integration_test.go

internal/enginebinding

• [reviewed] metadata.go, runtime.go

internal/subscriber

• [reviewed] events.go, events_test.go

internal/webhook/pod

• [reviewed] doc.go, envtest_integration_test.go, podinjector.go, podinjector_test.go

internal/webhook/v1alpha1

• [reviewed] cachebackend_defaulter.go, cachebackend_defaulter_envtest_test.go, cachebackend_defaulter_test.go
• [reviewed] cachebackend_integration_validation.go, cachebackend_integration_validation_test.go
• [reviewed] cachebackend_lmcache_mp_validation.go, cachebackend_lmcache_mp_validation_test.go
• [reviewed] cachebackend_override_validation_test.go
• [reviewed] cachebackend_storage_validation.go, cachebackend_storage_validation_test.go
• [reviewed] cachebackend_validator.go, cachebackend_validator_test.go, doc.go

pkg/adapters/backend

• [reviewed] backend.go, backend_test.go, endpoint.go

pkg/adapters/runtime

• [reviewed] adapter.go, adapter_test.go

site/content/en/docs/administration

• [reviewed] _index.md, observability-and-alerts.md

site/content/en/docs/concepts

• [reviewed] _index.md, architecture.md, cachebackend.md, pdtopology.md

site/content/en/docs/installation

• [reviewed] _index.md

site/content/en/docs/overview

• [reviewed] _index.md

site/content/en/docs/reference

• [reviewed] cli-doctor.md, crd-api.md, metrics.md

site/content/en/docs/tasks

• [reviewed] bind-an-engine.md, deploy-a-cache-backend.md

test/fixtures/sglang-lmcache

• [reviewed] Dockerfile

Findings

Blocking

• api/v1alpha1/cachebackend_types.go:481 — This rewrites the served v1alpha1 contract in place, removing fields such as spec.replicas, autoscaling, deploymentKind, template, legacy LMCache settings/providers, and status fields rather than retaining or converting them. Existing clients and stored objects will be pruned, rejected on update, or become unreconcilable, violating the required backward-compatibility rule.

Should-fix

• internal/adapters/builtin/runtime/lmcache_mp_nodelocal.go:136 — NodeLocal uses HostPathDirectoryOrCreate, which creates the UID directory as root-owned 0755, but no initialization step makes it writable by the engine/server runtime identities. Non-root engine images or configured non-root security contexts therefore cannot create/open the shared-memory objects; the roadmap itself only validates root-running images.
• docs/design/cachebackend-api.md:697 — The document says the controller does not watch engine Pod status, but cachebackend_reconciler.go:256 now installs an unconditional Pod watch. This gives operators an incorrect account of status/event update latency and should be synchronized with the implementation.

Nit

• README.md:39 — The README says legacy-compatibility sections are retained “until Phase 7,” while this PR declares Phase 7 complete and physically removes that implementation. Reword this as historical documentation or remove the obsolete future-tense statement.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: api/v1alpha1/cachebackend_types.go:481, docs/design/cachebackend-api.md:697, README.md:39
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: internal/adapters/builtin/runtime/lmcache_mp_nodelocal.go:136

Verdict

changes-requested

[github-actions]

Codex review

Files reviewed

• .github/workflows/: c2-reconciler-canary.yml [reviewed], c6-engine-wiring-canary.yml [reviewed], default-install-smoke.yml [reviewed]
• /: Makefile [reviewed], README.md [reviewed]
• api/v1alpha1/: cachebackend_effective_test.go [reviewed], cachebackend_types.go [reviewed], cachebackend_types_test.go [reviewed], zz_generated.deepcopy.go [skipped — generated]
• cmd/controller/: main.go [reviewed]
• cmd/inferencecache/: doctor.go [reviewed], doctor_unit_test.go [reviewed]
• config/crd/bases/: inferencecache.io_cachebackends.yaml [skipped — generated]
• config/manager/: manager.yaml [reviewed]
• config/observability/: kustomization.yaml [reviewed], lmcache-podmonitor.yaml [reviewed], podmonitor.yaml [reviewed]
• config/overlays/gpu-validation/: kustomization.yaml [reviewed]
• config/rbac/: role.yaml [skipped — generated]
• config/samples/: README.md [reviewed], _test/cachebackend-invalid-scale-to-zero-no-min.yaml [reviewed], cache_v1alpha1_cachebackend.yaml [reviewed], cachebackend-cpu-override.yaml [reviewed], cachebackend-events-only.yaml [reviewed], cachebackend-external.yaml [reviewed], cachebackend-lmcache-cpu.yaml [reviewed], cachebackend-lmcache.yaml [reviewed], cachebackend-mooncake.yaml [reviewed], cachebackend-sglang-hicache.yaml [reviewed], cachebackend-sglang-host-only.yaml [reviewed], cachebackend-sglang-nodelocal-host-only.yaml [reviewed], cachebackend-sglang-podlocal-external-redis.yaml [reviewed], cachebackend-sglang-podlocal-host-only.yaml [reviewed], cachebackend-sglang-podlocal-managed-redis.yaml [reviewed], cachebackend-sglang.yaml [reviewed], cachebackend-vllm-nodelocal-host-only.yaml [reviewed], cachebackend-vllm-podlocal-external-redis.yaml [reviewed], cachebackend-vllm-podlocal-host-only.yaml [reviewed], cachebackend-vllm-podlocal-managed-redis.yaml [reviewed], cachebackend-with-engine.yaml [reviewed], cachebackend-with-override.yaml [reviewed], recipe-cpu-dev.yaml [reviewed], recipe-external-cache.yaml [reviewed], recipe-gpu-production.yaml [reviewed], recipe-multi-tenant.yaml [reviewed], recipe-tuning.yaml [reviewed]
• docs/cli/: doctor.md [reviewed]
• docs/concepts/: cachebackend-engine-binding.md [reviewed], cachebackend-engine-overrides.md [reviewed]
• docs/design/: cachebackend-api.md [reviewed], crd-contract.md [reviewed], kvevent-subscriber-wiring.md [reviewed], lmcache-multiprocess-migration-roadmap.md [reviewed], lmcache-server-persistence.md [reviewed], sglang-lmcache-mp-mode.md [reviewed]
• docs/observability/: alerts.md [reviewed]
• docs/: quickstart.md [reviewed]
• docs/reference-stack/: GPU-RUNBOOK.md [reviewed], README.md [reviewed], VERSIONS.md [reviewed], helm/values-reference.yaml [reviewed], manifests/deployment.yaml [reviewed], manifests/sglang-lmcache/README.md [reviewed], manifests/sglang-lmcache/deployment.yaml [reviewed], scripts/canary_c2_reconcile.sh [reviewed], scripts/canary_c6_engine_wiring.sh [reviewed], scripts/default_install_smoke.sh [reviewed]
• docs/reference/: metrics.md [reviewed]
• hack/verify-samples/: admission_test.go [reviewed]
• internal/adapters/builtin/: registry.go [reviewed], registry_test.go [reviewed]
• internal/adapters/builtin/runtime/: lmcache_mp_nodelocal.go [reviewed], lmcache_mp_nodelocal_test.go [reviewed], lmcache_mp_renderer.go [reviewed], lmcache_mp_renderer_test.go [reviewed], lmcachecheck.go [reviewed], lmcachecheck_script_test.go [reviewed], lmcachecheck_test.go [reviewed], runtime_helpers.go [reviewed], runtime_helpers_unit_test.go [reviewed], sglang_hicache.go [reviewed], sglang_hicache_test.go [reviewed], sglang_lmcache.go [reviewed], sglang_lmcache_test.go [reviewed], sglang_lmcache_wire.go [reviewed], test_helpers_test.go [reviewed], vllm_lmcache.go [reviewed], vllm_lmcache_mp.go [reviewed], vllm_lmcache_mp_test.go [reviewed], vllm_lmcache_test.go [reviewed], vllm_lmcache_wire.go [reviewed], vllm_lmcache_wire_test.go [reviewed]
• internal/adapters/builtin/storage/: effective_config.go [reviewed], effective_config_test.go [reviewed], lmcache_server.go [reviewed], mooncake.go [reviewed], redis.go [reviewed], redis_test.go [reviewed], registry.go [reviewed], registry_test.go [reviewed]
• internal/cli/doctor/checks/: cachebackend.go [reviewed], checks.go [reviewed], checks_test.go [reviewed], podaudit.go [reviewed]
• internal/cli/doctor/: finding.go [reviewed]
• internal/controller/: cachebackend_autoscaling_test.go [reviewed], cachebackend_dispatch.go [reviewed], cachebackend_events_only_integration_test.go [reviewed], cachebackend_events_test.go [reviewed], cachebackend_hostnetwork_test.go [reviewed], cachebackend_kernelcheck.go [reviewed], cachebackend_kvevent_gate_test.go [reviewed], cachebackend_lmcache_mp_status.go [reviewed], cachebackend_lmcache_mp_status_test.go [reviewed], cachebackend_lmcache_nodelocal.go [reviewed], cachebackend_managed.go [reviewed], cachebackend_managed_test.go [reviewed], cachebackend_matched_pods_test.go [reviewed], cachebackend_mooncake_hostnetwork_integration_test.go [reviewed], cachebackend_mp_lifecycle_test.go [reviewed], cachebackend_nodelocal_integration_test.go [reviewed], cachebackend_probe.go [reviewed], cachebackend_probe_integration_test.go [reviewed], cachebackend_reconciler.go [reviewed], cachebackend_reconciler_test.go [reviewed], cachebackend_resources_integration_test.go [reviewed], cachebackend_schema_trim_integration_test.go [reviewed], cachebackend_server_restart.go [reviewed], cachebackend_server_restart_integration_test.go [reviewed], cachebackend_server_restart_test.go [reviewed], cachebackend_serverless.go [reviewed], cachebackend_serverless_test.go [reviewed], cachebackend_status.go [reviewed], cachebackend_status_test.go [reviewed], cachebackend_t2degraded_test.go [reviewed], cachebackend_workload.go [reviewed], cachebackend_workload_test.go [reviewed], cacheindex_controller.go [reviewed], cacheindex_controller_test.go [reviewed], contract_coverage_sweep_test.go [reviewed], envtest_helpers_test.go [reviewed], integration_test.go [reviewed]
• internal/enginebinding/: metadata.go [reviewed], runtime.go [reviewed]
• internal/subscriber/: events.go [reviewed], events_test.go [reviewed]
• internal/webhook/pod/: doc.go [reviewed], envtest_integration_test.go [reviewed], podinjector.go [reviewed], podinjector_test.go [reviewed]
• internal/webhook/v1alpha1/: cachebackend_defaulter.go [reviewed], cachebackend_defaulter_envtest_test.go [reviewed], cachebackend_defaulter_test.go [reviewed], cachebackend_integration_validation.go [reviewed], cachebackend_integration_validation_test.go [reviewed], cachebackend_lmcache_mp_validation.go [reviewed], cachebackend_lmcache_mp_validation_test.go [reviewed], cachebackend_override_validation_test.go [reviewed], cachebackend_storage_validation.go [reviewed], cachebackend_storage_validation_test.go [reviewed], cachebackend_validator.go [reviewed], cachebackend_validator_test.go [reviewed], doc.go [reviewed]
• pkg/adapters/backend/: backend.go [reviewed], backend_test.go [reviewed], endpoint.go [reviewed]
• pkg/adapters/runtime/: adapter.go [reviewed], adapter_test.go [reviewed]
• site/content/en/docs/administration/: _index.md [reviewed], observability-and-alerts.md [reviewed]
• site/content/en/docs/concepts/: _index.md [reviewed], architecture.md [reviewed], cachebackend.md [reviewed], pdtopology.md [reviewed]
• site/content/en/docs/installation/: _index.md [reviewed]
• site/content/en/docs/overview/: _index.md [reviewed]
• site/content/en/docs/reference/: cli-doctor.md [reviewed], crd-api.md [reviewed], metrics.md [reviewed]
• site/content/en/docs/tasks/: bind-an-engine.md [reviewed], deploy-a-cache-backend.md [reviewed]
• test/fixtures/sglang-lmcache/: Dockerfile [reviewed]

Findings

Blocking

• api/v1alpha1/cachebackend_types.go:313 — The PR replaces the existing v1alpha1 LMCache fields with required topology/podLocal/nodeLocal fields and removes legacy hostMemory, workerImage, workerPort, and remoteSerde; it likewise removes existing managed-workload, LMCacheServer, Mooncake, autoscaling, and status fields elsewhere in this same API. Existing persisted manifests and typed clients therefore stop round-tripping or validating after a CRD upgrade, violating the review requirement that v1alpha1 remain backward-compatible; this needs a new API version/conversion path or a compatibility period.

Should-fix

• internal/adapters/builtin/storage/redis.go:147 — Authenticated managed Redis always executes /usr/local/bin/docker-entrypoint.sh, even though spec.remoteStorage.redis.image permits arbitrary Redis-compatible images. A valid custom image without the official Docker image’s private entrypoint path will crash only when authentication is enabled; either constrain/validate the image contract or launch Redis without relying on that image-specific path.

Nit

• internal/webhook/pod/podinjector.go:40 — The comment says AnnotationInjectedBy is informational and the webhook “only reads” the skip annotation, but NodeLocal demand, status, and pod-to-backend event mapping now rely on the injected-by name/UID/generation annotations. Update the exported documentation to describe their operational ownership role.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: api/v1alpha1/cachebackend_types.go:313
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: internal/adapters/builtin/storage/redis.go:147, internal/webhook/pod/podinjector.go:40

Verdict

changes-requested

[github-actions]

Codex review

Files reviewed

• .github/workflows/: [reviewed] c2-reconciler-canary.yml, c6-engine-wiring-canary.yml, default-install-smoke.yml
• root: [reviewed] Makefile, README.md
• api/v1alpha1/: [reviewed] cachebackend_effective_test.go, cachebackend_types.go, cachebackend_types_test.go; [skipped — generated] zz_generated.deepcopy.go
• cmd/controller/: [reviewed] main.go
• cmd/inferencecache/: [reviewed] doctor.go, doctor_unit_test.go
• config/crd/bases/: [skipped — generated] inferencecache.io_cachebackends.yaml
• config/manager/: [reviewed] manager.yaml
• config/observability/: [reviewed] kustomization.yaml, lmcache-podmonitor.yaml, podmonitor.yaml
• config/overlays/gpu-validation/: [reviewed] kustomization.yaml
• config/rbac/: [skipped — generated] role.yaml
• config/samples/: [reviewed] README.md, cache_v1alpha1_cachebackend.yaml, cachebackend-cpu-override.yaml, cachebackend-events-only.yaml, cachebackend-external.yaml, cachebackend-invalid-scale-to-zero-no-min.yaml, cachebackend-lmcache-cpu.yaml, cachebackend-lmcache.yaml, cachebackend-mooncake.yaml, cachebackend-sglang-hicache.yaml, cachebackend-sglang-host-only.yaml, cachebackend-sglang-nodelocal-host-only.yaml, cachebackend-sglang-podlocal-external-redis.yaml, cachebackend-sglang-podlocal-host-only.yaml, cachebackend-sglang-podlocal-managed-redis.yaml, cachebackend-sglang.yaml, cachebackend-vllm-nodelocal-host-only.yaml, cachebackend-vllm-podlocal-external-redis.yaml, cachebackend-vllm-podlocal-host-only.yaml, cachebackend-vllm-podlocal-managed-redis.yaml, cachebackend-with-engine.yaml, cachebackend-with-override.yaml, recipe-cpu-dev.yaml, recipe-external-cache.yaml, recipe-gpu-production.yaml, recipe-multi-tenant.yaml, recipe-tuning.yaml
• docs/cli/: [reviewed] doctor.md
• docs/concepts/: [reviewed] cachebackend-engine-binding.md, cachebackend-engine-overrides.md
• docs/design/: [reviewed] cachebackend-api.md, crd-contract.md, kvevent-subscriber-wiring.md, lmcache-multiprocess-migration-roadmap.md, lmcache-server-persistence.md, sglang-lmcache-mp-mode.md
• docs/observability/: [reviewed] alerts.md
• docs/: [reviewed] quickstart.md
• docs/reference-stack/: [reviewed] GPU-RUNBOOK.md, README.md, VERSIONS.md
• docs/reference-stack/helm/: [reviewed] values-reference.yaml
• docs/reference-stack/manifests/: [reviewed] deployment.yaml
• docs/reference-stack/manifests/sglang-lmcache/: [reviewed] README.md, deployment.yaml
• docs/reference-stack/scripts/: [reviewed] canary_c2_reconcile.sh, canary_c6_engine_wiring.sh, default_install_smoke.sh
• docs/reference/: [reviewed] metrics.md
• hack/verify-samples/: [reviewed] admission_test.go
• internal/adapters/builtin/: [reviewed] registry.go, registry_test.go
• internal/adapters/builtin/runtime/: [reviewed] lmcache_mp_nodelocal.go, lmcache_mp_nodelocal_test.go, lmcache_mp_renderer.go, lmcache_mp_renderer_test.go, lmcachecheck.go, lmcachecheck_script_test.go, lmcachecheck_test.go, runtime_helpers.go, runtime_helpers_unit_test.go, sglang_hicache.go, sglang_hicache_test.go, sglang_lmcache.go, sglang_lmcache_test.go, sglang_lmcache_wire.go, test_helpers_test.go, vllm_lmcache.go, vllm_lmcache_mp.go, vllm_lmcache_mp_test.go, vllm_lmcache_test.go, vllm_lmcache_wire.go, vllm_lmcache_wire_test.go
• internal/adapters/builtin/storage/: [reviewed] effective_config.go, effective_config_test.go, lmcache_server.go, mooncake.go, redis.go, redis_test.go, registry.go, registry_test.go
• internal/cli/doctor/checks/: [reviewed] cachebackend.go, checks.go, checks_test.go, podaudit.go
• internal/cli/doctor/: [reviewed] finding.go
• internal/controller/: [reviewed] cachebackend_autoscaling_test.go, cachebackend_dispatch.go, cachebackend_events_only_integration_test.go, cachebackend_events_test.go, cachebackend_hostnetwork_test.go, cachebackend_kernelcheck.go, cachebackend_kvevent_gate_test.go, cachebackend_lmcache_mp_status.go, cachebackend_lmcache_mp_status_test.go, cachebackend_lmcache_nodelocal.go, cachebackend_managed.go, cachebackend_managed_test.go, cachebackend_matched_pods_test.go, cachebackend_mooncake_hostnetwork_integration_test.go, cachebackend_mp_lifecycle_test.go, cachebackend_nodelocal_integration_test.go, cachebackend_probe.go, cachebackend_probe_integration_test.go, cachebackend_reconciler.go, cachebackend_reconciler_test.go, cachebackend_resources_integration_test.go, cachebackend_schema_trim_integration_test.go, cachebackend_server_restart.go, cachebackend_server_restart_integration_test.go, cachebackend_server_restart_test.go, cachebackend_serverless.go, cachebackend_serverless_test.go, cachebackend_status.go, cachebackend_status_test.go, cachebackend_t2degraded_test.go, cachebackend_workload.go, cachebackend_workload_test.go, cacheindex_controller.go, cacheindex_controller_test.go, contract_coverage_sweep_test.go, envtest_helpers_test.go, integration_test.go
• internal/enginebinding/: [reviewed] metadata.go, runtime.go
• internal/subscriber/: [reviewed] events.go, events_test.go
• internal/webhook/pod/: [reviewed] doc.go, envtest_integration_test.go, podinjector.go, podinjector_test.go
• internal/webhook/v1alpha1/: [reviewed] cachebackend_defaulter.go, cachebackend_defaulter_envtest_test.go, cachebackend_defaulter_test.go, cachebackend_integration_validation.go, cachebackend_integration_validation_test.go, cachebackend_lmcache_mp_validation.go, cachebackend_lmcache_mp_validation_test.go, cachebackend_override_validation_test.go, cachebackend_storage_validation.go, cachebackend_storage_validation_test.go, cachebackend_validator.go, cachebackend_validator_test.go, doc.go
• pkg/adapters/backend/: [reviewed] backend.go, backend_test.go, endpoint.go
• pkg/adapters/runtime/: [reviewed] adapter.go, adapter_test.go
• site/content/en/docs/administration/: [reviewed] _index.md, observability-and-alerts.md
• site/content/en/docs/concepts/: [reviewed] _index.md, architecture.md, cachebackend.md, pdtopology.md
• site/content/en/docs/installation/: [reviewed] _index.md
• site/content/en/docs/overview/: [reviewed] _index.md
• site/content/en/docs/reference/: [reviewed] cli-doctor.md, crd-api.md, metrics.md
• site/content/en/docs/tasks/: [reviewed] bind-an-engine.md, deploy-a-cache-backend.md
• test/fixtures/sglang-lmcache/: [reviewed] Dockerfile

Findings

Blocking

• api/v1alpha1/cachebackend_types.go:41 — The published v1alpha1 provider enum removes LMCacheServer and Mooncake instead of retaining deprecated values or introducing a new API version. Existing CRs using either provider become invalid on CRD upgrade, violating the required backward-compatibility contract.
• api/v1alpha1/cachebackend_types.go:481 — CacheBackendSpec removes published fields including deploymentKind, replicas, autoscaling, and the generic workload scheduling fields in place. Existing manifests and typed clients no longer round-trip against the same v1alpha1; this requires compatibility fields or a versioned conversion/migration.
• api/v1alpha1/cachebackend_types.go:852 — status.endpoint and status.observedServerInstance are removed from the existing status schema rather than retained as deprecated compatibility fields. Consumers compiled against or reading the published v1alpha1 status contract will break.
• internal/webhook/v1alpha1/cachebackend_validator.go:152 — Every update to an existing CacheBackend is rejected until its previously valid selector is replaced with the new single inferencecache.io/cache-domain selector. This traps old objects from unrelated updates and is an explicitly breaking behavioral change to v1alpha1.
• internal/controller/cachebackend_lmcache_nodelocal.go:244 — NodeLocal provisioning treats user-controlled name/UID annotations as proof that the webhook injected a Pod. Because the webhook is fail-open and CacheBackend UIDs can be copied from an existing Pod or readable CR, a Pod creator can spoof these annotations and make the controller create a host-networked, hostPath-mounted server on the chosen node.

Should-fix

• internal/webhook/pod/podinjector.go:58 — The documentation calls the UID annotation “webhook-only proof-of-injection” and claims it closes the failurePolicy=Ignore hole, but Kubernetes annotations and object UIDs are not secrets or unforgeable provenance. Replace this trust model with admission-enforced data that users cannot supply, or ensure the controller never grants/provisions additional capability based solely on these annotations.

Nit

None.

Per-category coverage

1. Vendor-neutral naming: Clean
2. Contract & spec / docs sync: findings above at: api/v1alpha1/cachebackend_types.go:41, api/v1alpha1/cachebackend_types.go:481, api/v1alpha1/cachebackend_types.go:852, internal/webhook/v1alpha1/cachebackend_validator.go:152
3. gRPC fail-open semantics: Clean
4. Quality (tests / error wrapping / package placement / generated drift): findings above at: internal/controller/cachebackend_lmcache_nodelocal.go:244, internal/webhook/pod/podinjector.go:58

Verdict

changes-requested

[fredericsun]

I am not addressing the remaining blockers in this PR for two reasons:

• The v1alpha1 compatibility findings conflict with the documented migration decision: this is a pre-launch alpha API, and the project owner confirmed there are no external consumers or installed legacy objects, so the roadmap explicitly selects an in-place cleanup rather than preserving the old enum/spec/status/selector contract.
• The NodeLocal UID annotation is routing metadata, not authentication; the "proof-of-injection" wording is too strong. NodeLocal is explicitly scoped to one trusted tenant/node domain. Spoofing requires same-namespace Pod creation and can only trigger the deterministic server defined by the operator-owned CacheBackend, not an attacker-selected image or command. Strong provenance for mutually untrusted Pod creators is separate future hardening, and making the cache webhook fail-closed would violate the intentional serving-availability contract.

Therefore I am not making behavioral changes for these blocking findings in this PR.

[EdHasNoLife]

Two operational risks worth resolving before merge

The core MP mechanism (injection safety, endpoint posture, UID isolation, ownership gate) is genuinely well done and builds/tests clean. Two failure-mode risks stand out — both share one shape: fail-closed and correct on the happy path, but silent on the sad path. Neither needs an architectural change.

🔴 Risk 1 — NodeLocal /dev/shm pool is never reclaimed (node-RAM leak)

NodeLocal servers + engines mount a hostPath /dev/shm/inference-cache/<uid> (tmpfs = node RAM, DirectoryOrCreate). The controller deletes the server Pod on idle-expiry / node-demand loss / backend delete (cleanupLMCacheNodeLocalServerPods and the idle path just Client.Delete), but Kubernetes never GCs hostPath contents, and there is no preStop, lifecycle, finalizer, or reaper anywhere in the PR (the DaemonSet model that could have owned node cleanup was rejected in D5).

This is confirmed by the roadmap's own SJC validation note:

"the final run also removed only its two exact UID host directories after confirming that deleted engine/server Pods had left CUDA, torch, and semaphore files behind."

i.e. cleanup was done manually. Because the shm dir/name are "distinct after CacheBackend delete/recreate," every delete→recreate (config change, GitOps redeploy) mints a new UID and orphans the old GiB-scale pool forever on every node that ran it — monotonic node-RAM erosion until reboot, not charged to any surviving pod so it won't surface as a pod OOM.

emptyDir: memory isn't an option (cross-Pod CUDA IPC needs the shared host inode), so the fix is a reclaimer:

• preStop best-effort unlink on the server container — covers the graceful 90% (idle-expiry, delete, rollout).
• Optional reaper DaemonSet GCing /dev/shm/inference-cache/<uid> dirs whose UID no longer maps to a live CacheBackend — covers node-crash/SIGKILL; a reaper does not re-introduce the placement inversion D5 rejected.
• At minimum, ship the operator runbook you already followed by hand.

Related: the adversarial isolation item (roadmap "Validate the tenant boundary with unauthorized open/unlink tests") is still unchecked — the boundary is asserted but not yet tested against a hostile co-tenant.

🟡 Risk 2 — Ownership gate blocks engine startup forever, with no Kubernetes-native signal

The gate is a while True: sleep 2 init container (lmcache_mp_nodelocal.go) with no activeDeadlineSeconds on the engine or gate. Fail-closed is defensible here (the /dev/shm L1 pool is the KV data plane, not a soft cache) — but a misconfigured server (bad image, crash-loop, starved resources) leaves the engine in Init:0/N indefinitely, surfacing only as gate stdout. No Event, no terminal state, no bounded backoff.

The sharp edge is the interaction with declared host-ports: if backend B reuses backend A's hostPort on a shared node, B's server stays Pending, B's engines dial A's server, the gate's instance_id check correctly rejects it (UID mismatch) — and B's engines wait forever. A one-line port misconfig becomes a silent deadlock diagnosable only by reading gate logs and correlating UIDs.

Fixes are observability + bounded failure, not redesign:

• Give the gate a configurable max-wait, then exit(1) → Init:Error/CrashLoopBackOff is a native, alertable signal, and kubelet retries (bounded-but-retrying, not silent-forever).
• Have the gate distinguish "no server yet" from "a different backend owns the port" (it already holds the UID mismatch) and log/emit that loudly.
• Surface a gate-blocked engine on status as not-covered with a reason (the new per-engine coverage / desired-ready counts are the right place), and emit an Event after N failed polls.

I'd treat Risk 1 as the one closest to blocking — it's a confirmed resource leak currently cleaned up by hand. Happy to draft the preStop/reaper + gate-deadline changes if useful.

[fredericsun]

Thanks @EdHasNoLife. These are great findings.

For risk1, I've already noticed the problem. Actually, not only the the shared memory leak, but the way the current design uses shared memory without proper tenant and network segregation is kinda risky. I did not address it in this PR because it needs a bigger scope of design and I put it as the first priority for future work here: https://github.com/cachebox-project/inference-cache/blob/lmcache-multiprocess-migration/docs/design/lmcache-multiprocess-migration-roadmap.md#:~:text=objects%20and%20their%20tmpfs%20pages%20can%20remain%20until%20explicit%20node%20cleanup%20or%20reboot
WIll fix the gap in a separate PR

For risk2, I understand your concern but this port conflict failure is not silent. The existing implementation already provides several Kubernetes-native signals:

• The NodeLocal server Pod reports the host-port conflict as Unschedulable
• CacheBackend.status reports ConnectorReady=False with reason NodeLocalHostPortConflict
• The gate logs the actual and expected instance_id when it reaches a server owned by another backend
  The engine intentionally remains in init until the correct server becomes available, preventing it from connecting to the wrong cache pool. Once the port conflict is fixed, the gate retries automatically and the engine can continue starting.