feat(v2): enable prompt caching for Claude behind LiteLLM

CHANGELOG.md

@@ -48,12 +48,14 @@
 
 ### Bug Fixes
 
+* **v2:** register Chat Completions models whose deployment accepts Anthropic `cache_control` with `compatibility.cacheFormat: "anthropic"` so prompt caching works for Claude behind LiteLLM
 * restore baseline reasoning-effort variants from sparse LiteLLM metadata (including GPT-6 Astra), respect explicit effort lists and disabled levels
 * avoid inferring reasoning-effort variants from negative-only flags or unsupported request parameters
 * constrain inferred GPT-5 Pro effort variants to provider-supported levels, including aliased and dated deployments
 
 ### Features
 
+* **v2:** add the `modelCompatibility` option to set or remove the inferred `cacheFormat` per model
 * route models LiteLLM reports with `mode: "responses"` through the OpenAI Responses API (per-model runtime package on OpenCode 2; a `<provider>-responses` sibling on OpenCode 1)
 * support the OpenCode 2 API with the OpenCode 1 server entrypoint
 * add `formatModelNames` option to keep raw model ids in the picker

README.md

@@ -82,6 +82,7 @@ also shows how to configure a provider directly in `opencode.json`.
 | 🌐 **Gateway-friendly** | Supports provider `headers` (and the legacy `customHeaders` option) for proxies behind Cloudflare Access or other API gateways. |
 | 🧩 **Splittable catalog** | `includeModels` / `excludeModels` (glob patterns) let one LiteLLM proxy be divided into several OpenCode providers — e.g. by naming prefix — without hand-maintaining a model list. |
 | 🎚️ **Capability overrides** | `modelCapabilities` forces or retracts per-model capability flags (`supports_vision`, `supports_function_calling`, …) when `/v1/model/info` is unavailable or disagrees with your deployment. |
+| 💾 **Prompt caching (OpenCode 2.0.28+)** | Chat models whose deployment accepts Anthropic `cache_control` markers are registered with `compatibility.cacheFormat: "anthropic"`, so OpenCode 2 places cache breakpoints as it does for a native Anthropic provider. `modelCompatibility` corrects this per model. |
 | ⏱️ **Bounded startup** | Health checks fail fast (3 s); discovery fetches are capped at **15 s** (configurable via `LITELLM_REQUEST_TIMEOUT_MS`) for slow remote proxies. V2 model refresh runs in the background. |
 | 📝 **OpenCode logging** | V1 logs go through OpenCode's log API; V2 logs use the V2 plugin host's console logging. |
 | 🤝 **Non-destructive merge** | Only adds models you don't already have configured. Hand-curated entries are preserved verbatim. |
@@ -423,6 +424,33 @@ By default the plugin prettifies each discovered id into a display name — `ant
 - Like the other options above, `formatModelNames` is part of the cache identity, so toggling it triggers a fresh discovery on the next start instead of serving previously cached names.
 - To rename just a handful of models while keeping smart formatting for the rest, use the per-model `name` override described in [Overriding or curating individual models](#overriding-or-curating-individual-models-optional).
 
+### Prompt caching (`modelCompatibility`)
+
+On OpenCode 2.0.28+ (earlier 2.x releases ignore the setting), the OpenAI-compatible Chat Completions runtime only places Anthropic-style `cache_control` breakpoints when a model declares `compatibility.cacheFormat: "anthropic"`. The plugin sets that for every discovered Chat Completions model whose `/v1/model/info` entry lists `cache_control` under `supported_openai_params` — which LiteLLM reports for Anthropic and Vertex AI Anthropic deployments — so prompt caching keeps working for Claude behind the proxy. Only the Chat Completions runtimes read this setting: the Anthropic Messages runtime adds its own cache breakpoints, and the Responses runtimes do not use it, so models on those packages are left alone.
+
+To correct the inference per model id, set `cacheFormat` to `"anthropic"`, `"alibaba"`, or `null` to remove it:
+
+```jsonc
+{
+  "providers": {
+    "litellm": {
+      "settings": {
+        "baseURL": "http://localhost:4000/v1",
+        "modelCompatibility": {
+          "bedrock/claude-sonnet-4": { "cacheFormat": "anthropic" },
+          "gateway/claude-opus": { "cacheFormat": null }
+        }
+      }
+    }
+  }
+}
+```
+
+- Keys are exact model ids as they appear in `/v1/models` (not globs). Only `cacheFormat`, the one key the plugin sets, is accepted.
+- OpenCode's own `providers.<id>.models.<model>.compatibility` is merged over the plugin's value, so it can add or change a key but not remove one the plugin inferred; that is what `null` here is for.
+- Overrides are applied when models are registered, not during discovery, so changing them does not start a new discovery.
+- Neither applies on OpenCode 1.
+
 ## 🔧 How it works
 
 ```mermaid
@@ -601,6 +629,7 @@ src/
 │   ├── model-cache.ts          # stale-while-revalidate on-disk model cache
 │   ├── model-filter.ts         # includeModels/excludeModels glob filtering
 │   ├── model-capabilities.ts   # per-model capability flag overrides
+│   ├── model-compatibility.ts  # cacheFormat inference and per-model overrides
 │   └── opencode-auth.ts        # fallback to OpenCode's /connect-stored credentials
 └── plugin/
     ├── index.ts                # OpenCode 1 server entry and shared discovery logic

src/plugin/index.ts

@@ -14,6 +14,7 @@ import {
 import { requiresResponsesAPI } from '../utils/transport'
 import type { LiteLLMModel, LiteLLMModelInfo } from '../types'
 import { getOpenCodeStoredApiKey } from '../utils/opencode-auth'
+import type { ModelHints } from '../utils/model-cache'
 import {
   buildCacheKey,
   readModelCacheEntry,
@@ -331,19 +332,21 @@ export function toConfigModel(
  * `models` is the `{ id -> entry }` map written into an OpenCode provider
  * config (or translated into `Model.Info` on OpenCode 2). `responsesIds`
  * lists the subset that must be invoked through the OpenAI Responses API;
- * callers route those accordingly without inspecting each entry.
+ * callers route those accordingly without inspecting each entry. `hints`
+ * carries per-model upstream facts OpenCode 2 maps to `compatibility`.
  */
 export interface DiscoveredModels {
   models: Record<string, unknown>
   responsesIds: string[]
+  hints: Record<string, ModelHints>
 }
 
 /**
  * Fetch and build OpenCode model entries from a LiteLLM proxy.
  *
  * Pure with respect to plugin config: it performs the network calls,
  * classifies + formats each model, and returns the built entries plus the
- * ids that require the Responses API. The provider's
+ * ids that require the Responses API and per-model hints. The provider's
  * `includeModels`/`excludeModels` filters, `modelCapabilities` overrides and
  * `formatModelNames` choice are applied here (not at merge time) so every
  * path that persists or serves a cache — cold discovery and background
@@ -411,6 +414,7 @@ export async function discoverModels(
 
   const built: Record<string, unknown> = {}
   const responsesIds: string[] = []
+  const hints: Record<string, ModelHints> = {}
   let skipped = 0
   let wildcards = 0
   let filtered = 0
@@ -440,6 +444,10 @@ export async function discoverModels(
     }
     built[model.id] = entry
     if (requiresResponsesAPI(enriched)) responsesIds.push(model.id)
+    const params = info?.supported_openai_params
+    hints[model.id] = {
+      acceptsCacheControl: Array.isArray(params) && params.includes('cache_control'),
+    }
   }
 
   if (unmatched.length > 0) {
@@ -471,7 +479,7 @@ export async function discoverModels(
       ')',
   )
 
-  return { models: built, responsesIds }
+  return { models: built, responsesIds, hints }
 }
 
 /**
@@ -655,7 +663,7 @@ async function backgroundRefresh(cacheKey: string): Promise<void> {
       DISCOVERY_TIMEOUT_MS,
     )
     if (discovered && Object.keys(discovered.models).length > 0) {
-      writeModelCache(cacheKey, discovered.models, discovered.responsesIds)
+      writeModelCache(cacheKey, discovered.models, discovered.responsesIds, discovered.hints)
       log(
         'info',
         `[opencode-litellm] Background-refreshed model cache for ${ctx.baseURL} (${Object.keys(discovered.models).length} models)`,
@@ -913,7 +921,7 @@ export const LiteLLMPlugin: Plugin = async (input: PluginInput) => {
             customHeaders,
           )
           injectedModelIds.set(cacheKey, new Set(added))
-          writeModelCache(cacheKey, discovered.models, discovered.responsesIds)
+          writeModelCache(cacheKey, discovered.models, discovered.responsesIds, discovered.hints)
         }
       }
     },

src/plugin/v2.ts

@@ -21,10 +21,24 @@ import {
 import { normalizeBaseURL, autoDetectLiteLLM } from '../utils/litellm-api'
 import { parseModelCapabilities } from '../utils/model-capabilities'
 import type { ModelCapabilities } from '../utils/model-capabilities'
+import {
+  inferCompatibility,
+  parseModelCompatibility,
+  resolveCompatibility,
+} from '../utils/model-compatibility'
+import type { ModelCompatibility } from '../utils/model-compatibility'
 import type { ModelFilters } from '../utils/model-filter'
+import type { ModelHints } from '../utils/model-cache'
 
 const CHAT_PROVIDER_ID = 'litellm'
 const OPENAI_COMPATIBLE_PACKAGE = '@opencode/ai/providers/openai-compatible'
+// Of the packages a LiteLLM provider uses, only the Chat Completions ones read
+// `compatibility.cacheFormat`; `openai` (Responses by default), the Responses
+// packages and the Anthropic packages do not.
+const CHAT_COMPLETIONS_PACKAGES = new Set([
+  OPENAI_COMPATIBLE_PACKAGE,
+  '@opencode/ai/providers/openai/chat',
+])
 // Runtime package that talks to `/v1/responses`. OpenCode 2 lets one model
 // override the provider's package, so Responses models stay under their
 // original provider id and no sibling provider is needed here.
@@ -78,6 +92,7 @@ interface ProviderSource {
   pendingPublication: boolean
   filters: ModelFilters
   capabilities: ModelCapabilities
+  compatibility: ModelCompatibility
   formatModelNames: boolean
   cacheKey: string
   models: Model.Info[]
@@ -111,12 +126,21 @@ function numberOrZero(value: unknown): number {
   return typeof value === 'number' && Number.isFinite(value) ? value : 0
 }
 
+interface ToProviderModelsOptions {
+  /** The provider's runtime package; models not overridden inherit it. */
+  providerPackage: string
+  responsesIds: ReadonlySet<string>
+  hints: Record<string, ModelHints>
+  compatibility: ModelCompatibility
+}
+
 function toProviderModels(
   providerID: string,
   entries: Record<string, unknown>,
-  responsesIds: ReadonlySet<string> = new Set<string>(),
+  options: ToProviderModelsOptions,
 ): Model.Info[] {
   const provider = Provider.ID.make(providerID)
+  const { responsesIds, hints, compatibility: overrides } = options
 
   return Object.entries(entries).flatMap(([modelID, value]) => {
     const entry = asRecord(value)
@@ -152,13 +176,23 @@ function toProviderModels(
       id: id as Model.VariantID,
       settings: asRecord(settings),
     })) as Model.Info['variants']
+    const pkg = responsesIds.has(modelID) ? RESPONSES_PACKAGE : options.providerPackage
+    const compatibility = resolveCompatibility(
+      inferCompatibility(hints[modelID], CHAT_COMPLETIONS_PACKAGES.has(pkg)),
+      Object.hasOwn(overrides, modelID) ? overrides[modelID] : undefined,
+    )
     return [
       {
         ...defaults,
         // A model that targets `/v1/responses` overrides the provider's
         // runtime package for itself, so it keeps the provider id while
         // still reaching the Responses API.
         ...(responsesIds.has(modelID) ? { package: RESPONSES_PACKAGE } : {}),
+        // `cacheFormat` exists on the host's `Model.Compatibility` (2.0.28+)
+        // but not in the 2.0.15 schema this package compiles against.
+        ...(compatibility
+          ? { compatibility: compatibility as unknown as Model.Info['compatibility'] }
+          : {}),
         name: readString(entry.name) ?? modelID,
         limit: {
           context: numberOrDefault(limit.context, defaults.limit.context),
@@ -228,6 +262,7 @@ function runtimeSettings(source: ProviderSource): Record<string, unknown> {
     'includeModels',
     'excludeModels',
     'modelCapabilities',
+    'modelCompatibility',
     'formatModelNames',
     'providerID',
   ]) {
@@ -277,6 +312,7 @@ async function makeProviderSource(
   const apiKey = configuredApiKey ?? connectedApiKey
   const filters = readModelFilters(settings)
   const capabilities = parseModelCapabilities(settings.modelCapabilities)
+  const compatibility = parseModelCompatibility(settings.modelCompatibility)
   const formatModelNames = readFormatModelNames(settings)
   const detectedBaseURL = configuredBaseURL
     ? normalizeBaseURL(configuredBaseURL)
@@ -293,6 +329,7 @@ async function makeProviderSource(
   const cacheEntry = readModelCacheEntry(cacheKey)
   let entries = cacheEntry?.models ?? null
   let responsesIds = cacheEntry?.responsesIds ?? []
+  let hints = cacheEntry?.hints ?? {}
 
   if (!entries || Object.keys(entries).length === 0) {
     const discovered = await withTimeout(
@@ -310,22 +347,25 @@ async function makeProviderSource(
     if (discovered && Object.keys(discovered.models).length > 0) {
       entries = discovered.models
       responsesIds = discovered.responsesIds
-      writeModelCache(cacheKey, entries, responsesIds)
+      hints = discovered.hints
+      writeModelCache(cacheKey, entries, responsesIds, hints)
     } else {
       entries = null
       responsesIds = []
+      hints = {}
     }
   } else {
     logInfo(
       `[opencode-litellm] Loaded ${Object.keys(entries).length} models from cache for provider "${id}" (${baseURL}); refresh happens in the background on new sessions.`,
     )
   }
 
+  const providerPackage = provider?.package || OPENAI_COMPATIBLE_PACKAGE
   return {
     id,
     name: provider?.name || 'LiteLLM (proxy)',
     activation: provider?.activation ?? 'enabled',
-    package: provider?.package || OPENAI_COMPATIBLE_PACKAGE,
+    package: providerPackage,
     settings,
     headers: customHeaders,
     integrationID: provider?.integrationID,
@@ -339,9 +379,15 @@ async function makeProviderSource(
     customHeaders: Object.keys(customHeaders).length > 0 ? customHeaders : undefined,
     filters,
     capabilities,
+    compatibility,
     formatModelNames,
     cacheKey,
-    models: toProviderModels(id, entries ?? {}, new Set(responsesIds)),
+    models: toProviderModels(id, entries ?? {}, {
+      providerPackage,
+      responsesIds: new Set(responsesIds),
+      hints,
+      compatibility,
+    }),
     ownedModelIds: new Set<string>(),
   }
 }
@@ -395,13 +441,14 @@ async function refreshProviderSource(
         return
       }
 
-      const models = toProviderModels(
-        source.id,
-        discovered.models,
-        new Set(discovered.responsesIds),
-      )
+      const models = toProviderModels(source.id, discovered.models, {
+        providerPackage: source.package,
+        responsesIds: new Set(discovered.responsesIds),
+        hints: discovered.hints,
+        compatibility: source.compatibility,
+      })
       if (JSON.stringify(models) === JSON.stringify(source.models)) {
-        writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds)
+        writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds, discovered.hints)
         source.refreshRequired = false
         return
       }
@@ -415,7 +462,7 @@ async function refreshProviderSource(
         throw error
       }
 
-      writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds)
+      writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds, discovered.hints)
       source.refreshRequired = false
       logInfo(
         `[opencode-litellm] Refreshed ${models.length} models for provider "${source.id}"; the provider registry was reloaded.`,