diff --git a/CHANGELOG.md b/CHANGELOG.md index b82aa49..ffe3f3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,12 +48,14 @@ ### Bug Fixes +* **v2:** register Chat Completions models whose deployment accepts Anthropic `cache_control` with `compatibility.cacheFormat: "anthropic"` so prompt caching works for Claude behind LiteLLM * restore baseline reasoning-effort variants from sparse LiteLLM metadata (including GPT-6 Astra), respect explicit effort lists and disabled levels * avoid inferring reasoning-effort variants from negative-only flags or unsupported request parameters * constrain inferred GPT-5 Pro effort variants to provider-supported levels, including aliased and dated deployments ### Features +* **v2:** add the `modelCompatibility` option to set or remove the inferred `cacheFormat` per model * route models LiteLLM reports with `mode: "responses"` through the OpenAI Responses API (per-model runtime package on OpenCode 2; a `-responses` sibling on OpenCode 1) * support the OpenCode 2 API with the OpenCode 1 server entrypoint * add `formatModelNames` option to keep raw model ids in the picker diff --git a/README.md b/README.md index 609b31e..668413f 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,7 @@ also shows how to configure a provider directly in `opencode.json`. | 🌐 **Gateway-friendly** | Supports provider `headers` (and the legacy `customHeaders` option) for proxies behind Cloudflare Access or other API gateways. | | 🧩 **Splittable catalog** | `includeModels` / `excludeModels` (glob patterns) let one LiteLLM proxy be divided into several OpenCode providers — e.g. by naming prefix — without hand-maintaining a model list. | | 🎚️ **Capability overrides** | `modelCapabilities` forces or retracts per-model capability flags (`supports_vision`, `supports_function_calling`, …) when `/v1/model/info` is unavailable or disagrees with your deployment. | +| 💾 **Prompt caching (OpenCode 2.0.28+)** | Chat models whose deployment accepts Anthropic `cache_control` markers are registered with `compatibility.cacheFormat: "anthropic"`, so OpenCode 2 places cache breakpoints as it does for a native Anthropic provider. `modelCompatibility` corrects this per model. | | ⏱️ **Bounded startup** | Health checks fail fast (3 s); discovery fetches are capped at **15 s** (configurable via `LITELLM_REQUEST_TIMEOUT_MS`) for slow remote proxies. V2 model refresh runs in the background. | | 📝 **OpenCode logging** | V1 logs go through OpenCode's log API; V2 logs use the V2 plugin host's console logging. | | 🤝 **Non-destructive merge** | Only adds models you don't already have configured. Hand-curated entries are preserved verbatim. | @@ -423,6 +424,33 @@ By default the plugin prettifies each discovered id into a display name — `ant - Like the other options above, `formatModelNames` is part of the cache identity, so toggling it triggers a fresh discovery on the next start instead of serving previously cached names. - To rename just a handful of models while keeping smart formatting for the rest, use the per-model `name` override described in [Overriding or curating individual models](#overriding-or-curating-individual-models-optional). +### Prompt caching (`modelCompatibility`) + +On OpenCode 2.0.28+ (earlier 2.x releases ignore the setting), the OpenAI-compatible Chat Completions runtime only places Anthropic-style `cache_control` breakpoints when a model declares `compatibility.cacheFormat: "anthropic"`. The plugin sets that for every discovered Chat Completions model whose `/v1/model/info` entry lists `cache_control` under `supported_openai_params` — which LiteLLM reports for Anthropic and Vertex AI Anthropic deployments — so prompt caching keeps working for Claude behind the proxy. Only the Chat Completions runtimes read this setting: the Anthropic Messages runtime adds its own cache breakpoints, and the Responses runtimes do not use it, so models on those packages are left alone. + +To correct the inference per model id, set `cacheFormat` to `"anthropic"`, `"alibaba"`, or `null` to remove it: + +```jsonc +{ + "providers": { + "litellm": { + "settings": { + "baseURL": "http://localhost:4000/v1", + "modelCompatibility": { + "bedrock/claude-sonnet-4": { "cacheFormat": "anthropic" }, + "gateway/claude-opus": { "cacheFormat": null } + } + } + } + } +} +``` + +- Keys are exact model ids as they appear in `/v1/models` (not globs). Only `cacheFormat`, the one key the plugin sets, is accepted. +- OpenCode's own `providers..models..compatibility` is merged over the plugin's value, so it can add or change a key but not remove one the plugin inferred; that is what `null` here is for. +- Overrides are applied when models are registered, not during discovery, so changing them does not start a new discovery. +- Neither applies on OpenCode 1. + ## 🔧 How it works ```mermaid @@ -601,6 +629,7 @@ src/ │ ├── model-cache.ts # stale-while-revalidate on-disk model cache │ ├── model-filter.ts # includeModels/excludeModels glob filtering │ ├── model-capabilities.ts # per-model capability flag overrides +│ ├── model-compatibility.ts # cacheFormat inference and per-model overrides │ └── opencode-auth.ts # fallback to OpenCode's /connect-stored credentials └── plugin/ ├── index.ts # OpenCode 1 server entry and shared discovery logic diff --git a/src/plugin/index.ts b/src/plugin/index.ts index 8b4dc3f..4d0f62e 100644 --- a/src/plugin/index.ts +++ b/src/plugin/index.ts @@ -14,6 +14,7 @@ import { import { requiresResponsesAPI } from '../utils/transport' import type { LiteLLMModel, LiteLLMModelInfo } from '../types' import { getOpenCodeStoredApiKey } from '../utils/opencode-auth' +import type { ModelHints } from '../utils/model-cache' import { buildCacheKey, readModelCacheEntry, @@ -331,11 +332,13 @@ export function toConfigModel( * `models` is the `{ id -> entry }` map written into an OpenCode provider * config (or translated into `Model.Info` on OpenCode 2). `responsesIds` * lists the subset that must be invoked through the OpenAI Responses API; - * callers route those accordingly without inspecting each entry. + * callers route those accordingly without inspecting each entry. `hints` + * carries per-model upstream facts OpenCode 2 maps to `compatibility`. */ export interface DiscoveredModels { models: Record responsesIds: string[] + hints: Record } /** @@ -343,7 +346,7 @@ export interface DiscoveredModels { * * Pure with respect to plugin config: it performs the network calls, * classifies + formats each model, and returns the built entries plus the - * ids that require the Responses API. The provider's + * ids that require the Responses API and per-model hints. The provider's * `includeModels`/`excludeModels` filters, `modelCapabilities` overrides and * `formatModelNames` choice are applied here (not at merge time) so every * path that persists or serves a cache — cold discovery and background @@ -411,6 +414,7 @@ export async function discoverModels( const built: Record = {} const responsesIds: string[] = [] + const hints: Record = {} let skipped = 0 let wildcards = 0 let filtered = 0 @@ -440,6 +444,10 @@ export async function discoverModels( } built[model.id] = entry if (requiresResponsesAPI(enriched)) responsesIds.push(model.id) + const params = info?.supported_openai_params + hints[model.id] = { + acceptsCacheControl: Array.isArray(params) && params.includes('cache_control'), + } } if (unmatched.length > 0) { @@ -471,7 +479,7 @@ export async function discoverModels( ')', ) - return { models: built, responsesIds } + return { models: built, responsesIds, hints } } /** @@ -655,7 +663,7 @@ async function backgroundRefresh(cacheKey: string): Promise { DISCOVERY_TIMEOUT_MS, ) if (discovered && Object.keys(discovered.models).length > 0) { - writeModelCache(cacheKey, discovered.models, discovered.responsesIds) + writeModelCache(cacheKey, discovered.models, discovered.responsesIds, discovered.hints) log( 'info', `[opencode-litellm] Background-refreshed model cache for ${ctx.baseURL} (${Object.keys(discovered.models).length} models)`, @@ -913,7 +921,7 @@ export const LiteLLMPlugin: Plugin = async (input: PluginInput) => { customHeaders, ) injectedModelIds.set(cacheKey, new Set(added)) - writeModelCache(cacheKey, discovered.models, discovered.responsesIds) + writeModelCache(cacheKey, discovered.models, discovered.responsesIds, discovered.hints) } } }, diff --git a/src/plugin/v2.ts b/src/plugin/v2.ts index ad0b9ee..1223491 100644 --- a/src/plugin/v2.ts +++ b/src/plugin/v2.ts @@ -21,10 +21,24 @@ import { import { normalizeBaseURL, autoDetectLiteLLM } from '../utils/litellm-api' import { parseModelCapabilities } from '../utils/model-capabilities' import type { ModelCapabilities } from '../utils/model-capabilities' +import { + inferCompatibility, + parseModelCompatibility, + resolveCompatibility, +} from '../utils/model-compatibility' +import type { ModelCompatibility } from '../utils/model-compatibility' import type { ModelFilters } from '../utils/model-filter' +import type { ModelHints } from '../utils/model-cache' const CHAT_PROVIDER_ID = 'litellm' const OPENAI_COMPATIBLE_PACKAGE = '@opencode/ai/providers/openai-compatible' +// Of the packages a LiteLLM provider uses, only the Chat Completions ones read +// `compatibility.cacheFormat`; `openai` (Responses by default), the Responses +// packages and the Anthropic packages do not. +const CHAT_COMPLETIONS_PACKAGES = new Set([ + OPENAI_COMPATIBLE_PACKAGE, + '@opencode/ai/providers/openai/chat', +]) // Runtime package that talks to `/v1/responses`. OpenCode 2 lets one model // override the provider's package, so Responses models stay under their // original provider id and no sibling provider is needed here. @@ -78,6 +92,7 @@ interface ProviderSource { pendingPublication: boolean filters: ModelFilters capabilities: ModelCapabilities + compatibility: ModelCompatibility formatModelNames: boolean cacheKey: string models: Model.Info[] @@ -111,12 +126,21 @@ function numberOrZero(value: unknown): number { return typeof value === 'number' && Number.isFinite(value) ? value : 0 } +interface ToProviderModelsOptions { + /** The provider's runtime package; models not overridden inherit it. */ + providerPackage: string + responsesIds: ReadonlySet + hints: Record + compatibility: ModelCompatibility +} + function toProviderModels( providerID: string, entries: Record, - responsesIds: ReadonlySet = new Set(), + options: ToProviderModelsOptions, ): Model.Info[] { const provider = Provider.ID.make(providerID) + const { responsesIds, hints, compatibility: overrides } = options return Object.entries(entries).flatMap(([modelID, value]) => { const entry = asRecord(value) @@ -152,6 +176,11 @@ function toProviderModels( id: id as Model.VariantID, settings: asRecord(settings), })) as Model.Info['variants'] + const pkg = responsesIds.has(modelID) ? RESPONSES_PACKAGE : options.providerPackage + const compatibility = resolveCompatibility( + inferCompatibility(hints[modelID], CHAT_COMPLETIONS_PACKAGES.has(pkg)), + Object.hasOwn(overrides, modelID) ? overrides[modelID] : undefined, + ) return [ { ...defaults, @@ -159,6 +188,11 @@ function toProviderModels( // runtime package for itself, so it keeps the provider id while // still reaching the Responses API. ...(responsesIds.has(modelID) ? { package: RESPONSES_PACKAGE } : {}), + // `cacheFormat` exists on the host's `Model.Compatibility` (2.0.28+) + // but not in the 2.0.15 schema this package compiles against. + ...(compatibility + ? { compatibility: compatibility as unknown as Model.Info['compatibility'] } + : {}), name: readString(entry.name) ?? modelID, limit: { context: numberOrDefault(limit.context, defaults.limit.context), @@ -228,6 +262,7 @@ function runtimeSettings(source: ProviderSource): Record { 'includeModels', 'excludeModels', 'modelCapabilities', + 'modelCompatibility', 'formatModelNames', 'providerID', ]) { @@ -277,6 +312,7 @@ async function makeProviderSource( const apiKey = configuredApiKey ?? connectedApiKey const filters = readModelFilters(settings) const capabilities = parseModelCapabilities(settings.modelCapabilities) + const compatibility = parseModelCompatibility(settings.modelCompatibility) const formatModelNames = readFormatModelNames(settings) const detectedBaseURL = configuredBaseURL ? normalizeBaseURL(configuredBaseURL) @@ -293,6 +329,7 @@ async function makeProviderSource( const cacheEntry = readModelCacheEntry(cacheKey) let entries = cacheEntry?.models ?? null let responsesIds = cacheEntry?.responsesIds ?? [] + let hints = cacheEntry?.hints ?? {} if (!entries || Object.keys(entries).length === 0) { const discovered = await withTimeout( @@ -310,10 +347,12 @@ async function makeProviderSource( if (discovered && Object.keys(discovered.models).length > 0) { entries = discovered.models responsesIds = discovered.responsesIds - writeModelCache(cacheKey, entries, responsesIds) + hints = discovered.hints + writeModelCache(cacheKey, entries, responsesIds, hints) } else { entries = null responsesIds = [] + hints = {} } } else { logInfo( @@ -321,11 +360,12 @@ async function makeProviderSource( ) } + const providerPackage = provider?.package || OPENAI_COMPATIBLE_PACKAGE return { id, name: provider?.name || 'LiteLLM (proxy)', activation: provider?.activation ?? 'enabled', - package: provider?.package || OPENAI_COMPATIBLE_PACKAGE, + package: providerPackage, settings, headers: customHeaders, integrationID: provider?.integrationID, @@ -339,9 +379,15 @@ async function makeProviderSource( customHeaders: Object.keys(customHeaders).length > 0 ? customHeaders : undefined, filters, capabilities, + compatibility, formatModelNames, cacheKey, - models: toProviderModels(id, entries ?? {}, new Set(responsesIds)), + models: toProviderModels(id, entries ?? {}, { + providerPackage, + responsesIds: new Set(responsesIds), + hints, + compatibility, + }), ownedModelIds: new Set(), } } @@ -395,13 +441,14 @@ async function refreshProviderSource( return } - const models = toProviderModels( - source.id, - discovered.models, - new Set(discovered.responsesIds), - ) + const models = toProviderModels(source.id, discovered.models, { + providerPackage: source.package, + responsesIds: new Set(discovered.responsesIds), + hints: discovered.hints, + compatibility: source.compatibility, + }) if (JSON.stringify(models) === JSON.stringify(source.models)) { - writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds) + writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds, discovered.hints) source.refreshRequired = false return } @@ -415,7 +462,7 @@ async function refreshProviderSource( throw error } - writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds) + writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds, discovered.hints) source.refreshRequired = false logInfo( `[opencode-litellm] Refreshed ${models.length} models for provider "${source.id}"; the provider registry was reloaded.`, diff --git a/src/utils/model-cache.ts b/src/utils/model-cache.ts index 2e4e446..8d80441 100644 --- a/src/utils/model-cache.ts +++ b/src/utils/model-cache.ts @@ -3,12 +3,25 @@ import { readFileSync, mkdirSync, writeFileSync, renameSync, rmSync } from 'node import { homedir, tmpdir } from 'node:os' import { join } from 'node:path' +/** + * Facts about a discovered model's upstream that OpenCode 2 turns into + * `Model.Info.compatibility`. They ride alongside the config-shaped model + * entries (which have no field for them) through discovery and the cache. + */ +export interface ModelHints { + /** + * The deployment accepts Anthropic-style `cache_control` markers on + * Chat Completions (`supported_openai_params` lists `cache_control`). + */ + acceptsCacheControl: boolean +} + /** * Bump when the shape of a cached model entry changes so stale caches * from older plugin versions are ignored rather than deserialized into * an incompatible structure. */ -const CACHE_VERSION = 3 +const CACHE_VERSION = 4 /** * Maximum age of a cache entry before it's treated as a miss. Without a @@ -29,6 +42,11 @@ interface ModelCacheFile { * no transport marker (their shape is the user-facing OpenCode config). */ responsesIds?: string[] + /** + * Per-model facts about the upstream that OpenCode 2 needs but that have + * no place in a provider-config entry (see `ModelHints`). + */ + hints?: Record } /** @@ -134,6 +152,18 @@ function normalizeResponsesIds(value: unknown): string[] { return value.filter((id): id is string => typeof id === 'string') } +/** Keep only well-formed entries of a cached `hints` map. */ +function normalizeHints(value: unknown): Record { + if (!value || typeof value !== 'object' || Array.isArray(value)) return {} + const out: Record = {} + for (const [id, hint] of Object.entries(value as Record)) { + if (!hint || typeof hint !== 'object' || Array.isArray(hint)) continue + const { acceptsCacheControl } = hint as Record + out[id] = { acceptsCacheControl: acceptsCacheControl === true } + } + return out +} + /** * Read cached model entries for a cache key. Returns `null` on any * problem (missing file, parse error, version mismatch, or an entry @@ -155,14 +185,18 @@ export function readModelCache( } /** - * Read the models and Responses-API model ids for a cache key in one - * parse. Returns `null` under the same conditions as - * {@link readModelCache} (missing/expired cache), so the two can't + * Read the models, Responses-API model ids and model hints for a cache key + * in one parse. Returns `null` under the same conditions as + * {@link readModelCache} (missing/expired cache), so the three can't * disagree within a single lookup. */ export function readModelCacheEntry( cacheKey: string, -): { models: Record; responsesIds: string[] } | null { +): { + models: Record + responsesIds: string[] + hints: Record +} | null { const parsed = readCacheFile(cacheKey) if (!parsed) return null if ( @@ -171,7 +205,11 @@ export function readModelCacheEntry( ) { return null } - return { models: parsed.models, responsesIds: normalizeResponsesIds(parsed.responsesIds) } + return { + models: parsed.models, + responsesIds: normalizeResponsesIds(parsed.responsesIds), + hints: normalizeHints(parsed.hints), + } } /** @@ -204,6 +242,7 @@ export function writeModelCache( cacheKey: string, models: Record, responsesIds: readonly string[] = [], + hints: Record = {}, ): void { try { mkdirSync(cacheDir(), { recursive: true }) @@ -212,6 +251,7 @@ export function writeModelCache( savedAt: Date.now(), models, responsesIds: [...responsesIds], + hints, } // Write to a temp file and rename so concurrent OpenCode processes // never observe a partially-written JSON file (rename is atomic on diff --git a/src/utils/model-compatibility.ts b/src/utils/model-compatibility.ts new file mode 100644 index 0000000..15cc4f5 --- /dev/null +++ b/src/utils/model-compatibility.ts @@ -0,0 +1,65 @@ +import type { ModelHints } from './model-cache' + +/** + * Values `cacheFormat` accepts. Validated here because OpenCode validates + * plugin-registered models only when it serializes them, where one bad + * value fails the whole model list. + */ +const CACHE_FORMATS: ReadonlySet = new Set(['anthropic', 'alibaba']) + +/** `null` removes the `cacheFormat` the plugin inferred. */ +export interface CompatibilityOverride { + cacheFormat: string | null +} + +/** Per-model compatibility overrides keyed by exact model id. */ +export type ModelCompatibility = Record + +/** + * Defensively parse the `modelCompatibility` provider option. Non-object + * input, non-object per-model values and values outside the accepted set + * are dropped; `null` is kept. + */ +export function parseModelCompatibility(raw: unknown): ModelCompatibility { + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return {} + const out: ModelCompatibility = {} + for (const [modelId, entry] of Object.entries(raw as Record)) { + if (!entry || typeof entry !== 'object' || Array.isArray(entry)) continue + const { cacheFormat } = entry as Record + if (cacheFormat === null || (typeof cacheFormat === 'string' && CACHE_FORMATS.has(cacheFormat))) { + out[modelId] = { cacheFormat } + } + } + return out +} + +/** + * Compatibility OpenCode 2 needs for a discovered model, from what the + * proxy reported about its upstream. Only the Chat Completions runtimes + * read `cacheFormat`: Anthropic Messages adds its own cache breakpoints + * and the Responses runtimes do not use the setting. + */ +export function inferCompatibility( + hints: ModelHints | undefined, + chatCompletions: boolean, +): Record { + if (chatCompletions && hints?.acceptsCacheControl) return { cacheFormat: 'anthropic' } + return {} +} + +/** + * Overlay a user override on inferred compatibility. `null` deletes the + * key. Returns `undefined` when nothing remains so callers can leave + * `compatibility` unset. + */ +export function resolveCompatibility( + inferred: Record, + override?: CompatibilityOverride, +): Record | undefined { + const out: Record = { ...inferred } + if (override) { + if (override.cacheFormat === null) delete out.cacheFormat + else out.cacheFormat = override.cacheFormat + } + return Object.keys(out).length > 0 ? out : undefined +} diff --git a/test/model-cache.test.ts b/test/model-cache.test.ts index c3f0dee..ef598d0 100644 --- a/test/model-cache.test.ts +++ b/test/model-cache.test.ts @@ -75,9 +75,35 @@ describe('model cache', () => { expect(readModelCacheEntry('entry@http://x')).toEqual({ models: { chat: {}, responses: {} }, responsesIds: ['responses'], + hints: {}, }) }) + it('round-trips model hints and drops malformed ones', () => { + writeModelCache('hints@http://x', { a: {}, b: {} }, [], { + a: { acceptsCacheControl: true }, + b: { acceptsCacheControl: false }, + }) + expect(readModelCacheEntry('hints@http://x')?.hints).toEqual({ + a: { acceptsCacheControl: true }, + b: { acceptsCacheControl: false }, + }) + rewriteCacheFile('hints@http://x', (parsed) => { + parsed.hints = { a: { acceptsCacheControl: 'yes' }, b: null, c: [] } + }) + expect(readModelCacheEntry('hints@http://x')?.hints).toEqual({ a: { acceptsCacheControl: false } }) + }) + + it('treats caches that predate the model hints as a miss', () => { + writeModelCache(KEY, { m: {} }, ['m']) + rewriteCacheFile(KEY, (parsed) => { + parsed.version = 3 + delete parsed.hints + }) + expect(readModelCacheEntry(KEY)).toBeNull() + expect(readModelCache(KEY)).toBeNull() + }) + it('ignores cache files written by a different plugin version', () => { writeModelCache(KEY, { m: {} }) rewriteCacheFile(KEY, (parsed) => { diff --git a/test/model-compatibility.test.ts b/test/model-compatibility.test.ts new file mode 100644 index 0000000..294b374 --- /dev/null +++ b/test/model-compatibility.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, it } from 'vitest' +import { + inferCompatibility, + parseModelCompatibility, + resolveCompatibility, +} from '../src/utils/model-compatibility' + +describe('parseModelCompatibility', () => { + it('returns an empty map for non-object input', () => { + expect(parseModelCompatibility(undefined)).toEqual({}) + expect(parseModelCompatibility(null)).toEqual({}) + expect(parseModelCompatibility('cacheFormat')).toEqual({}) + expect(parseModelCompatibility([{ cacheFormat: 'anthropic' }])).toEqual({}) + }) + + it('keeps an accepted cacheFormat or null', () => { + expect( + parseModelCompatibility({ + m1: { cacheFormat: 'anthropic' }, + m2: { cacheFormat: 'alibaba' }, + m3: { cacheFormat: null }, + }), + ).toEqual({ + m1: { cacheFormat: 'anthropic' }, + m2: { cacheFormat: 'alibaba' }, + m3: { cacheFormat: null }, + }) + }) + + it('drops rejected values, other keys and empty entries', () => { + expect( + parseModelCompatibility({ + m1: { cacheFormat: 'Anthropic' }, + m2: { cacheFormat: true }, + m3: { supportsThinkingBlockBinding: false }, + m4: 'anthropic', + m5: {}, + }), + ).toEqual({}) + }) +}) + +describe('inferCompatibility', () => { + it('asks Chat Completions for Anthropic cache markers when the proxy accepts them', () => { + expect(inferCompatibility({ acceptsCacheControl: true }, true)).toEqual({ cacheFormat: 'anthropic' }) + }) + + it('infers nothing without the hint or outside Chat Completions', () => { + expect(inferCompatibility({ acceptsCacheControl: false }, true)).toEqual({}) + expect(inferCompatibility(undefined, true)).toEqual({}) + expect(inferCompatibility({ acceptsCacheControl: true }, false)).toEqual({}) + }) +}) + +describe('resolveCompatibility', () => { + it('lets overrides add, replace and remove keys', () => { + expect(resolveCompatibility({ cacheFormat: 'anthropic' }, { cacheFormat: 'alibaba' })).toEqual({ + cacheFormat: 'alibaba', + }) + expect(resolveCompatibility({}, { cacheFormat: 'anthropic' })).toEqual({ cacheFormat: 'anthropic' }) + }) + + it('returns undefined when nothing remains', () => { + expect(resolveCompatibility({}, undefined)).toBeUndefined() + expect(resolveCompatibility({ cacheFormat: 'anthropic' }, { cacheFormat: null })).toBeUndefined() + }) +}) diff --git a/test/plugin-v1.test.ts b/test/plugin-v1.test.ts index d7ae16f..e569741 100644 --- a/test/plugin-v1.test.ts +++ b/test/plugin-v1.test.ts @@ -3,7 +3,12 @@ import { tmpdir } from 'node:os' import { join } from 'node:path' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { LiteLLMPlugin } from '../src/plugin' -import { buildCacheKey, writeModelCache } from '../src/utils/model-cache' +import { + buildCacheKey, + readModelCacheEntry, + readModelCacheSavedAt, + writeModelCache, +} from '../src/utils/model-cache' import { __resetOpenCodeAuthCacheForTests } from '../src/utils/opencode-auth' interface ProviderConfig { @@ -59,7 +64,7 @@ describe('OpenCode 1 config hook', () => { process.env.HOME = homeDirectory } - function mockProxy(models: Array<{ id: string; mode?: string }>): void { + function mockProxy(models: Array<{ id: string; mode?: string; params?: string[] }>): void { globalThis.fetch = vi.fn(async (input) => { const url = String(input) if (url.endsWith('/v1/model/info')) { @@ -67,7 +72,11 @@ describe('OpenCode 1 config hook', () => { JSON.stringify({ data: models.map((model) => ({ model_name: model.id, - model_info: { key: model.id, mode: model.mode ?? 'chat' }, + model_info: { + key: model.id, + mode: model.mode ?? 'chat', + supported_openai_params: model.params, + }, })), }), { status: 200 }, @@ -122,6 +131,52 @@ describe('OpenCode 1 config hook', () => { expect(Object.keys(sibling.models ?? {})).toEqual(['responses-model']) }) + it('persists model hints so an OpenCode 2 start can reuse the shared cache', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45557/v1' + mockProxy([ + { id: 'claude', params: ['max_tokens', 'cache_control'] }, + { id: 'gpt', params: ['max_tokens'] }, + ]) + + await runConfigHook(baseURL) + + const cacheKey = buildCacheKey('litellm', 'http://127.0.0.1:45557', {}, {}) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + claude: { acceptsCacheControl: true }, + gpt: { acceptsCacheControl: false }, + }) + }) + + it('keeps model hints in the shared cache across a background refresh', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45558/v1' + mockProxy([{ id: 'claude', params: ['cache_control'] }]) + const input = { client: { app: { log: vi.fn(async () => {}) } } } + const plugin = await LiteLLMPlugin(input as never) + const config: OpenCodeConfig = { + provider: { + litellm: { + npm: '@ai-sdk/openai-compatible', + name: 'LiteLLM (proxy)', + options: { baseURL, apiKey: 'example-key' }, + models: {}, + }, + }, + } + // Cold start, backdated so the next session is allowed to refresh. + const started = Date.now() + const dateNow = vi.spyOn(Date, 'now').mockReturnValue(started - 10 * 60 * 1000) + await plugin.config!(config as never) + dateNow.mockRestore() + const cacheKey = buildCacheKey('litellm', 'http://127.0.0.1:45558', {}, {}) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ claude: { acceptsCacheControl: true } }) + + await plugin.event!({ event: { type: 'session.created' } } as never) + await vi.waitFor(() => expect(readModelCacheSavedAt(cacheKey)).toBeGreaterThanOrEqual(started)) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ claude: { acceptsCacheControl: true } }) + }) + it('creates no sibling provider when no model needs the Responses API', async () => { useIsolatedState() const baseURL = 'http://127.0.0.1:45552/v1' diff --git a/test/plugin-v2.test.ts b/test/plugin-v2.test.ts index 1e49dc1..2199eeb 100644 --- a/test/plugin-v2.test.ts +++ b/test/plugin-v2.test.ts @@ -4,7 +4,12 @@ import { join } from 'node:path' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import plugin from '../src' import type { Context } from '@opencode/plugin/promise/plugin' -import { buildCacheKey, writeModelCache } from '../src/utils/model-cache' +import { + buildCacheKey, + readModelCacheEntry, + readModelCacheSavedAt, + writeModelCache, +} from '../src/utils/model-cache' import { __resetOpenCodeAuthCacheForTests } from '../src/utils/opencode-auth' function restoreEnv(name: string, value: string | undefined): void { @@ -96,6 +101,7 @@ describe('OpenCode 2 plugin entrypoint', () => { supports_vision: true, supports_reasoning: true, supports_reasoning_efforts: ['low', 'high'], + supported_openai_params: ['max_tokens', 'cache_control', 'reasoning_effort'], input_cost_per_token: 0.000003, output_cost_per_token: 0.000015, cache_read_input_token_cost: 0.000001, @@ -137,7 +143,7 @@ describe('OpenCode 2 plugin entrypoint', () => { supports_minimal_reasoning_effort: false, supports_xhigh_reasoning_effort: true, reasoning_effort_levels: null, - supported_openai_params: ['reasoning'], + supported_openai_params: ['reasoning', 'cache_control'], }, }, ], @@ -228,6 +234,8 @@ describe('OpenCode 2 plugin entrypoint', () => { { id: 'low', settings: { reasoningEffort: 'low' } }, { id: 'high', settings: { reasoningEffort: 'high' } }, ], + // Chat Completions only emits Anthropic cache markers when told to. + compatibility: { cacheFormat: 'anthropic' }, }) expect(registered[0].models[1]).toMatchObject({ @@ -261,11 +269,235 @@ describe('OpenCode 2 plugin entrypoint', () => { expect(registered[0].models[0].package).toBeUndefined() expect(registered[0].models[1].package).toBeUndefined() expect(registered[0].models[2].package).toBeUndefined() + // No `cache_control` reported, or not a Chat Completions model: nothing inferred. + expect(registered[0].models[1].compatibility).toBeUndefined() + expect(registered[0].models[2].compatibility).toBeUndefined() + expect(registered[0].models[3].compatibility).toBeUndefined() + const cacheKey = buildCacheKey('litellm', 'http://127.0.0.1:44444', {}, {}) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + 'anthropic/claude-3-5-sonnet': { acceptsCacheControl: true }, + 'gpt-6-astra': { acceptsCacheControl: false }, + 'gpt-5-search-api': { acceptsCacheControl: false }, + 'company-pro': { acceptsCacheControl: true }, + }) await cleanup?.() expect(disposeTransform).toHaveBeenCalledOnce() }) + it('applies modelCompatibility overrides on top of inferred compatibility', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'claude-direct', + model_info: { + key: 'claude-direct', + mode: 'chat', + supported_openai_params: ['cache_control'], + }, + }, + { + model_name: 'claude-bedrock', + model_info: { + key: 'claude-bedrock', + mode: 'chat', + supported_openai_params: ['max_tokens'], + }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response( + JSON.stringify({ data: [ + { id: 'claude-direct', object: 'model' }, + { id: 'claude-bedrock', object: 'model' }, + ] }), + { status: 200 }, + ) + }) + + const registered: Array<{ info: Record; models: Array> }> = [] + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { info: Record; models: Array> }) => { + registered.push(entry) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: { + baseURL: 'http://127.0.0.1:44444/v1', + modelCompatibility: { + // `null` removes the inferred key. + 'claude-direct': { cacheFormat: null }, + // Set where the proxy does not advertise `cache_control`. + 'claude-bedrock': { cacheFormat: 'anthropic' }, + // An id the proxy does not list adds no model. + unknown: { cacheFormat: 'anthropic' }, + }, + }, + provider: { + list: vi.fn(async () => ({ data: [] })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor as never) + return { dispose: vi.fn(async () => {}) } + }), + reload: vi.fn(async () => {}), + }, + event: { subscribe: () => (async function* () {})() }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + + expect(registered).toHaveLength(1) + expect(registered[0].models.map((model) => [model.id, model.compatibility])).toEqual([ + ['claude-direct', undefined], + ['claude-bedrock', { cacheFormat: 'anthropic' }], + ]) + // runtimeSettings strips plugin-only options. + expect(registered[0].info.settings).not.toHaveProperty('modelCompatibility') + await cleanup?.() + }) + + it('applies compatibility from cached hints without touching the network', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + writeModelCache( + buildCacheKey('litellm', 'http://127.0.0.1:44444', {}, {}), + { claude: { name: 'Claude' }, qwen: { name: 'Qwen' }, constructor: { name: 'Constructor' } }, + [], + { + claude: { acceptsCacheControl: true }, + qwen: { acceptsCacheControl: true }, + constructor: { acceptsCacheControl: true }, + }, + ) + globalThis.fetch = vi.fn(async () => { + throw new Error('the cached path must not hit the network') + }) + + const registered: Array<{ info: Record; models: Array> }> = [] + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { info: Record; models: Array> }) => { + registered.push(entry) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: { + baseURL: 'http://127.0.0.1:44444/v1', + modelCompatibility: { qwen: { cacheFormat: 'alibaba' } }, + }, + provider: { + list: vi.fn(async () => ({ data: [] })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor as never) + return { dispose: vi.fn(async () => {}) } + }), + reload: vi.fn(async () => {}), + }, + event: { subscribe: () => (async function* () {})() }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + + expect(globalThis.fetch).not.toHaveBeenCalled() + expect(registered[0].models.map((model) => [model.id, model.compatibility])).toEqual([ + ['claude', { cacheFormat: 'anthropic' }], + ['qwen', { cacheFormat: 'alibaba' }], + // An id that is also an `Object.prototype` member has no override. + ['constructor', { cacheFormat: 'anthropic' }], + ]) + await cleanup?.() + }) + + it('infers cacheFormat only for Chat Completions runtime packages', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'claude', + model_info: { key: 'claude', mode: 'chat', supported_openai_params: ['cache_control'] }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response(JSON.stringify({ data: [{ id: 'claude', object: 'model' }] }), { status: 200 }) + }) + + const registered: Array<{ info: Record; models: Array> }> = [] + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { info: Record; models: Array> }) => { + registered.push(entry) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: {}, + provider: { + list: vi.fn(async () => ({ + data: [ + // `openai` defaults to the Responses API, which does not read `cacheFormat`. + ...[ + ['litellm-openai', '@opencode/ai/providers/openai'], + ['litellm-anthropic', '@opencode/ai/providers/anthropic-compatible'], + ['litellm-chat', '@opencode/ai/providers/openai/chat'], + ].map(([id, pkg]) => ({ + id, + activation: 'enabled', + package: pkg, + settings: { baseURL: 'http://127.0.0.1:44444/v1' }, + })), + ], + })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor as never) + return { dispose: vi.fn(async () => {}) } + }), + reload: vi.fn(async () => {}), + }, + event: { subscribe: () => (async function* () {})() }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + + expect(registered.map((entry) => [entry.info.id, entry.models[0].compatibility])).toEqual([ + ['litellm-openai', undefined], + ['litellm-anthropic', undefined], + ['litellm-chat', { cacheFormat: 'anthropic' }], + ]) + await cleanup?.() + }) + it('uses service environment variables to create the default provider', async () => { cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-env-test-')) process.env.XDG_CACHE_HOME = cacheDirectory @@ -348,7 +580,20 @@ describe('OpenCode 2 plugin entrypoint', () => { globalThis.fetch = vi.fn(async (input) => { const url = String(input) if (url.endsWith('/v1/model/info')) { - return new Response(JSON.stringify({ data: [] }), { status: 200 }) + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'anthropic/claude-3-5-sonnet', + model_info: { + key: 'anthropic/claude-3-5-sonnet', + supported_openai_params: ['cache_control'], + }, + }, + ], + }), + { status: 200 }, + ) } return new Response( JSON.stringify({ data: [{ id: 'anthropic/claude-3-5-sonnet', object: 'model' }] }), @@ -361,7 +606,11 @@ describe('OpenCode 2 plugin entrypoint', () => { name: 'Configured LiteLLM', activation: 'enabled', package: '@opencode/ai/providers/openai-compatible', - settings: { baseURL: `${baseURL}/v1`, apiKey: 'example-api-key' }, + settings: { + baseURL: `${baseURL}/v1`, + apiKey: 'example-api-key', + modelCompatibility: { 'anthropic/claude-3-5-sonnet': { cacheFormat: 'alibaba' } }, + }, headers: { 'X-Gateway': 'test' }, } const curatedModel = { id: 'curated-model', name: 'Curated model' } @@ -406,6 +655,8 @@ describe('OpenCode 2 plugin entrypoint', () => { expect(configuredProvider.headers).toMatchObject({ 'X-Gateway': 'test' }) expect(updatedModels.map((model) => model.id)).toContain('curated-model') expect(updatedModels.map((model) => model.id)).toContain('anthropic/claude-3-5-sonnet') + const discovered = updatedModels.find((model) => model.id === 'anthropic/claude-3-5-sonnet') + expect(discovered?.compatibility).toEqual({ cacheFormat: 'alibaba' }) await cleanup?.() }) @@ -423,10 +674,37 @@ describe('OpenCode 2 plugin entrypoint', () => { globalThis.fetch = vi.fn(async (input) => { const url = String(input) if (url.endsWith('/v1/model/info')) { - return new Response(JSON.stringify({ data: [] }), { status: 200 }) + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'anthropic/claude-3-5-sonnet', + model_info: { + key: 'anthropic/claude-3-5-sonnet', + mode: 'chat', + supported_openai_params: ['cache_control'], + }, + }, + { + model_name: 'vertex/claude-sonnet', + model_info: { + key: 'vertex/claude-sonnet', + mode: 'chat', + supported_openai_params: ['cache_control'], + }, + }, + ], + }), + { status: 200 }, + ) } return new Response( - JSON.stringify({ data: [{ id: 'anthropic/claude-3-5-sonnet', object: 'model' }] }), + JSON.stringify({ + data: [ + { id: 'anthropic/claude-3-5-sonnet', object: 'model' }, + { id: 'vertex/claude-sonnet', object: 'model' }, + ], + }), { status: 200 }, ) }) @@ -452,7 +730,10 @@ describe('OpenCode 2 plugin entrypoint', () => { }) const context = { app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, - options: { baseURL: `${baseURL}/v1` }, + options: { + baseURL: `${baseURL}/v1`, + modelCompatibility: { 'anthropic/claude-3-5-sonnet': { cacheFormat: 'alibaba' } }, + }, provider: { list: vi.fn(async () => ({ data: [] })), transform: vi.fn(async (transform: (editor: unknown) => void) => { @@ -474,12 +755,166 @@ describe('OpenCode 2 plugin entrypoint', () => { await vi.waitFor(() => expect(reload).toHaveBeenCalledOnce()) expect(registeredModels).toHaveLength(2) expect(registeredModels[0].map((model) => model.id)).toEqual(['cached-model']) - expect(registeredModels[1].map((model) => model.id)).toEqual([ - 'anthropic/claude-3-5-sonnet', + // Inference and the override are both applied on refresh, and the refreshed hints are persisted. + expect(registeredModels[1].map((model) => [model.id, model.compatibility])).toEqual([ + ['anthropic/claude-3-5-sonnet', { cacheFormat: 'alibaba' }], + ['vertex/claude-sonnet', { cacheFormat: 'anthropic' }], ]) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + 'anthropic/claude-3-5-sonnet': { acceptsCacheControl: true }, + 'vertex/claude-sonnet': { acceptsCacheControl: true }, + }) + await cleanup?.() + }) + + it('infers no cacheFormat for an Anthropic package, including after a refresh', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-refresh-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + const baseURL = 'http://127.0.0.1:44444' + const cacheKey = buildCacheKey('litellm-claude', baseURL, {}, {}) + const now = Date.now() + const dateNow = vi.spyOn(Date, 'now').mockReturnValue(now - 10 * 60 * 1000) + writeModelCache(cacheKey, { 'cached-model': { name: 'Cached Model' } }, [], { + 'cached-model': { acceptsCacheControl: true }, + }) + dateNow.mockRestore() + + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'claude', + model_info: { key: 'claude', mode: 'chat', supported_openai_params: ['cache_control'] }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response(JSON.stringify({ data: [{ id: 'claude', object: 'model' }] }), { status: 200 }) + }) + + const registeredModels: Array>> = [] + let providerTransform: ((editor: unknown) => void) | undefined + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { models: Array> }) => { + registeredModels.push(entry.models) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const reload = vi.fn(async () => { + providerTransform?.(editor) + }) + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: {}, + provider: { + list: vi.fn(async () => ({ + data: [ + { + id: 'litellm-claude', + activation: 'enabled', + package: '@opencode/ai/providers/anthropic-compatible', + settings: { baseURL: `${baseURL}/v1` }, + }, + ], + })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + providerTransform = transform + transform(editor) + return { dispose: vi.fn(async () => {}) } + }), + reload, + }, + event: { + subscribe: () => + (async function* () { + yield { type: 'session.created' } + })(), + }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + await vi.waitFor(() => expect(reload).toHaveBeenCalledOnce()) + expect(registeredModels[0].map((model) => model.compatibility)).toEqual([undefined]) + expect(registeredModels[1].map((model) => model.id)).toEqual(['claude']) + expect(registeredModels[1][0].compatibility).toBeUndefined() await cleanup?.() }) + it('keeps hints in the cache when a background refresh finds no changes', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-refresh-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + const baseURL = 'http://127.0.0.1:44444' + const cacheKey = buildCacheKey('litellm', baseURL, {}, {}) + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'claude', + model_info: { key: 'claude', mode: 'chat', supported_openai_params: ['cache_control'] }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response(JSON.stringify({ data: [{ id: 'claude', object: 'model' }] }), { status: 200 }) + }) + const editor = { + list: () => [], + get: () => undefined, + add: vi.fn(), + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const reload = vi.fn(async () => {}) + const makeContext = (events: Array<{ type: string }>) => + ({ + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: { baseURL: `${baseURL}/v1` }, + provider: { + list: vi.fn(async () => ({ data: [] })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor) + return { dispose: vi.fn(async () => {}) } + }), + reload, + }, + event: { + subscribe: () => + (async function* () { + yield* events + })(), + }, + }) as unknown as Context + + // Cold start, backdated so the next session is allowed to refresh. + const started = Date.now() + const dateNow = vi.spyOn(Date, 'now').mockReturnValue(started - 10 * 60 * 1000) + const first = await plugin.setup(makeContext([])) + dateNow.mockRestore() + await first?.() + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ claude: { acceptsCacheControl: true } }) + + const second = await plugin.setup(makeContext([{ type: 'session.created' }])) + await vi.waitFor(() => expect(readModelCacheSavedAt(cacheKey)).toBeGreaterThanOrEqual(started)) + expect(reload).not.toHaveBeenCalled() + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ claude: { acceptsCacheControl: true } }) + await second?.() + }) + it('registers providers that only appear after setup (provider.updated)', async () => { cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-late-provider-test-')) process.env.XDG_CACHE_HOME = cacheDirectory