diff --git a/CHANGELOG.md b/CHANGELOG.md index b82aa49..4296565 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -48,12 +48,15 @@ ### Bug Fixes +* **v2:** register Chat Completions models whose deployment accepts Anthropic `cache_control` with `compatibility.cacheFormat: "anthropic"` so prompt caching works for Claude behind LiteLLM +* **v2:** spell reasoning-effort variants as `thinking` + `effort` for providers on an Anthropic runtime package, where `reasoningEffort` is ignored; models without adaptive thinking get none there * restore baseline reasoning-effort variants from sparse LiteLLM metadata (including GPT-6 Astra), respect explicit effort lists and disabled levels * avoid inferring reasoning-effort variants from negative-only flags or unsupported request parameters * constrain inferred GPT-5 Pro effort variants to provider-supported levels, including aliased and dated deployments ### Features +* **v2:** add the `modelCompatibility` option to set or remove the inferred `cacheFormat` per model * route models LiteLLM reports with `mode: "responses"` through the OpenAI Responses API (per-model runtime package on OpenCode 2; a `-responses` sibling on OpenCode 1) * support the OpenCode 2 API with the OpenCode 1 server entrypoint * add `formatModelNames` option to keep raw model ids in the picker diff --git a/README.md b/README.md index 609b31e..6f44b05 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,7 @@ also shows how to configure a provider directly in `opencode.json`. | 🌐 **Gateway-friendly** | Supports provider `headers` (and the legacy `customHeaders` option) for proxies behind Cloudflare Access or other API gateways. | | 🧩 **Splittable catalog** | `includeModels` / `excludeModels` (glob patterns) let one LiteLLM proxy be divided into several OpenCode providers — e.g. by naming prefix — without hand-maintaining a model list. | | 🎚️ **Capability overrides** | `modelCapabilities` forces or retracts per-model capability flags (`supports_vision`, `supports_function_calling`, …) when `/v1/model/info` is unavailable or disagrees with your deployment. | +| 💾 **Prompt caching (OpenCode 2.0.28+)** | Chat models whose deployment accepts Anthropic `cache_control` markers are registered with `compatibility.cacheFormat: "anthropic"`, so OpenCode 2 places cache breakpoints as it does for a native Anthropic provider. `modelCompatibility` corrects this per model. | | ⏱️ **Bounded startup** | Health checks fail fast (3 s); discovery fetches are capped at **15 s** (configurable via `LITELLM_REQUEST_TIMEOUT_MS`) for slow remote proxies. V2 model refresh runs in the background. | | 📝 **OpenCode logging** | V1 logs go through OpenCode's log API; V2 logs use the V2 plugin host's console logging. | | 🤝 **Non-destructive merge** | Only adds models you don't already have configured. Hand-curated entries are preserved verbatim. | @@ -244,6 +245,31 @@ between effort levels from the model picker without hand-curating every entry. > `mode: "responses"` the plugin routes it to `/v1/responses` automatically; > otherwise reach it without `reasoningEffort`. See the FAQ entry below. +On OpenCode 2, models whose effective package is `@opencode/ai/providers/anthropic` +or `@opencode/ai/providers/anthropic-compatible` get the same levels in the +shape that runtime reads, `{ thinking: { type: "adaptive", display: "summarized" }, effort }`, +for models LiteLLM reports with `supports_adaptive_thinking`, with `none` +becoming `thinking: { type: "disabled" }` (dropped when `thinking_always_on`). +Models without adaptive thinking get no variants on those packages, since +`reasoningEffort` would be ignored there. To reach Claude over the proxy's +`/v1/messages` while the rest of the catalog keeps its usual transport, split it +with [`includeModels`/`excludeModels`](#splitting-one-proxy-into-multiple-providers-includemodels--excludemodels): + +```jsonc +{ + "providers": { + "litellm": { + "package": "@opencode/ai/providers/openai-compatible", + "settings": { "baseURL": "http://localhost:4000/v1", "excludeModels": ["*claude*"] } + }, + "litellm-claude": { + "package": "@opencode/ai/providers/anthropic-compatible", + "settings": { "baseURL": "http://localhost:4000/v1", "includeModels": ["*claude*"] } + } + } +} +``` + ### Responses API models LiteLLM reports `mode` per model on `/v1/model/info` (the plugin fetches it @@ -423,6 +449,33 @@ By default the plugin prettifies each discovered id into a display name — `ant - Like the other options above, `formatModelNames` is part of the cache identity, so toggling it triggers a fresh discovery on the next start instead of serving previously cached names. - To rename just a handful of models while keeping smart formatting for the rest, use the per-model `name` override described in [Overriding or curating individual models](#overriding-or-curating-individual-models-optional). +### Prompt caching (`modelCompatibility`) + +On OpenCode 2.0.28+ (earlier 2.x releases ignore the setting), the OpenAI-compatible Chat Completions runtime only places Anthropic-style `cache_control` breakpoints when a model declares `compatibility.cacheFormat: "anthropic"`. The plugin sets that for every discovered Chat Completions model whose `/v1/model/info` entry lists `cache_control` under `supported_openai_params` — which LiteLLM reports for Anthropic and Vertex AI Anthropic deployments — so prompt caching keeps working for Claude behind the proxy. Only the Chat Completions runtimes read this setting: the Anthropic Messages runtime adds its own cache breakpoints, and the Responses runtimes do not use it, so models on those packages are left alone. + +To correct the inference per model id, set `cacheFormat` to `"anthropic"`, `"alibaba"`, or `null` to remove it: + +```jsonc +{ + "providers": { + "litellm": { + "settings": { + "baseURL": "http://localhost:4000/v1", + "modelCompatibility": { + "bedrock/claude-sonnet-4": { "cacheFormat": "anthropic" }, + "gateway/claude-opus": { "cacheFormat": null } + } + } + } + } +} +``` + +- Keys are exact model ids as they appear in `/v1/models` (not globs). Only `cacheFormat`, the one key the plugin sets, is accepted. +- OpenCode's own `providers..models..compatibility` is merged over the plugin's value, so it can add or change a key but not remove one the plugin inferred; that is what `null` here is for. +- Overrides are applied when models are registered, not during discovery, so changing them does not start a new discovery. +- Neither applies on OpenCode 1. + ## 🔧 How it works ```mermaid @@ -601,6 +654,7 @@ src/ │ ├── model-cache.ts # stale-while-revalidate on-disk model cache │ ├── model-filter.ts # includeModels/excludeModels glob filtering │ ├── model-capabilities.ts # per-model capability flag overrides +│ ├── model-compatibility.ts # cacheFormat inference and per-model overrides │ └── opencode-auth.ts # fallback to OpenCode's /connect-stored credentials └── plugin/ ├── index.ts # OpenCode 1 server entry and shared discovery logic diff --git a/src/plugin/index.ts b/src/plugin/index.ts index 8b4dc3f..197519f 100644 --- a/src/plugin/index.ts +++ b/src/plugin/index.ts @@ -14,6 +14,7 @@ import { import { requiresResponsesAPI } from '../utils/transport' import type { LiteLLMModel, LiteLLMModelInfo } from '../types' import { getOpenCodeStoredApiKey } from '../utils/opencode-auth' +import type { ModelHints } from '../utils/model-cache' import { buildCacheKey, readModelCacheEntry, @@ -331,11 +332,14 @@ export function toConfigModel( * `models` is the `{ id -> entry }` map written into an OpenCode provider * config (or translated into `Model.Info` on OpenCode 2). `responsesIds` * lists the subset that must be invoked through the OpenAI Responses API; - * callers route those accordingly without inspecting each entry. + * callers route those accordingly without inspecting each entry. `hints` + * carries per-model upstream facts OpenCode 2 maps to `compatibility` and + * `variants`. */ export interface DiscoveredModels { models: Record responsesIds: string[] + hints: Record } /** @@ -343,7 +347,7 @@ export interface DiscoveredModels { * * Pure with respect to plugin config: it performs the network calls, * classifies + formats each model, and returns the built entries plus the - * ids that require the Responses API. The provider's + * ids that require the Responses API and per-model hints. The provider's * `includeModels`/`excludeModels` filters, `modelCapabilities` overrides and * `formatModelNames` choice are applied here (not at merge time) so every * path that persists or serves a cache — cold discovery and background @@ -411,6 +415,7 @@ export async function discoverModels( const built: Record = {} const responsesIds: string[] = [] + const hints: Record = {} let skipped = 0 let wildcards = 0 let filtered = 0 @@ -440,6 +445,12 @@ export async function discoverModels( } built[model.id] = entry if (requiresResponsesAPI(enriched)) responsesIds.push(model.id) + const params = info?.supported_openai_params + hints[model.id] = { + acceptsCacheControl: Array.isArray(params) && params.includes('cache_control'), + adaptiveThinking: info?.supports_adaptive_thinking === true, + thinkingAlwaysOn: info?.thinking_always_on === true, + } } if (unmatched.length > 0) { @@ -471,7 +482,7 @@ export async function discoverModels( ')', ) - return { models: built, responsesIds } + return { models: built, responsesIds, hints } } /** @@ -655,7 +666,7 @@ async function backgroundRefresh(cacheKey: string): Promise { DISCOVERY_TIMEOUT_MS, ) if (discovered && Object.keys(discovered.models).length > 0) { - writeModelCache(cacheKey, discovered.models, discovered.responsesIds) + writeModelCache(cacheKey, discovered.models, discovered.responsesIds, discovered.hints) log( 'info', `[opencode-litellm] Background-refreshed model cache for ${ctx.baseURL} (${Object.keys(discovered.models).length} models)`, @@ -913,7 +924,7 @@ export const LiteLLMPlugin: Plugin = async (input: PluginInput) => { customHeaders, ) injectedModelIds.set(cacheKey, new Set(added)) - writeModelCache(cacheKey, discovered.models, discovered.responsesIds) + writeModelCache(cacheKey, discovered.models, discovered.responsesIds, discovered.hints) } } }, diff --git a/src/plugin/v2.ts b/src/plugin/v2.ts index ad0b9ee..e19f400 100644 --- a/src/plugin/v2.ts +++ b/src/plugin/v2.ts @@ -21,10 +21,29 @@ import { import { normalizeBaseURL, autoDetectLiteLLM } from '../utils/litellm-api' import { parseModelCapabilities } from '../utils/model-capabilities' import type { ModelCapabilities } from '../utils/model-capabilities' +import { + inferCompatibility, + parseModelCompatibility, + resolveCompatibility, +} from '../utils/model-compatibility' +import type { ModelCompatibility } from '../utils/model-compatibility' import type { ModelFilters } from '../utils/model-filter' +import type { ModelHints } from '../utils/model-cache' const CHAT_PROVIDER_ID = 'litellm' const OPENAI_COMPATIBLE_PACKAGE = '@opencode/ai/providers/openai-compatible' +// Of the packages a LiteLLM provider uses, only the Chat Completions ones read +// `compatibility.cacheFormat`; `openai` (Responses by default), the Responses +// packages and the Anthropic packages do not. +const CHAT_COMPLETIONS_PACKAGES = new Set([ + OPENAI_COMPATIBLE_PACKAGE, + '@opencode/ai/providers/openai/chat', +]) +const ANTHROPIC_PACKAGES = new Set([ + '@opencode/ai/providers/anthropic', + '@opencode/ai/providers/anthropic-compatible', +]) +const ADAPTIVE_THINKING = { type: 'adaptive', display: 'summarized' } // Runtime package that talks to `/v1/responses`. OpenCode 2 lets one model // override the provider's package, so Responses models stay under their // original provider id and no sibling provider is needed here. @@ -78,6 +97,7 @@ interface ProviderSource { pendingPublication: boolean filters: ModelFilters capabilities: ModelCapabilities + compatibility: ModelCompatibility formatModelNames: boolean cacheKey: string models: Model.Info[] @@ -111,12 +131,44 @@ function numberOrZero(value: unknown): number { return typeof value === 'number' && Number.isFinite(value) ? value : 0 } +/** + * Discovery spells every effort level as `{ reasoningEffort }`, which the + * OpenAI runtimes read. The Anthropic runtimes read `thinking` and `effort` + * instead, so on those packages the levels are respelled the way OpenCode's + * own catalog does for adaptive-thinking Claude models. A model that does + * not report adaptive thinking gets no variants rather than inert ones. + */ +function anthropicVariants( + variants: Model.Info['variants'], + hints: ModelHints | undefined, +): Model.Info['variants'] { + if (!hints?.adaptiveThinking) return [] + return variants.flatMap((variant) => { + const effort = variant.settings?.reasoningEffort + if (typeof effort !== 'string') return [] + if (effort === 'none') { + if (hints.thinkingAlwaysOn) return [] + return [{ ...variant, settings: { thinking: { type: 'disabled' } } }] + } + return [{ ...variant, settings: { thinking: ADAPTIVE_THINKING, effort } }] + }) +} + +interface ToProviderModelsOptions { + /** The provider's runtime package; models not overridden inherit it. */ + providerPackage: string + responsesIds: ReadonlySet + hints: Record + compatibility: ModelCompatibility +} + function toProviderModels( providerID: string, entries: Record, - responsesIds: ReadonlySet = new Set(), + options: ToProviderModelsOptions, ): Model.Info[] { const provider = Provider.ID.make(providerID) + const { responsesIds, hints, compatibility: overrides } = options return Object.entries(entries).flatMap(([modelID, value]) => { const entry = asRecord(value) @@ -148,10 +200,18 @@ function toProviderModels( : [] const rawVariants = asRecord(entry.variants) - const variants = Object.entries(rawVariants).map(([id, settings]) => ({ + const openAIVariants = Object.entries(rawVariants).map(([id, settings]) => ({ id: id as Model.VariantID, settings: asRecord(settings), })) as Model.Info['variants'] + const pkg = responsesIds.has(modelID) ? RESPONSES_PACKAGE : options.providerPackage + const variants = ANTHROPIC_PACKAGES.has(pkg) + ? anthropicVariants(openAIVariants, hints[modelID]) + : openAIVariants + const compatibility = resolveCompatibility( + inferCompatibility(hints[modelID], CHAT_COMPLETIONS_PACKAGES.has(pkg)), + Object.hasOwn(overrides, modelID) ? overrides[modelID] : undefined, + ) return [ { ...defaults, @@ -159,6 +219,11 @@ function toProviderModels( // runtime package for itself, so it keeps the provider id while // still reaching the Responses API. ...(responsesIds.has(modelID) ? { package: RESPONSES_PACKAGE } : {}), + // `cacheFormat` exists on the host's `Model.Compatibility` (2.0.28+) + // but not in the 2.0.15 schema this package compiles against. + ...(compatibility + ? { compatibility: compatibility as unknown as Model.Info['compatibility'] } + : {}), name: readString(entry.name) ?? modelID, limit: { context: numberOrDefault(limit.context, defaults.limit.context), @@ -228,6 +293,7 @@ function runtimeSettings(source: ProviderSource): Record { 'includeModels', 'excludeModels', 'modelCapabilities', + 'modelCompatibility', 'formatModelNames', 'providerID', ]) { @@ -277,6 +343,7 @@ async function makeProviderSource( const apiKey = configuredApiKey ?? connectedApiKey const filters = readModelFilters(settings) const capabilities = parseModelCapabilities(settings.modelCapabilities) + const compatibility = parseModelCompatibility(settings.modelCompatibility) const formatModelNames = readFormatModelNames(settings) const detectedBaseURL = configuredBaseURL ? normalizeBaseURL(configuredBaseURL) @@ -293,6 +360,7 @@ async function makeProviderSource( const cacheEntry = readModelCacheEntry(cacheKey) let entries = cacheEntry?.models ?? null let responsesIds = cacheEntry?.responsesIds ?? [] + let hints = cacheEntry?.hints ?? {} if (!entries || Object.keys(entries).length === 0) { const discovered = await withTimeout( @@ -310,10 +378,12 @@ async function makeProviderSource( if (discovered && Object.keys(discovered.models).length > 0) { entries = discovered.models responsesIds = discovered.responsesIds - writeModelCache(cacheKey, entries, responsesIds) + hints = discovered.hints + writeModelCache(cacheKey, entries, responsesIds, hints) } else { entries = null responsesIds = [] + hints = {} } } else { logInfo( @@ -321,11 +391,12 @@ async function makeProviderSource( ) } + const providerPackage = provider?.package || OPENAI_COMPATIBLE_PACKAGE return { id, name: provider?.name || 'LiteLLM (proxy)', activation: provider?.activation ?? 'enabled', - package: provider?.package || OPENAI_COMPATIBLE_PACKAGE, + package: providerPackage, settings, headers: customHeaders, integrationID: provider?.integrationID, @@ -339,9 +410,15 @@ async function makeProviderSource( customHeaders: Object.keys(customHeaders).length > 0 ? customHeaders : undefined, filters, capabilities, + compatibility, formatModelNames, cacheKey, - models: toProviderModels(id, entries ?? {}, new Set(responsesIds)), + models: toProviderModels(id, entries ?? {}, { + providerPackage, + responsesIds: new Set(responsesIds), + hints, + compatibility, + }), ownedModelIds: new Set(), } } @@ -395,13 +472,14 @@ async function refreshProviderSource( return } - const models = toProviderModels( - source.id, - discovered.models, - new Set(discovered.responsesIds), - ) + const models = toProviderModels(source.id, discovered.models, { + providerPackage: source.package, + responsesIds: new Set(discovered.responsesIds), + hints: discovered.hints, + compatibility: source.compatibility, + }) if (JSON.stringify(models) === JSON.stringify(source.models)) { - writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds) + writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds, discovered.hints) source.refreshRequired = false return } @@ -415,7 +493,7 @@ async function refreshProviderSource( throw error } - writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds) + writeModelCache(source.cacheKey, discovered.models, discovered.responsesIds, discovered.hints) source.refreshRequired = false logInfo( `[opencode-litellm] Refreshed ${models.length} models for provider "${source.id}"; the provider registry was reloaded.`, diff --git a/src/types/index.ts b/src/types/index.ts index d35b41e..766ceae 100644 --- a/src/types/index.ts +++ b/src/types/index.ts @@ -71,6 +71,8 @@ export interface LiteLLMModelInfo { supported_openai_params?: string[] | null supports_pdf_input?: boolean supports_audio_input?: boolean + supports_adaptive_thinking?: boolean + thinking_always_on?: boolean input_cost_per_token?: number output_cost_per_token?: number cache_read_input_token_cost?: number diff --git a/src/utils/model-cache.ts b/src/utils/model-cache.ts index 2e4e446..355844e 100644 --- a/src/utils/model-cache.ts +++ b/src/utils/model-cache.ts @@ -3,12 +3,33 @@ import { readFileSync, mkdirSync, writeFileSync, renameSync, rmSync } from 'node import { homedir, tmpdir } from 'node:os' import { join } from 'node:path' +/** + * Facts about a discovered model's upstream that OpenCode 2 turns into + * `Model.Info.compatibility` and `variants`. They ride alongside the + * config-shaped model entries (which have no field for them) through + * discovery and the cache. + */ +export interface ModelHints { + /** + * The deployment accepts Anthropic-style `cache_control` markers on + * Chat Completions (`supported_openai_params` lists `cache_control`). + */ + acceptsCacheControl: boolean + /** + * The model takes `thinking: { type: "adaptive" }` plus an `effort` level + * (`supports_adaptive_thinking`). + */ + adaptiveThinking: boolean + /** Thinking cannot be turned off (`thinking_always_on`). */ + thinkingAlwaysOn: boolean +} + /** * Bump when the shape of a cached model entry changes so stale caches * from older plugin versions are ignored rather than deserialized into * an incompatible structure. */ -const CACHE_VERSION = 3 +const CACHE_VERSION = 5 /** * Maximum age of a cache entry before it's treated as a miss. Without a @@ -29,6 +50,11 @@ interface ModelCacheFile { * no transport marker (their shape is the user-facing OpenCode config). */ responsesIds?: string[] + /** + * Per-model facts about the upstream that OpenCode 2 needs but that have + * no place in a provider-config entry (see `ModelHints`). + */ + hints?: Record } /** @@ -134,6 +160,22 @@ function normalizeResponsesIds(value: unknown): string[] { return value.filter((id): id is string => typeof id === 'string') } +/** Keep only well-formed entries of a cached `hints` map. */ +function normalizeHints(value: unknown): Record { + if (!value || typeof value !== 'object' || Array.isArray(value)) return {} + const out: Record = {} + for (const [id, hint] of Object.entries(value as Record)) { + if (!hint || typeof hint !== 'object' || Array.isArray(hint)) continue + const { acceptsCacheControl, adaptiveThinking, thinkingAlwaysOn } = hint as Record + out[id] = { + acceptsCacheControl: acceptsCacheControl === true, + adaptiveThinking: adaptiveThinking === true, + thinkingAlwaysOn: thinkingAlwaysOn === true, + } + } + return out +} + /** * Read cached model entries for a cache key. Returns `null` on any * problem (missing file, parse error, version mismatch, or an entry @@ -155,14 +197,18 @@ export function readModelCache( } /** - * Read the models and Responses-API model ids for a cache key in one - * parse. Returns `null` under the same conditions as - * {@link readModelCache} (missing/expired cache), so the two can't + * Read the models, Responses-API model ids and model hints for a cache key + * in one parse. Returns `null` under the same conditions as + * {@link readModelCache} (missing/expired cache), so the three can't * disagree within a single lookup. */ export function readModelCacheEntry( cacheKey: string, -): { models: Record; responsesIds: string[] } | null { +): { + models: Record + responsesIds: string[] + hints: Record +} | null { const parsed = readCacheFile(cacheKey) if (!parsed) return null if ( @@ -171,7 +217,11 @@ export function readModelCacheEntry( ) { return null } - return { models: parsed.models, responsesIds: normalizeResponsesIds(parsed.responsesIds) } + return { + models: parsed.models, + responsesIds: normalizeResponsesIds(parsed.responsesIds), + hints: normalizeHints(parsed.hints), + } } /** @@ -204,6 +254,7 @@ export function writeModelCache( cacheKey: string, models: Record, responsesIds: readonly string[] = [], + hints: Record = {}, ): void { try { mkdirSync(cacheDir(), { recursive: true }) @@ -212,6 +263,7 @@ export function writeModelCache( savedAt: Date.now(), models, responsesIds: [...responsesIds], + hints, } // Write to a temp file and rename so concurrent OpenCode processes // never observe a partially-written JSON file (rename is atomic on diff --git a/src/utils/model-compatibility.ts b/src/utils/model-compatibility.ts new file mode 100644 index 0000000..15cc4f5 --- /dev/null +++ b/src/utils/model-compatibility.ts @@ -0,0 +1,65 @@ +import type { ModelHints } from './model-cache' + +/** + * Values `cacheFormat` accepts. Validated here because OpenCode validates + * plugin-registered models only when it serializes them, where one bad + * value fails the whole model list. + */ +const CACHE_FORMATS: ReadonlySet = new Set(['anthropic', 'alibaba']) + +/** `null` removes the `cacheFormat` the plugin inferred. */ +export interface CompatibilityOverride { + cacheFormat: string | null +} + +/** Per-model compatibility overrides keyed by exact model id. */ +export type ModelCompatibility = Record + +/** + * Defensively parse the `modelCompatibility` provider option. Non-object + * input, non-object per-model values and values outside the accepted set + * are dropped; `null` is kept. + */ +export function parseModelCompatibility(raw: unknown): ModelCompatibility { + if (!raw || typeof raw !== 'object' || Array.isArray(raw)) return {} + const out: ModelCompatibility = {} + for (const [modelId, entry] of Object.entries(raw as Record)) { + if (!entry || typeof entry !== 'object' || Array.isArray(entry)) continue + const { cacheFormat } = entry as Record + if (cacheFormat === null || (typeof cacheFormat === 'string' && CACHE_FORMATS.has(cacheFormat))) { + out[modelId] = { cacheFormat } + } + } + return out +} + +/** + * Compatibility OpenCode 2 needs for a discovered model, from what the + * proxy reported about its upstream. Only the Chat Completions runtimes + * read `cacheFormat`: Anthropic Messages adds its own cache breakpoints + * and the Responses runtimes do not use the setting. + */ +export function inferCompatibility( + hints: ModelHints | undefined, + chatCompletions: boolean, +): Record { + if (chatCompletions && hints?.acceptsCacheControl) return { cacheFormat: 'anthropic' } + return {} +} + +/** + * Overlay a user override on inferred compatibility. `null` deletes the + * key. Returns `undefined` when nothing remains so callers can leave + * `compatibility` unset. + */ +export function resolveCompatibility( + inferred: Record, + override?: CompatibilityOverride, +): Record | undefined { + const out: Record = { ...inferred } + if (override) { + if (override.cacheFormat === null) delete out.cacheFormat + else out.cacheFormat = override.cacheFormat + } + return Object.keys(out).length > 0 ? out : undefined +} diff --git a/test/model-cache.test.ts b/test/model-cache.test.ts index c3f0dee..cead715 100644 --- a/test/model-cache.test.ts +++ b/test/model-cache.test.ts @@ -75,9 +75,33 @@ describe('model cache', () => { expect(readModelCacheEntry('entry@http://x')).toEqual({ models: { chat: {}, responses: {} }, responsesIds: ['responses'], + hints: {}, }) }) + it('round-trips model hints and drops malformed ones', () => { + const a = { acceptsCacheControl: true, adaptiveThinking: true, thinkingAlwaysOn: false } + const b = { acceptsCacheControl: false, adaptiveThinking: false, thinkingAlwaysOn: true } + writeModelCache('hints@http://x', { a: {}, b: {} }, [], { a, b }) + expect(readModelCacheEntry('hints@http://x')?.hints).toEqual({ a, b }) + rewriteCacheFile('hints@http://x', (parsed) => { + parsed.hints = { a: { acceptsCacheControl: 'yes', adaptiveThinking: 1 }, b: null, c: [] } + }) + expect(readModelCacheEntry('hints@http://x')?.hints).toEqual({ + a: { acceptsCacheControl: false, adaptiveThinking: false, thinkingAlwaysOn: false }, + }) + }) + + it('treats caches that predate the thinking hints as a miss', () => { + writeModelCache(KEY, { m: {} }, ['m']) + rewriteCacheFile(KEY, (parsed) => { + parsed.version = 4 + parsed.hints = { m: { acceptsCacheControl: true } } + }) + expect(readModelCacheEntry(KEY)).toBeNull() + expect(readModelCache(KEY)).toBeNull() + }) + it('ignores cache files written by a different plugin version', () => { writeModelCache(KEY, { m: {} }) rewriteCacheFile(KEY, (parsed) => { diff --git a/test/model-compatibility.test.ts b/test/model-compatibility.test.ts new file mode 100644 index 0000000..6619ecf --- /dev/null +++ b/test/model-compatibility.test.ts @@ -0,0 +1,74 @@ +import { describe, expect, it } from 'vitest' +import { + inferCompatibility, + parseModelCompatibility, + resolveCompatibility, +} from '../src/utils/model-compatibility' +import type { ModelHints } from '../src/utils/model-cache' + +const hints = (acceptsCacheControl: boolean): ModelHints => ({ + acceptsCacheControl, + adaptiveThinking: false, + thinkingAlwaysOn: false, +}) + +describe('parseModelCompatibility', () => { + it('returns an empty map for non-object input', () => { + expect(parseModelCompatibility(undefined)).toEqual({}) + expect(parseModelCompatibility(null)).toEqual({}) + expect(parseModelCompatibility('cacheFormat')).toEqual({}) + expect(parseModelCompatibility([{ cacheFormat: 'anthropic' }])).toEqual({}) + }) + + it('keeps an accepted cacheFormat or null', () => { + expect( + parseModelCompatibility({ + m1: { cacheFormat: 'anthropic' }, + m2: { cacheFormat: 'alibaba' }, + m3: { cacheFormat: null }, + }), + ).toEqual({ + m1: { cacheFormat: 'anthropic' }, + m2: { cacheFormat: 'alibaba' }, + m3: { cacheFormat: null }, + }) + }) + + it('drops rejected values, other keys and empty entries', () => { + expect( + parseModelCompatibility({ + m1: { cacheFormat: 'Anthropic' }, + m2: { cacheFormat: true }, + m3: { supportsThinkingBlockBinding: false }, + m4: 'anthropic', + m5: {}, + }), + ).toEqual({}) + }) +}) + +describe('inferCompatibility', () => { + it('asks Chat Completions for Anthropic cache markers when the proxy accepts them', () => { + expect(inferCompatibility(hints(true), true)).toEqual({ cacheFormat: 'anthropic' }) + }) + + it('infers nothing without the hint or outside Chat Completions', () => { + expect(inferCompatibility(hints(false), true)).toEqual({}) + expect(inferCompatibility(undefined, true)).toEqual({}) + expect(inferCompatibility(hints(true), false)).toEqual({}) + }) +}) + +describe('resolveCompatibility', () => { + it('lets overrides add, replace and remove keys', () => { + expect(resolveCompatibility({ cacheFormat: 'anthropic' }, { cacheFormat: 'alibaba' })).toEqual({ + cacheFormat: 'alibaba', + }) + expect(resolveCompatibility({}, { cacheFormat: 'anthropic' })).toEqual({ cacheFormat: 'anthropic' }) + }) + + it('returns undefined when nothing remains', () => { + expect(resolveCompatibility({}, undefined)).toBeUndefined() + expect(resolveCompatibility({ cacheFormat: 'anthropic' }, { cacheFormat: null })).toBeUndefined() + }) +}) diff --git a/test/plugin-v1.test.ts b/test/plugin-v1.test.ts index d7ae16f..040238c 100644 --- a/test/plugin-v1.test.ts +++ b/test/plugin-v1.test.ts @@ -3,7 +3,12 @@ import { tmpdir } from 'node:os' import { join } from 'node:path' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import { LiteLLMPlugin } from '../src/plugin' -import { buildCacheKey, writeModelCache } from '../src/utils/model-cache' +import { + buildCacheKey, + readModelCacheEntry, + readModelCacheSavedAt, + writeModelCache, +} from '../src/utils/model-cache' import { __resetOpenCodeAuthCacheForTests } from '../src/utils/opencode-auth' interface ProviderConfig { @@ -59,7 +64,7 @@ describe('OpenCode 1 config hook', () => { process.env.HOME = homeDirectory } - function mockProxy(models: Array<{ id: string; mode?: string }>): void { + function mockProxy(models: Array<{ id: string; mode?: string; params?: string[] }>): void { globalThis.fetch = vi.fn(async (input) => { const url = String(input) if (url.endsWith('/v1/model/info')) { @@ -67,7 +72,11 @@ describe('OpenCode 1 config hook', () => { JSON.stringify({ data: models.map((model) => ({ model_name: model.id, - model_info: { key: model.id, mode: model.mode ?? 'chat' }, + model_info: { + key: model.id, + mode: model.mode ?? 'chat', + supported_openai_params: model.params, + }, })), }), { status: 200 }, @@ -122,6 +131,56 @@ describe('OpenCode 1 config hook', () => { expect(Object.keys(sibling.models ?? {})).toEqual(['responses-model']) }) + it('persists model hints so an OpenCode 2 start can reuse the shared cache', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45557/v1' + mockProxy([ + { id: 'claude', params: ['max_tokens', 'cache_control'] }, + { id: 'gpt', params: ['max_tokens'] }, + ]) + + await runConfigHook(baseURL) + + const cacheKey = buildCacheKey('litellm', 'http://127.0.0.1:45557', {}, {}) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + claude: { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + gpt: { acceptsCacheControl: false, adaptiveThinking: false, thinkingAlwaysOn: false }, + }) + }) + + it('keeps model hints in the shared cache across a background refresh', async () => { + useIsolatedState() + const baseURL = 'http://127.0.0.1:45558/v1' + mockProxy([{ id: 'claude', params: ['cache_control'] }]) + const input = { client: { app: { log: vi.fn(async () => {}) } } } + const plugin = await LiteLLMPlugin(input as never) + const config: OpenCodeConfig = { + provider: { + litellm: { + npm: '@ai-sdk/openai-compatible', + name: 'LiteLLM (proxy)', + options: { baseURL, apiKey: 'example-key' }, + models: {}, + }, + }, + } + // Cold start, backdated so the next session is allowed to refresh. + const started = Date.now() + const dateNow = vi.spyOn(Date, 'now').mockReturnValue(started - 10 * 60 * 1000) + await plugin.config!(config as never) + dateNow.mockRestore() + const cacheKey = buildCacheKey('litellm', 'http://127.0.0.1:45558', {}, {}) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + claude: { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + }) + + await plugin.event!({ event: { type: 'session.created' } } as never) + await vi.waitFor(() => expect(readModelCacheSavedAt(cacheKey)).toBeGreaterThanOrEqual(started)) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + claude: { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + }) + }) + it('creates no sibling provider when no model needs the Responses API', async () => { useIsolatedState() const baseURL = 'http://127.0.0.1:45552/v1' diff --git a/test/plugin-v2.test.ts b/test/plugin-v2.test.ts index 1e49dc1..7ac6dc8 100644 --- a/test/plugin-v2.test.ts +++ b/test/plugin-v2.test.ts @@ -4,7 +4,12 @@ import { join } from 'node:path' import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' import plugin from '../src' import type { Context } from '@opencode/plugin/promise/plugin' -import { buildCacheKey, writeModelCache } from '../src/utils/model-cache' +import { + buildCacheKey, + readModelCacheEntry, + readModelCacheSavedAt, + writeModelCache, +} from '../src/utils/model-cache' import { __resetOpenCodeAuthCacheForTests } from '../src/utils/opencode-auth' function restoreEnv(name: string, value: string | undefined): void { @@ -96,6 +101,8 @@ describe('OpenCode 2 plugin entrypoint', () => { supports_vision: true, supports_reasoning: true, supports_reasoning_efforts: ['low', 'high'], + supports_adaptive_thinking: true, + supported_openai_params: ['max_tokens', 'cache_control', 'reasoning_effort'], input_cost_per_token: 0.000003, output_cost_per_token: 0.000015, cache_read_input_token_cost: 0.000001, @@ -137,7 +144,7 @@ describe('OpenCode 2 plugin entrypoint', () => { supports_minimal_reasoning_effort: false, supports_xhigh_reasoning_effort: true, reasoning_effort_levels: null, - supported_openai_params: ['reasoning'], + supported_openai_params: ['reasoning', 'cache_control'], }, }, ], @@ -228,6 +235,8 @@ describe('OpenCode 2 plugin entrypoint', () => { { id: 'low', settings: { reasoningEffort: 'low' } }, { id: 'high', settings: { reasoningEffort: 'high' } }, ], + // Chat Completions only emits Anthropic cache markers when told to. + compatibility: { cacheFormat: 'anthropic' }, }) expect(registered[0].models[1]).toMatchObject({ @@ -261,11 +270,383 @@ describe('OpenCode 2 plugin entrypoint', () => { expect(registered[0].models[0].package).toBeUndefined() expect(registered[0].models[1].package).toBeUndefined() expect(registered[0].models[2].package).toBeUndefined() + // No `cache_control` reported, or not a Chat Completions model: nothing inferred. + expect(registered[0].models[1].compatibility).toBeUndefined() + expect(registered[0].models[2].compatibility).toBeUndefined() + expect(registered[0].models[3].compatibility).toBeUndefined() + const cacheKey = buildCacheKey('litellm', 'http://127.0.0.1:44444', {}, {}) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + 'anthropic/claude-3-5-sonnet': { + acceptsCacheControl: true, + adaptiveThinking: true, + thinkingAlwaysOn: false, + }, + 'gpt-6-astra': { acceptsCacheControl: false, adaptiveThinking: false, thinkingAlwaysOn: false }, + 'gpt-5-search-api': { acceptsCacheControl: false, adaptiveThinking: false, thinkingAlwaysOn: false }, + 'company-pro': { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + }) await cleanup?.() expect(disposeTransform).toHaveBeenCalledOnce() }) + it('applies modelCompatibility overrides on top of inferred compatibility', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'claude-direct', + model_info: { + key: 'claude-direct', + mode: 'chat', + supported_openai_params: ['cache_control'], + }, + }, + { + model_name: 'claude-bedrock', + model_info: { + key: 'claude-bedrock', + mode: 'chat', + supported_openai_params: ['max_tokens'], + }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response( + JSON.stringify({ data: [ + { id: 'claude-direct', object: 'model' }, + { id: 'claude-bedrock', object: 'model' }, + ] }), + { status: 200 }, + ) + }) + + const registered: Array<{ info: Record; models: Array> }> = [] + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { info: Record; models: Array> }) => { + registered.push(entry) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: { + baseURL: 'http://127.0.0.1:44444/v1', + modelCompatibility: { + // `null` removes the inferred key. + 'claude-direct': { cacheFormat: null }, + // Set where the proxy does not advertise `cache_control`. + 'claude-bedrock': { cacheFormat: 'anthropic' }, + // An id the proxy does not list adds no model. + unknown: { cacheFormat: 'anthropic' }, + }, + }, + provider: { + list: vi.fn(async () => ({ data: [] })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor as never) + return { dispose: vi.fn(async () => {}) } + }), + reload: vi.fn(async () => {}), + }, + event: { subscribe: () => (async function* () {})() }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + + expect(registered).toHaveLength(1) + expect(registered[0].models.map((model) => [model.id, model.compatibility])).toEqual([ + ['claude-direct', undefined], + ['claude-bedrock', { cacheFormat: 'anthropic' }], + ]) + // runtimeSettings strips plugin-only options. + expect(registered[0].info.settings).not.toHaveProperty('modelCompatibility') + await cleanup?.() + }) + + it('applies compatibility from cached hints without touching the network', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + writeModelCache( + buildCacheKey('litellm', 'http://127.0.0.1:44444', {}, {}), + { claude: { name: 'Claude' }, qwen: { name: 'Qwen' }, constructor: { name: 'Constructor' } }, + [], + { + claude: { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + qwen: { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + constructor: { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + }, + ) + globalThis.fetch = vi.fn(async () => { + throw new Error('the cached path must not hit the network') + }) + + const registered: Array<{ info: Record; models: Array> }> = [] + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { info: Record; models: Array> }) => { + registered.push(entry) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: { + baseURL: 'http://127.0.0.1:44444/v1', + modelCompatibility: { qwen: { cacheFormat: 'alibaba' } }, + }, + provider: { + list: vi.fn(async () => ({ data: [] })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor as never) + return { dispose: vi.fn(async () => {}) } + }), + reload: vi.fn(async () => {}), + }, + event: { subscribe: () => (async function* () {})() }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + + expect(globalThis.fetch).not.toHaveBeenCalled() + expect(registered[0].models.map((model) => [model.id, model.compatibility])).toEqual([ + ['claude', { cacheFormat: 'anthropic' }], + ['qwen', { cacheFormat: 'alibaba' }], + // An id that is also an `Object.prototype` member has no override. + ['constructor', { cacheFormat: 'anthropic' }], + ]) + await cleanup?.() + }) + + it('infers cacheFormat only for Chat Completions runtime packages', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'claude', + model_info: { key: 'claude', mode: 'chat', supported_openai_params: ['cache_control'] }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response(JSON.stringify({ data: [{ id: 'claude', object: 'model' }] }), { status: 200 }) + }) + + const registered: Array<{ info: Record; models: Array> }> = [] + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { info: Record; models: Array> }) => { + registered.push(entry) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: {}, + provider: { + list: vi.fn(async () => ({ + data: [ + // `openai` defaults to the Responses API, which does not read `cacheFormat`. + ...[ + ['litellm-openai', '@opencode/ai/providers/openai'], + ['litellm-anthropic', '@opencode/ai/providers/anthropic-compatible'], + ['litellm-chat', '@opencode/ai/providers/openai/chat'], + ].map(([id, pkg]) => ({ + id, + activation: 'enabled', + package: pkg, + settings: { baseURL: 'http://127.0.0.1:44444/v1' }, + })), + ], + })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor as never) + return { dispose: vi.fn(async () => {}) } + }), + reload: vi.fn(async () => {}), + }, + event: { subscribe: () => (async function* () {})() }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + + expect(registered.map((entry) => [entry.info.id, entry.models[0].compatibility])).toEqual([ + ['litellm-openai', undefined], + ['litellm-anthropic', undefined], + ['litellm-chat', { cacheFormat: 'anthropic' }], + ]) + await cleanup?.() + }) + + it('respells effort variants for providers on an Anthropic runtime package', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + // Adaptive thinking, can be switched off: `none` becomes `disabled`. + model_name: 'claude-adaptive', + model_info: { + key: 'claude-adaptive', + mode: 'chat', + supports_reasoning: true, + supports_adaptive_thinking: true, + reasoning_effort_levels: ['none', 'low', 'max'], + supported_openai_params: ['cache_control', 'reasoning_effort'], + }, + }, + { + // Adaptive thinking that is always on: `none` is dropped. + model_name: 'claude-always-on', + model_info: { + key: 'claude-always-on', + mode: 'chat', + supports_reasoning: true, + supports_adaptive_thinking: true, + thinking_always_on: true, + reasoning_effort_levels: ['none', 'high'], + }, + }, + { + // No adaptive thinking reported: `reasoningEffort` would be inert, so no variants. + model_name: 'claude-manual', + model_info: { + key: 'claude-manual', + mode: 'chat', + supports_reasoning: true, + reasoning_effort_levels: ['low', 'high'], + }, + }, + { + // Responses override on the same provider: the OpenAI runtime keeps `reasoningEffort`. + model_name: 'gpt-responses', + model_info: { + key: 'gpt-responses', + mode: 'responses', + supports_reasoning: true, + supports_adaptive_thinking: true, + reasoning_effort_levels: ['low'], + }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response( + JSON.stringify({ data: [ + { id: 'claude-adaptive', object: 'model' }, + { id: 'claude-always-on', object: 'model' }, + { id: 'claude-manual', object: 'model' }, + { id: 'gpt-responses', object: 'model' }, + ] }), + { status: 200 }, + ) + }) + + const registered: Array<{ info: Record; models: Array> }> = [] + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { info: Record; models: Array> }) => { + registered.push(entry) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: { baseURL: 'http://127.0.0.1:44444/v1' }, + provider: { + list: vi.fn(async () => ({ + data: [ + { + id: 'litellm-claude', + name: 'LiteLLM Claude', + activation: 'enabled', + package: '@opencode/ai/providers/anthropic-compatible', + settings: { baseURL: 'http://127.0.0.1:44444/v1' }, + }, + { + id: 'litellm-anthropic', + name: 'LiteLLM Anthropic', + activation: 'enabled', + package: '@opencode/ai/providers/anthropic', + settings: { baseURL: 'http://127.0.0.1:44444/v1' }, + }, + ], + })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor as never) + return { dispose: vi.fn(async () => {}) } + }), + reload: vi.fn(async () => {}), + }, + event: { subscribe: () => (async function* () {})() }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + + expect(registered.map((entry) => entry.info.id)).toEqual(['litellm-claude', 'litellm-anthropic']) + expect(registered[0].info).toMatchObject({ + id: 'litellm-claude', + package: '@opencode/ai/providers/anthropic-compatible', + }) + const adaptive = { type: 'adaptive', display: 'summarized' } + expect(registered[0].models.map((model) => [model.id, model.variants])).toEqual([ + [ + 'claude-adaptive', + [ + { id: 'none', settings: { thinking: { type: 'disabled' } } }, + { id: 'low', settings: { thinking: adaptive, effort: 'low' } }, + { id: 'max', settings: { thinking: adaptive, effort: 'max' } }, + ], + ], + ['claude-always-on', [{ id: 'high', settings: { thinking: adaptive, effort: 'high' } }]], + ['claude-manual', []], + ['gpt-responses', [{ id: 'low', settings: { reasoningEffort: 'low' } }]], + ]) + // Both Anthropic packages get the same spelling. + expect(registered[1].models.map((model) => model.variants)).toEqual( + registered[0].models.map((model) => model.variants), + ) + // The Anthropic runtime places cache breakpoints itself; nothing to declare, + // even though `claude-adaptive` advertises `cache_control`. + const all = registered.flatMap((entry) => entry.models) + expect(all.every((model) => model.compatibility === undefined)).toBe(true) + await cleanup?.() + }) + it('uses service environment variables to create the default provider', async () => { cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-env-test-')) process.env.XDG_CACHE_HOME = cacheDirectory @@ -348,7 +729,20 @@ describe('OpenCode 2 plugin entrypoint', () => { globalThis.fetch = vi.fn(async (input) => { const url = String(input) if (url.endsWith('/v1/model/info')) { - return new Response(JSON.stringify({ data: [] }), { status: 200 }) + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'anthropic/claude-3-5-sonnet', + model_info: { + key: 'anthropic/claude-3-5-sonnet', + supported_openai_params: ['cache_control'], + }, + }, + ], + }), + { status: 200 }, + ) } return new Response( JSON.stringify({ data: [{ id: 'anthropic/claude-3-5-sonnet', object: 'model' }] }), @@ -361,7 +755,11 @@ describe('OpenCode 2 plugin entrypoint', () => { name: 'Configured LiteLLM', activation: 'enabled', package: '@opencode/ai/providers/openai-compatible', - settings: { baseURL: `${baseURL}/v1`, apiKey: 'example-api-key' }, + settings: { + baseURL: `${baseURL}/v1`, + apiKey: 'example-api-key', + modelCompatibility: { 'anthropic/claude-3-5-sonnet': { cacheFormat: 'alibaba' } }, + }, headers: { 'X-Gateway': 'test' }, } const curatedModel = { id: 'curated-model', name: 'Curated model' } @@ -406,6 +804,8 @@ describe('OpenCode 2 plugin entrypoint', () => { expect(configuredProvider.headers).toMatchObject({ 'X-Gateway': 'test' }) expect(updatedModels.map((model) => model.id)).toContain('curated-model') expect(updatedModels.map((model) => model.id)).toContain('anthropic/claude-3-5-sonnet') + const discovered = updatedModels.find((model) => model.id === 'anthropic/claude-3-5-sonnet') + expect(discovered?.compatibility).toEqual({ cacheFormat: 'alibaba' }) await cleanup?.() }) @@ -423,10 +823,37 @@ describe('OpenCode 2 plugin entrypoint', () => { globalThis.fetch = vi.fn(async (input) => { const url = String(input) if (url.endsWith('/v1/model/info')) { - return new Response(JSON.stringify({ data: [] }), { status: 200 }) + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'anthropic/claude-3-5-sonnet', + model_info: { + key: 'anthropic/claude-3-5-sonnet', + mode: 'chat', + supported_openai_params: ['cache_control'], + }, + }, + { + model_name: 'vertex/claude-sonnet', + model_info: { + key: 'vertex/claude-sonnet', + mode: 'chat', + supported_openai_params: ['cache_control'], + }, + }, + ], + }), + { status: 200 }, + ) } return new Response( - JSON.stringify({ data: [{ id: 'anthropic/claude-3-5-sonnet', object: 'model' }] }), + JSON.stringify({ + data: [ + { id: 'anthropic/claude-3-5-sonnet', object: 'model' }, + { id: 'vertex/claude-sonnet', object: 'model' }, + ], + }), { status: 200 }, ) }) @@ -452,7 +879,10 @@ describe('OpenCode 2 plugin entrypoint', () => { }) const context = { app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, - options: { baseURL: `${baseURL}/v1` }, + options: { + baseURL: `${baseURL}/v1`, + modelCompatibility: { 'anthropic/claude-3-5-sonnet': { cacheFormat: 'alibaba' } }, + }, provider: { list: vi.fn(async () => ({ data: [] })), transform: vi.fn(async (transform: (editor: unknown) => void) => { @@ -474,12 +904,194 @@ describe('OpenCode 2 plugin entrypoint', () => { await vi.waitFor(() => expect(reload).toHaveBeenCalledOnce()) expect(registeredModels).toHaveLength(2) expect(registeredModels[0].map((model) => model.id)).toEqual(['cached-model']) - expect(registeredModels[1].map((model) => model.id)).toEqual([ - 'anthropic/claude-3-5-sonnet', + // Inference and the override are both applied on refresh, and the refreshed hints are persisted. + expect(registeredModels[1].map((model) => [model.id, model.compatibility])).toEqual([ + ['anthropic/claude-3-5-sonnet', { cacheFormat: 'alibaba' }], + ['vertex/claude-sonnet', { cacheFormat: 'anthropic' }], ]) + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + 'anthropic/claude-3-5-sonnet': { + acceptsCacheControl: true, + adaptiveThinking: false, + thinkingAlwaysOn: false, + }, + 'vertex/claude-sonnet': { + acceptsCacheControl: true, + adaptiveThinking: false, + thinkingAlwaysOn: false, + }, + }) + await cleanup?.() + }) + + it('keeps Anthropic variants and no cacheFormat across a background refresh', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-refresh-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + const baseURL = 'http://127.0.0.1:44444' + const cacheKey = buildCacheKey('litellm-claude', baseURL, {}, {}) + const now = Date.now() + const dateNow = vi.spyOn(Date, 'now').mockReturnValue(now - 10 * 60 * 1000) + writeModelCache( + cacheKey, + { 'cached-model': { name: 'Cached Model', variants: { high: { reasoningEffort: 'high' } } } }, + [], + { 'cached-model': { acceptsCacheControl: true, adaptiveThinking: true, thinkingAlwaysOn: false } }, + ) + dateNow.mockRestore() + + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'claude', + model_info: { + key: 'claude', + mode: 'chat', + supports_reasoning: true, + supports_adaptive_thinking: true, + reasoning_effort_levels: ['high'], + supported_openai_params: ['cache_control'], + }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response(JSON.stringify({ data: [{ id: 'claude', object: 'model' }] }), { status: 200 }) + }) + + const registeredModels: Array>> = [] + let providerTransform: ((editor: unknown) => void) | undefined + const editor = { + list: () => [], + get: () => undefined, + add: (entry: { models: Array> }) => { + registeredModels.push(entry.models) + }, + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const reload = vi.fn(async () => { + providerTransform?.(editor) + }) + const context = { + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: {}, + provider: { + list: vi.fn(async () => ({ + data: [ + { + id: 'litellm-claude', + activation: 'enabled', + package: '@opencode/ai/providers/anthropic-compatible', + settings: { baseURL: `${baseURL}/v1` }, + }, + ], + })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + providerTransform = transform + transform(editor) + return { dispose: vi.fn(async () => {}) } + }), + reload, + }, + event: { + subscribe: () => + (async function* () { + yield { type: 'session.created' } + })(), + }, + } as unknown as Context + + const cleanup = await plugin.setup(context) + await vi.waitFor(() => expect(reload).toHaveBeenCalledOnce()) + const high = [ + { id: 'high', settings: { thinking: { type: 'adaptive', display: 'summarized' }, effort: 'high' } }, + ] + // Cached hints drive the warm start; discovered hints drive the refresh. + expect(registeredModels[0].map((model) => model.compatibility)).toEqual([undefined]) + expect(registeredModels[0].map((model) => model.variants)).toEqual([high]) + expect(registeredModels[1].map((model) => model.id)).toEqual(['claude']) + expect(registeredModels[1][0].compatibility).toBeUndefined() + expect(registeredModels[1][0].variants).toEqual(high) await cleanup?.() }) + it('keeps hints in the cache when a background refresh finds no changes', async () => { + cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-refresh-test-')) + process.env.XDG_CACHE_HOME = cacheDirectory + const baseURL = 'http://127.0.0.1:44444' + const cacheKey = buildCacheKey('litellm', baseURL, {}, {}) + globalThis.fetch = vi.fn(async (input) => { + const url = String(input) + if (url.endsWith('/v1/model/info')) { + return new Response( + JSON.stringify({ + data: [ + { + model_name: 'claude', + model_info: { key: 'claude', mode: 'chat', supported_openai_params: ['cache_control'] }, + }, + ], + }), + { status: 200 }, + ) + } + return new Response(JSON.stringify({ data: [{ id: 'claude', object: 'model' }] }), { status: 200 }) + }) + const editor = { + list: () => [], + get: () => undefined, + add: vi.fn(), + update: vi.fn(), + remove: vi.fn(), + models: { set: vi.fn(), update: vi.fn(), remove: vi.fn() }, + } + const reload = vi.fn(async () => {}) + const makeContext = (events: Array<{ type: string }>) => + ({ + app: { name: 'OpenCode', version: '2.0.14', channel: 'stable' }, + options: { baseURL: `${baseURL}/v1` }, + provider: { + list: vi.fn(async () => ({ data: [] })), + transform: vi.fn(async (transform: (editor: unknown) => void) => { + transform(editor) + return { dispose: vi.fn(async () => {}) } + }), + reload, + }, + event: { + subscribe: () => + (async function* () { + yield* events + })(), + }, + }) as unknown as Context + + // Cold start, backdated so the next session is allowed to refresh. + const started = Date.now() + const dateNow = vi.spyOn(Date, 'now').mockReturnValue(started - 10 * 60 * 1000) + const first = await plugin.setup(makeContext([])) + dateNow.mockRestore() + await first?.() + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + claude: { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + }) + + const second = await plugin.setup(makeContext([{ type: 'session.created' }])) + await vi.waitFor(() => expect(readModelCacheSavedAt(cacheKey)).toBeGreaterThanOrEqual(started)) + expect(reload).not.toHaveBeenCalled() + expect(readModelCacheEntry(cacheKey)?.hints).toEqual({ + claude: { acceptsCacheControl: true, adaptiveThinking: false, thinkingAlwaysOn: false }, + }) + await second?.() + }) + it('registers providers that only appear after setup (provider.updated)', async () => { cacheDirectory = mkdtempSync(join(tmpdir(), 'opencode-litellm-late-provider-test-')) process.env.XDG_CACHE_HOME = cacheDirectory