diff --git a/CHANGELOG.md b/CHANGELOG.md index e08df45..c26839a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,10 @@ ## Unreleased +### Breaking changes + +- Require documented Provider API access. `403 upgrade_required` now returns a clear entitlement/migration error instead of automatically calling the internal `/alpha/generate` endpoint. Go-plan users must use the official CLI within its supported plan or an account with Provider API entitlement; signing in alone does not grant API access. Existing credentials and sessions are preserved. See the README migration section. The Go live-test profile is removed; mock Pi/OMP tests assert that no generation fallback request occurs. + ## 0.7.8 - 2026-10-15 - Redact sensitive text in direct stream errors and normalized context-overflow messages before returning them to the host. This closes two error-reporting paths that did not use the shared redactor; it is defensive hardening, not a claim of a confirmed credential leak ([#155](https://github.com/patlux/pi-commandcode-provider/pull/155)). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 931a69e..ced2a82 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -45,12 +45,11 @@ Both commands accept additional pi arguments after `--`, for example `npm run pi Run the transport-specific live tests with separate credentials: ```sh -COMMANDCODE_E2E_GO_API_KEY_FILE=/path/to/go-key npm run test:e2e:live:go COMMANDCODE_E2E_GOAT_API_KEY_FILE=/path/to/goat-key npm run test:e2e:live:goat COMMANDCODE_E2E_PROVIDER_API_KEY_FILE=/path/to/provider-key npm run test:e2e:live:provider ``` -Use `npm run test:e2e:live:all` with the Go and GOAT file variables to run both subscription transports sequentially. Store keys in a secret manager and export each one to a new mode-`0600` temporary file for the test; never add key files to the repository. Direct `*_API_KEY` variables are intended primarily for protected CI secrets. +Use `npm run test:e2e:live:all` with the GOAT and Provider file variables to run both entitled profiles sequentially. The Go profile is removed: a mock `403 upgrade_required` must produce the migration error without an internal generation request on both real hosts. Store keys in a secret manager and export each one to a new mode-`0600` temporary file for the test; never add key files to the repository. Direct `*_API_KEY` variables are intended primarily for protected CI secrets. ### pi end-to-end diff --git a/README.md b/README.md index fb383e4..46625df 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,18 @@ A custom provider for [pi](https://github.com/earendil-works/pi) that connects t > **Disclaimer:** This is an unofficial, community-maintained integration. It is not affiliated with, endorsed by, or supported by Command Code. You need your own Command Code account, API key, and a plan with Provider API access. Command Code's terms, availability, and pricing apply. +## Breaking migration: Provider API access required + +The next breaking release removes automatic `/alpha/generate` fallback. A `403 upgrade_required` now stops with an actionable error; it never retries your prompt against the internal CLI endpoint. The [official Provider API documentation](https://commandcode.ai/docs/provider) excludes API use from the Go plan. A working login or quota response does not establish API entitlement. + +If you used the Go fallback: + +1. Check the current documented plans and model access with Command Code. Use an account with Provider API entitlement, or use the official CLI within its supported plan. +2. After changing accounts, run `/login` again and reload the host. Existing sessions, credentials and model selections are not deleted or migrated by this change. +3. Retry a small request and check `/commandcode-status`. There is no switch to re-enable internal generation. Do not use alternate endpoint overrides to bypass plan restrictions. + +GOAT/Provider accounts with API entitlement continue to use the same documented inference endpoints. Quota reporting is unchanged. This is a breaking-change proposal, not a claim that the version on npm already includes it. + ## Install ```sh @@ -98,7 +110,7 @@ A model whose catalog entry omits `supported_endpoints` keeps Chat Completions, ### Reasoning support -Reasoning capability and selectable effort levels follow the official CLI catalog independently. Models can therefore be marked as reasoning-capable even when Command Code chooses their depth automatically. Models with explicit effort support register a model-specific `thinkingLevelMap`, so pi and OMP expose only valid levels. For a few reasoning models the CLI catalog ships no effort levels although the endpoint accepts `reasoning_effort`; `src/commandcode-catalog-overrides.ts` adds a manual level set for those on top of the generated catalog (currently empty because upstream publishes every level). The catalog sync removes an override as soon as upstream publishes its own levels. Pi's native OpenAI- and Anthropic-compatible providers translate the selected level for Provider API accounts; the existing Command Code generate transport sends the matching `reasoning_effort` for Go accounts. +Reasoning capability and selectable effort levels follow the official CLI catalog independently. Models can therefore be marked as reasoning-capable even when Command Code chooses their depth automatically. Models with explicit effort support register a model-specific `thinkingLevelMap`, so pi and OMP expose only valid levels. For a few reasoning models the CLI catalog ships no effort levels although the endpoint accepts `reasoning_effort`; `src/commandcode-catalog-overrides.ts` adds a manual level set for those on top of the generated catalog (currently empty because upstream publishes every level). The catalog sync removes an override as soon as upstream publishes its own levels. Pi's native OpenAI- and Anthropic-compatible providers translate the selected level for Provider API accounts. List Command Code models from the terminal: @@ -155,11 +167,11 @@ The provider advertises image input only for models marked with the `image` inpu For vision-capable models, Pi's native provider adapters forward image blocks from user messages and tool results using the documented OpenAI or Anthropic message schema. Unknown and text-only models remain marked text-only in Pi. -The legacy generate transport resolves image support from the host's `model.input` when the host supplies it, and falls back to the capability snapshot otherwise. Because the snapshot is generated from a single CLI release, a host that marks a model as `["text", "image"]` — for example through a `models.yml` or `models.json` model override — can send images on both transports before the catalog catches up. A host that narrows a catalogued vision model to text is likewise honored. +The native host adapter uses the registered model's input capabilities on the Provider API. A host model override can advertise image support before the catalog catches up, but cannot grant missing upstream model or plan access. ## Pricing display -The Command Code Provider API does not currently include prices in its model catalog. This extension therefore keeps a static table for models with known prices so pi and OMP can display estimated request costs. DeepSeek V4 uses time-dependent rates: the extension applies the documented 2x peak rates on weekdays from 01:00-04:00 and 06:00-10:00 UTC, and off-peak rates at all other times, including the entire weekend. The same request-time selection is used by the native Provider API path and the legacy generate fallback. +The Command Code Provider API does not currently include prices in its model catalog. This extension therefore keeps a static table for models with known prices so pi and OMP can display estimated request costs. DeepSeek V4 uses time-dependent rates: the extension applies the documented 2x peak rates on weekdays from 01:00-04:00 and 06:00-10:00 UTC, and off-peak rates at all other times, including the entire weekend. These rates are selected at request time on the Provider API path. Models missing from that table display zero cost in pi. This does **not** mean that Command Code will bill the request at zero. The Command Code Usage page remains authoritative for each request. Check the current [Command Code pricing](https://commandcode.ai/docs/resources/pricing-limits) before relying on the displayed value. @@ -209,26 +221,23 @@ Both commands accept additional pi arguments after `--`, for example `npm run pi ### Live transport tests -Keep Go-, GOAT-, and optional Provider-plan test keys in separate secret-manager entries. Pass them through protected files so the keys do not enter shell history: +Keep GOAT- and Provider-plan test keys in separate secret-manager entries. Pass them through protected files so the keys do not enter shell history: ```sh -COMMANDCODE_E2E_GO_API_KEY_FILE=/path/to/go-key \ - npm run test:e2e:live:go - COMMANDCODE_E2E_GOAT_API_KEY_FILE=/path/to/goat-key \ npm run test:e2e:live:goat COMMANDCODE_E2E_PROVIDER_API_KEY_FILE=/path/to/provider-key \ npm run test:e2e:live:provider -COMMANDCODE_E2E_GO_API_KEY_FILE=/path/to/go-key \ +COMMANDCODE_E2E_PROVIDER_API_KEY_FILE=/path/to/provider-key \ COMMANDCODE_E2E_GOAT_API_KEY_FILE=/path/to/goat-key \ npm run test:e2e:live:all ``` -Each profile runs with an isolated Pi agent directory and asserts transport selection, reasoning across turns, quota plan identity, abort handling, tool calls, and the packed npm artifact. Go must select `generate` and reject unsupported images; GOAT must select `provider` and complete a live vision request. The profile-specific `*_API_KEY` environment variables are also supported for CI secrets, but key files are preferred for local use. +Each profile runs with an isolated Pi agent directory and asserts transport selection, reasoning across turns, quota plan identity, abort handling, tool calls, and the packed npm artifact. Both profiles must select `provider`; GOAT also completes a live vision request. Go entitlement denial and the absence of any internal generation request are covered by mock Pi/OMP tests, not paid live probes. The profile-specific `*_API_KEY` environment variables are also supported for CI secrets, but key files are preferred for local use. -The Go profile defaults to DeepSeek V4 Flash; GOAT defaults to Grok 4.6 because its Provider API stream exposes reasoning consistently across consecutive turns. Override them with `COMMANDCODE_E2E_GO_MODEL`, `COMMANDCODE_E2E_GOAT_MODEL`, or `COMMANDCODE_E2E_PROVIDER_MODEL`. The GOAT vision phase defaults to GPT-5.6 Luna and can be overridden with `COMMANDCODE_E2E_GOAT_VISION_MODEL`. A successful live Anthropic `/provider/v1/messages` test requires a paid account whose plan includes the selected Claude model. +The Provider profile defaults to DeepSeek V4 Flash; GOAT defaults to Grok 4.6 because its Provider API stream exposes reasoning consistently across consecutive turns. Override them with `COMMANDCODE_E2E_GOAT_MODEL` or `COMMANDCODE_E2E_PROVIDER_MODEL`. The GOAT vision phase defaults to GPT-5.6 Luna and can be overridden with `COMMANDCODE_E2E_GOAT_VISION_MODEL`. A successful live Anthropic `/provider/v1/messages` test requires a paid account whose plan includes the selected Claude model. See [CONTRIBUTING.md](CONTRIBUTING.md) for local setup and tests. See [RELEASE.md](RELEASE.md) for the release process. diff --git a/index.ts b/index.ts index da485c6..234c4e8 100644 --- a/index.ts +++ b/index.ts @@ -18,8 +18,7 @@ import { join } from "node:path" import { getConfiguredApiKey } from "./src/api-key.ts" import { pickCommandCodeApiKey, withResolvedCommandCodeApiKey } from "./src/converters.ts" -import { createStreamCommandCode } from "./src/core.ts" -import { calculateCommandCodeCost, commandCodeCostRatesAt } from "./src/cost.ts" +import { commandCodeCostRatesAt } from "./src/cost.ts" import { apiForModelId, baseUrlForModel, @@ -41,7 +40,6 @@ import { registerCommandCodeQuota } from "./src/quota-command.ts" import { normalizeCommandCodeResponsesResponse } from "./src/responses-stream.ts" import { createCommandCodeRuntime } from "./src/runtime.ts" import { createCommandCodeUsageProvider, type UsageProvider } from "./src/usage.ts" -import { transcriptReadersFrom, withTranscriptPromptAndTools } from "./src/transcript.ts" import { createCommandCodeTransportRouter } from "./src/transport.ts" const COMMAND_CODE_API = "commandcode-custom" @@ -204,17 +202,8 @@ export default async function (pi: ExtensionAPI) { const modelsTimeoutMs = getModelsTimeoutMs() const modelsCachePath = process.env.COMMANDCODE_MODELS_CACHE ?? join(getAgentDir(), "commandcode-models.json") - const streamGenerate = createStreamCommandCode({ - createStream: () => new AssistantMessageEventStream(), - calculateCost: calculateCommandCodeCost, - apiBase: legacyApiBase(apiBase), - }) const resolveStreamOptions = (options?: Parameters[2]) => withResolvedCommandCodeApiKey(options, getConfiguredApiKey()) - // The provider transport delegates to pi-ai, which reads the transcript - // itself. The generate transport builds its own request body and needs the - // flat prompt and tool fields that pi 0.86+ no longer passes. - const transcriptReaders = transcriptReadersFrom(piAiCompat) // The host dispatches every Command Code model through one custom api id, so // the wire each model answers on is resolved here from the catalog that // `createProviderConfig` registered. Unknown ids fall back to the id rule. @@ -252,12 +241,6 @@ export default async function (pi: ExtensionAPI) { nativeOptions, ) }, - streamGenerate: (model, context, options) => - streamGenerate( - model, - withTranscriptPromptAndTools(context, transcriptReaders), - resolveStreamOptions(options), - ), }) // pi dispatches the main chat through the registered provider, but sibling diff --git a/package.json b/package.json index c9ae9bb..66825f4 100644 --- a/package.json +++ b/package.json @@ -59,10 +59,9 @@ "test:release": "node --test tests/test-release.mjs", "test:release-package": "node tests/test-release-package.mjs", "test:e2e:live": "node tests/test-live-e2e.mjs", - "test:e2e:live:go": "node scripts/live-e2e-profile.mjs go", "test:e2e:live:goat": "node scripts/live-e2e-profile.mjs goat", "test:e2e:live:provider": "node scripts/live-e2e-profile.mjs provider", - "test:e2e:live:all": "node scripts/live-e2e-profile.mjs go goat", + "test:e2e:live:all": "node scripts/live-e2e-profile.mjs goat provider", "test:cost": "tsx tests/test-cost.ts" }, "pi": { diff --git a/scripts/live-e2e-profile.mjs b/scripts/live-e2e-profile.mjs index b2cca8d..a1d4f2b 100644 --- a/scripts/live-e2e-profile.mjs +++ b/scripts/live-e2e-profile.mjs @@ -11,19 +11,16 @@ const profiles = process.argv.slice(2) if ( profiles.length === 0 || - profiles.some((profile) => profile !== "go" && profile !== "goat" && profile !== "provider") + profiles.some((profile) => profile !== "goat" && profile !== "provider") ) { - console.error("Usage: node scripts/live-e2e-profile.mjs [go|goat|provider]") + console.error( + "Usage: node scripts/live-e2e-profile.mjs [goat|provider] (Go has no Provider API access)", + ) process.exit(2) } async function credentialFor(profile) { - const prefix = - profile === "go" - ? "COMMANDCODE_E2E_GO" - : profile === "goat" - ? "COMMANDCODE_E2E_GOAT" - : "COMMANDCODE_E2E_PROVIDER" + const prefix = profile === "goat" ? "COMMANDCODE_E2E_GOAT" : "COMMANDCODE_E2E_PROVIDER" const direct = process.env[`${prefix}_API_KEY`]?.trim() const file = process.env[`${prefix}_API_KEY_FILE`] @@ -40,11 +37,7 @@ async function credentialFor(profile) { function runProfile(profile, apiKey) { const modelVariable = - profile === "go" - ? "COMMANDCODE_E2E_GO_MODEL" - : profile === "goat" - ? "COMMANDCODE_E2E_GOAT_MODEL" - : "COMMANDCODE_E2E_PROVIDER_MODEL" + profile === "goat" ? "COMMANDCODE_E2E_GOAT_MODEL" : "COMMANDCODE_E2E_PROVIDER_MODEL" const model = process.env[modelVariable] ?? (profile === "goat" ? "xai/grok-4.6" : "deepseek/deepseek-v4-flash") diff --git a/src/runtime.ts b/src/runtime.ts index afb0ba2..dde830e 100644 --- a/src/runtime.ts +++ b/src/runtime.ts @@ -30,14 +30,14 @@ export interface CommandCodeRuntimeOptions { /** Cached catalog only; resolves to an empty list when no valid cache exists. */ loadCachedModels: () => Promise createProviderConfig: (models: readonly CommandCodeModel[]) => TProviderConfig - getTransport?: () => "unknown" | "provider" | "generate" + getTransport?: () => "unknown" | "provider" getStreamWarning?: () => string | undefined now?: () => number logWarning?: (message: string) => void } export interface CommandCodeRuntimeStatus { - transport: "unknown" | "provider" | "generate" + transport: "unknown" | "provider" source: LoadCommandCodeModelsResult["source"] modelCount: number lastSuccess?: number diff --git a/src/transport.ts b/src/transport.ts index 50b70bb..e409da7 100644 --- a/src/transport.ts +++ b/src/transport.ts @@ -1,12 +1,14 @@ import type { - AssistantMessageEvent, AssistantMessageEventStreamLike, ContextLike, ModelLike, StreamOptions, } from "./types.ts" -export type CommandCodeTransport = "unknown" | "provider" | "generate" +export type CommandCodeTransport = "unknown" | "provider" + +export const PROVIDER_ACCESS_REQUIRED = + "Command Code Provider API access is required (403 upgrade_required). The Go plan does not include API use. Use an account/plan with documented Provider API access, then sign in again. This extension no longer falls back to the internal /alpha/generate endpoint. See https://commandcode.ai/docs/provider." interface TransportDependencies { createStream: () => AssistantMessageEventStreamLike @@ -15,11 +17,6 @@ interface TransportDependencies { context: ContextLike, options?: StreamOptions, ) => AssistantMessageEventStreamLike - streamGenerate: ( - model: ModelLike, - context: ContextLike, - options?: StreamOptions, - ) => AssistantMessageEventStreamLike } function isRecord(value: unknown): value is Record { @@ -28,7 +25,6 @@ function isRecord(value: unknown): value is Record { async function isUpgradeRequired(response: Response): Promise { if (response.status !== 403) return false - try { const body: unknown = await response.clone().json() if (!isRecord(body)) return false @@ -41,39 +37,17 @@ async function isUpgradeRequired(response: Response): Promise { export function createCommandCodeTransportRouter(deps: TransportDependencies) { let transport: CommandCodeTransport = "unknown" - let apiKey: string | undefined - - function pipe( - source: AssistantMessageEventStreamLike, - target: AssistantMessageEventStreamLike, - ): Promise { - return (async () => { - for await (const event of source) target.push(event) - })() - } - return { - getTransport(): CommandCodeTransport { - return transport - }, - + getTransport: (): CommandCodeTransport => transport, reset(): void { transport = "unknown" - apiKey = undefined }, - stream( model: ModelLike, context: ContextLike, options?: StreamOptions, ): AssistantMessageEventStreamLike { - if (options?.apiKey !== apiKey) { - apiKey = options?.apiKey - transport = "unknown" - } - const requestApiKey = options?.apiKey - if (transport === "generate") return deps.streamGenerate(model, context, options) - + transport = "provider" const output = deps.createStream() let upgradeRequired = false const fetchImpl = options?.fetch ?? fetch @@ -84,41 +58,24 @@ export function createCommandCodeTransportRouter(deps: TransportDependencies) { if (await isUpgradeRequired(response)) upgradeRequired = true return response }, - onResponse: async (response, responseModel) => { - if (upgradeRequired) return - await options?.onResponse?.(response, responseModel) - }, } - const run = async () => { - const providerStream = deps.streamProvider(model, context, providerOptions) - try { - for await (const event of providerStream) { - if (!upgradeRequired) { - if (apiKey === requestApiKey) transport = "provider" - output.push(event) - } + for await (const event of deps.streamProvider(model, context, providerOptions)) { + if (!upgradeRequired) output.push(event) } } catch (error) { - // Defense in depth for adapters whose iterators reject after HTTP errors; - // current Pi/OMP adapters emit error events instead. Only an inspected - // upgrade_required response permits generate fallback. + // Both emitted error events and rejected host iterators must yield + // the same entitlement message, never a retry on an internal API. if (!upgradeRequired) throw error } - - if (upgradeRequired) { - if (apiKey === requestApiKey) transport = "generate" - await pipe(deps.streamGenerate(model, context, options), output) - } + if (upgradeRequired) throw new Error(PROVIDER_ACCESS_REQUIRED) output.end() } - run().catch((error: unknown) => { - const message = error instanceof Error ? error.message : String(error) output.push({ type: "error", - reason: "error", + reason: options?.signal?.aborted ? "aborted" : "error", error: { role: "assistant", content: [], @@ -133,14 +90,17 @@ export function createCommandCodeTransportRouter(deps: TransportDependencies) { totalTokens: 0, cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 }, }, - stopReason: "error", - errorMessage: message, + stopReason: options?.signal?.aborted ? "aborted" : "error", + errorMessage: upgradeRequired + ? PROVIDER_ACCESS_REQUIRED + : error instanceof Error + ? error.message + : String(error), timestamp: Date.now(), }, }) output.end() }) - return output }, } diff --git a/tests/test-live-e2e.mjs b/tests/test-live-e2e.mjs index 7a91d88..0b1e9d2 100644 --- a/tests/test-live-e2e.mjs +++ b/tests/test-live-e2e.mjs @@ -26,20 +26,11 @@ const projectDir = resolve(dirname(fileURLToPath(import.meta.url)), "..") const extensionPath = join(projectDir, "index.ts") const testModel = process.env.COMMANDCODE_E2E_MODEL ?? "deepseek/deepseek-v4-flash" const testProfile = process.env.COMMANDCODE_E2E_PROFILE -const expectedTransport = - testProfile === "go" - ? "generate" - : testProfile === "goat" || testProfile === "provider" - ? "provider" - : undefined -const expectedPlan = - testProfile === "go" - ? "go" - : testProfile === "goat" - ? "goat" - : testProfile === "provider" - ? "provider" - : undefined +if (testProfile && !["goat", "provider"].includes(testProfile)) { + throw new Error("Live profiles require Provider API access; Go is not supported") +} +const expectedTransport = "provider" +const expectedPlan = testProfile // GPT-5.6 Luna is available on every plan and has a ZDR-capable upstream; // Gemini 3.7 Flash currently fails with a zero-data-retention routing 404. const goatVisionModel = process.env.COMMANDCODE_E2E_GOAT_VISION_MODEL ?? "gpt-5.6-luna" @@ -437,33 +428,6 @@ try { assert.doesNotMatch(vision.stderr, /Bearer\s+\S+/i) } - if (testProfile === "go") { - console.log("[live-e2e] image rejection through real RPC host") - const image = await runRpc(extensionPath, async ({ send, waitFor, events }) => { - send({ - id: "image", - type: "prompt", - message: "Describe this image", - images: [{ type: "image", data: "iVBORw0KGgo=", mimeType: "image/png" }], - }) - await waitFor((event) => event.type === "response" && event.id === "image") - await waitFor( - (event) => - event.type === "message_end" && - event.message?.role === "assistant" && - event.message?.stopReason === "error", - ) - return events - }) - assert.ok( - image.some( - (event) => - event.type === "message_end" && - /does not support image content/i.test(event.message?.errorMessage ?? ""), - ), - ) - } - console.log("[live-e2e] packed artifact with existing authentication") const packDir = join(tempRoot, "pack") mkdirSync(packDir, { recursive: true }) diff --git a/tests/test-omp-compat.mjs b/tests/test-omp-compat.mjs index 573b063..85d0d31 100644 --- a/tests/test-omp-compat.mjs +++ b/tests/test-omp-compat.mjs @@ -56,7 +56,7 @@ let lastRequestBody let requestBodies = [] let lastRequestHeaders = {} // When true the mock Provider API answers 403 upgrade_required so the -// transport router falls back to the legacy /alpha/generate transport. +// transport must reject access rather than falling back to /alpha/generate. let providerUpgradeRequired = false const server = createServer((req, res) => { @@ -403,10 +403,20 @@ try { assert.equal(requestCount, 0, JSON.stringify(lastRequestHeaders)) assert.doesNotMatch(noKeyChat.stdout + noKeyChat.stderr, /\$COMMAND_CODE_API_KEY/) - console.log("[omp-compat] developer advisory reaches the legacy generate request body") + console.log("[omp-compat] denied Provider API access never retries generate") + providerUpgradeRequired = true requestCount = 0 requestBodies = [] - providerUpgradeRequired = true + const denied = await runOmp([...chatArgs, "--no-title"]) + assert.match(denied.stdout + denied.stderr, /Provider API access is required/) + assert.doesNotMatch(denied.stdout, /mock-omp-ok/) + assert.equal(requestCount, 1) + assert.equal(requestBodies[0]?.params, undefined) + + console.log("[omp-compat] developer advisory reaches the Provider API request body") + requestCount = 0 + requestBodies = [] + providerUpgradeRequired = false const advisoryRun = await runOmp( [ "-e", @@ -426,12 +436,12 @@ try { assert.match(advisoryRun.stdout, /mock-omp-ok/) const promptBodies = requestBodies.filter((body) => - JSON.stringify(body?.params?.messages ?? []).includes("say mock token"), + JSON.stringify(body?.messages ?? []).includes("say mock token"), ) - assert.ok(promptBodies.length >= 1, "expected at least one generate request with the prompt") + assert.ok(promptBodies.length >= 1, "expected at least one provider request with the prompt") for (const body of promptBodies) { - const messages = body?.params?.messages ?? [] + const messages = body?.messages ?? [] const advisoryMessages = messages.filter((message) => JSON.stringify(message).includes("Stop and correct the benchmark."), ) @@ -455,7 +465,7 @@ try { "advisory must keep its chronological position relative to the prompt", ) assert.doesNotMatch( - String(body?.params?.system ?? ""), + JSON.stringify(messages.filter((message) => message.role === "system")), /Stop and correct the benchmark| tool.name === "read"), - `generate request must declare pi's tools, got ${JSON.stringify(generateTools)}`, - ) - assert.ok( - typeof lastRequestBody?.params?.system === "string" && lastRequestBody.params.system.length > 0, - "generate request must carry pi's system prompt", - ) - assert.ok( - (lastRequestBody?.params?.messages ?? []).every((message) => message.role !== "system"), - "system messages must not leak into the generate message list", - ) + assert.match(deniedPrint.stdout + deniedPrint.stderr, /Provider API access is required/) + assert.doesNotMatch(deniedPrint.stdout, /mock-pi-ok/) + assert.equal(requestCount, 1) + assert.equal(lastRequestPath, "/provider/v1/chat/completions") // pi resolves `/login` credentials, `--api-key`, and env keys through the // provider's registered auth methods. Stored credentials and `--api-key` diff --git a/tests/test-transport.ts b/tests/test-transport.ts index 7da21e0..67ea4bd 100644 --- a/tests/test-transport.ts +++ b/tests/test-transport.ts @@ -2,7 +2,7 @@ import assert from "node:assert/strict" import { describe, it } from "node:test" import { normalizeCommandCodeResponsesResponse } from "../src/responses-stream.ts" -import { createCommandCodeTransportRouter } from "../src/transport.ts" +import { createCommandCodeTransportRouter, PROVIDER_ACCESS_REQUIRED } from "../src/transport.ts" import type { AssistantMessageEvent, AssistantMessageEventStreamLike, @@ -62,275 +62,151 @@ function providerStream( return stream } -describe("Command Code transport router", () => { - it("keeps using the Provider API after a successful request", async () => { - let providerCalls = 0 - let generateCalls = 0 +describe("Command Code Provider API entitlement", () => { + it("uses the provider and preserves successful events", async () => { + let calls = 0 const router = createCommandCodeTransportRouter({ createStream: createTestEventStream, - streamProvider: (_model, _context, options) => { - providerCalls += 1 - return providerStream(new Response("ok", { status: 200 }), "provider", options) - }, - streamGenerate: () => { - generateCalls += 1 - return completedStream("generate") + streamProvider: () => { + calls += 1 + return completedStream("provider") }, }) - - const options: StreamOptions = { - fetch: () => Promise.resolve(new Response("ok", { status: 200 })), + assert.equal(router.getTransport(), "unknown") + for (let index = 0; index < 2; index++) { + const events = await collectEvents(router.stream(makeModel(), makeContext())) + assert.equal(events.at(-1)?.type, "done") } - const first = await collectEvents(router.stream(makeModel(), makeContext(), options)) - const second = await collectEvents(router.stream(makeModel(), makeContext(), options)) - - assert.equal(first.at(-1)?.type, "done") - assert.equal(second.at(-1)?.type, "done") + assert.equal(calls, 2) assert.equal(router.getTransport(), "provider") - assert.equal(providerCalls, 2) - assert.equal(generateCalls, 0) - }) - - it("falls back only for 403 upgrade_required and remembers generate", async () => { - let providerCalls = 0 - let generateCalls = 0 - const responseBody = JSON.stringify({ - error: { code: "upgrade_required", type: "permission_error" }, - }) - const router = createCommandCodeTransportRouter({ - createStream: createTestEventStream, - streamProvider: (_model, _context, options) => { - providerCalls += 1 - return providerStream(new Response(responseBody, { status: 403 }), "blocked", options) - }, - streamGenerate: () => { - generateCalls += 1 - return completedStream("generate") - }, - }) - const options: StreamOptions = { - fetch: () => Promise.resolve(new Response(responseBody, { status: 403 })), - } - - const first = await collectEvents(router.stream(makeModel(), makeContext(), options)) - const second = await collectEvents(router.stream(makeModel(), makeContext(), options)) - - assert.equal(first.at(-1)?.type, "done") - assert.equal(second.at(-1)?.type, "done") - assert.equal(router.getTransport(), "generate") - assert.equal(providerCalls, 1) - assert.equal(generateCalls, 2) + router.reset() + assert.equal(router.getTransport(), "unknown") }) - it("re-detects the transport after the API key changes", async () => { - let providerCalls = 0 - let generateCalls = 0 - const upgradeBody = JSON.stringify({ error: { code: "upgrade_required" } }) - const router = createCommandCodeTransportRouter({ - createStream: createTestEventStream, - streamProvider: (_model, _context, options) => { - providerCalls += 1 - const response = - options?.apiKey === "go-key" - ? new Response(upgradeBody, { status: 403 }) - : new Response("ok", { status: 200 }) - return providerStream(response, "provider", options) - }, - streamGenerate: () => { - generateCalls += 1 - return completedStream("generate") - }, + for (const rejects of [false, true]) { + it(`rejects upgrade_required without fallback (iterator rejects=${rejects})`, async () => { + let requests = 0 + const statuses: number[] = [] + const router = createCommandCodeTransportRouter({ + createStream: createTestEventStream, + streamProvider: (_model, _context, options) => { + const stream = createTestEventStream() + stream[Symbol.asyncIterator] = async function* () { + const response = await (options?.fetch ?? fetch)("https://provider.test", {}) + await options?.onResponse?.({ status: response.status, headers: {} }, makeModel()) + yield* completedStream("blocked-provider-output") + if (rejects) throw new Error("raw-private-provider-error") + } + return stream + }, + }) + for (const apiKey of ["go-key", "go-key", "new-key"]) { + const events = await collectEvents( + router.stream(makeModel(), makeContext(), { + apiKey, + fetch: async () => { + requests += 1 + return new Response( + JSON.stringify({ + error: { code: "upgrade_required", message: "raw-private-provider-error" }, + }), + { status: 403 }, + ) + }, + onResponse: async (response) => { + statuses.push(response.status) + }, + }), + ) + assert.deepEqual( + events.map((event) => event.type), + ["error"], + ) + const terminal = events.at(-1) + assert.ok(terminal?.type === "error") + assert.equal(terminal.error.errorMessage, PROVIDER_ACCESS_REQUIRED) + assert.doesNotMatch( + JSON.stringify(events), + /raw-private-provider-error|blocked-provider-output/, + ) + } + assert.equal( + requests, + 3, + "exactly one provider request per stream; no fallback or cached bypass", + ) + assert.deepEqual(statuses, [403, 403, 403], "diagnostics still see denied responses") + assert.equal(router.getTransport(), "provider") }) + } - await collectEvents( - router.stream(makeModel(), makeContext(), { - apiKey: "go-key", - fetch: () => Promise.resolve(new Response(upgradeBody, { status: 403 })), - }), - ) - await collectEvents( - router.stream(makeModel(), makeContext(), { - apiKey: "provider-key", - fetch: () => Promise.resolve(new Response("ok", { status: 200 })), - }), - ) - - assert.equal(router.getTransport(), "provider") - assert.equal(providerCalls, 2) - assert.equal(generateCalls, 1) + it("does not rewrite unrelated errors or malformed entitlement responses", async () => { + for (const [status, body] of [ + [401, '{"error":{"code":"upgrade_required"}}'], + [403, '{"error":{"code":"permission_denied"}}'], + [403, "invalid JSON"], + [429, "limited"], + ]) { + const router = createCommandCodeTransportRouter({ + createStream: createTestEventStream, + streamProvider: (_model, _context, options) => { + const stream = createTestEventStream() + stream[Symbol.asyncIterator] = async function* () { + await (options?.fetch ?? fetch)("https://provider.test", {}) + throw new Error("original host error") + } + return stream + }, + }) + assert.ok(typeof status === "number" && typeof body === "string") + const events = await collectEvents( + router.stream(makeModel(), makeContext(), { + fetch: async () => new Response(body, { status }), + }), + ) + const terminal = events.at(-1) + assert.ok(terminal?.type === "error") + assert.equal(terminal.error.errorMessage, "original host error") + } }) - it("does not let a stale request overwrite the transport for a new API key", async () => { - let releaseGoRequest: (() => void) | undefined - const goRequestGate = new Promise((resolve) => { - releaseGoRequest = resolve - }) - let providerCalls = 0 - let generateCalls = 0 - const upgradeBody = JSON.stringify({ error: { code: "upgrade_required" } }) + it("does not cache an entitlement denial after the key changes", async () => { const router = createCommandCodeTransportRouter({ createStream: createTestEventStream, - streamProvider: (_model, _context, options) => { - providerCalls += 1 - const response = - options?.apiKey === "go-key" - ? new Response(upgradeBody, { status: 403 }) - : new Response("ok", { status: 200 }) - const stream = createTestEventStream() - const run = async () => { - if (options?.apiKey === "go-key") await goRequestGate - const received = await (options?.fetch ?? fetch)("https://provider.test", {}) - await options?.onResponse?.( - { status: received.status, headers: {} }, - makeModel({ api: "openai-completions" }), - ) - if (response.ok) { - for await (const event of completedStream("provider")) stream.push(event) - } - stream.end() - } - run().catch(() => stream.end()) - return stream - }, - streamGenerate: () => { - generateCalls += 1 - return completedStream("generate") - }, + streamProvider: (_model, _context, options) => + providerStream(new Response(), "provider", options), }) - - const staleGoRequest = collectEvents( + const denied = await collectEvents( router.stream(makeModel(), makeContext(), { apiKey: "go-key", - fetch: () => Promise.resolve(new Response(upgradeBody, { status: 403 })), + fetch: async () => new Response('{"code":"upgrade_required"}', { status: 403 }), }), ) - await collectEvents( + assert.equal(denied.at(-1)?.type, "error") + const accepted = await collectEvents( router.stream(makeModel(), makeContext(), { apiKey: "provider-key", - fetch: () => Promise.resolve(new Response("ok", { status: 200 })), + fetch: async () => new Response("ok"), }), ) - releaseGoRequest?.() - await staleGoRequest - await collectEvents( - router.stream(makeModel(), makeContext(), { - apiKey: "provider-key", - fetch: () => Promise.resolve(new Response("ok", { status: 200 })), - }), - ) - - assert.equal(router.getTransport(), "provider") - assert.equal(providerCalls, 3) - assert.equal(generateCalls, 1) - }) - - it("falls back when a Provider stream throws after an upgrade_required 403", async () => { - const upgradeBody = JSON.stringify({ error: { code: "upgrade_required" } }) - let providerCalls = 0 - let generateCalls = 0 - const router = createCommandCodeTransportRouter({ - createStream: createTestEventStream, - streamProvider: (_model, _context, options) => { - providerCalls += 1 - const stream = createTestEventStream() - stream[Symbol.asyncIterator] = async function* () { - const response = await (options?.fetch ?? fetch)("https://provider.test", {}) - await options?.onResponse?.( - { status: response.status, headers: {} }, - makeModel({ api: "openai-completions" }), - ) - // Even events emitted before the iterator rejects must stay suppressed. - yield* completedStream("blocked-provider-output") - throw new Error("Provider stream rejected after 403") - } - return stream - }, - streamGenerate: () => { - generateCalls += 1 - return completedStream("generate") - }, - }) - const options: StreamOptions = { - apiKey: "go-key", - fetch: () => Promise.resolve(new Response(upgradeBody, { status: 403 })), - } - - const first = await collectEvents(router.stream(makeModel(), makeContext(), options)) - const second = await collectEvents(router.stream(makeModel(), makeContext(), options)) - - for (const events of [first, second]) { - assert.deepEqual( - events.map((event) => event.type), - ["start", "text_start", "text_delta", "text_end", "done"], - ) - assert.deepEqual( - events.flatMap((event) => (event.type === "text_delta" ? [event.delta] : [])), - ["generate"], - ) - const terminal = events.at(-1) - assert.ok(terminal?.type === "done") - assert.deepEqual(terminal.message.content, [{ type: "text", text: "generate" }]) - } - assert.equal(router.getTransport(), "generate") - assert.equal(providerCalls, 1) - assert.equal(generateCalls, 2) + assert.equal(accepted.at(-1)?.type, "done") }) - it("does not fall back when a Provider stream rejects after another 403", async () => { - let generateCalls = 0 + it("preserves abort classification without a second request", async () => { + const controller = new AbortController() + controller.abort() const router = createCommandCodeTransportRouter({ createStream: createTestEventStream, - streamProvider: (_model, _context, options) => { - const stream = createTestEventStream() - stream[Symbol.asyncIterator] = async function* () { - await (options?.fetch ?? fetch)("https://provider.test", {}) - throw new Error("Provider stream rejected on denied request") - } - return stream - }, - streamGenerate: () => { - generateCalls += 1 - return completedStream("generate") + streamProvider: () => { + throw new Error("aborted") }, }) const events = await collectEvents( - router.stream(makeModel(), makeContext(), { - fetch: () => - Promise.resolve( - new Response(JSON.stringify({ error: { code: "permission_denied" } }), { status: 403 }), - ), - }), - ) - assert.deepEqual( - events.map((event) => event.type), - ["error"], + router.stream(makeModel(), makeContext(), { signal: controller.signal }), ) const terminal = events.at(-1) assert.ok(terminal?.type === "error") - assert.equal(terminal.error.errorMessage, "Provider stream rejected on denied request") - assert.equal(generateCalls, 0) - }) - - it("does not fall back for other 403 errors", async () => { - let generateCalls = 0 - const responseBody = JSON.stringify({ error: { code: "permission_denied" } }) - const router = createCommandCodeTransportRouter({ - createStream: createTestEventStream, - streamProvider: (_model, _context, options) => - providerStream(new Response(responseBody, { status: 403 }), "blocked", options), - streamGenerate: () => { - generateCalls += 1 - return completedStream("generate") - }, - }) - const options: StreamOptions = { - fetch: () => Promise.resolve(new Response(responseBody, { status: 403 })), - } - - await collectEvents(router.stream(makeModel(), makeContext(), options)) - - assert.equal(router.getTransport(), "provider") - assert.equal(generateCalls, 0) + assert.equal(terminal.reason, "aborted") }) })