From 39129109cc63e412bedd9cf9fd9535f1be02ac71 Mon Sep 17 00:00:00 2001 From: codingbo Date: Sun, 27 Sep 2026 02:02:00 +0900 Subject: [PATCH 01/19] feat(proxy): support maxConcurrentRequests in requestPacing at provider and model level (#5954) Carried from #5954 as one squashed commit. Co-authored-by: codingbo Co-authored-by: Brad Hallett <53977268+bradhallett@users.noreply.github.com> --- .../fr/reference/configuration/providers.md | 2 +- .../ja/reference/configuration/providers.md | 2 +- .../ko/reference/configuration/providers.md | 2 +- .../docs/reference/configuration/providers.md | 20 +++- .../ru/reference/configuration/providers.md | 2 +- .../reference/configuration/providers.md | 2 +- .../reference/configuration/providers.md | 2 +- src/config/schema/leaf-validators.ts | 11 ++- src/providers/request-pacing.ts | 60 ++++++++++-- src/server/responses/fetch-helpers.ts | 10 +- src/types/provider.ts | 2 + structure/config.md | 2 +- structure/providers-and-adapters.md | 8 ++ .../management-provider-validation.test.ts | 29 ++++++ tests/usage/request-pacing.test.ts | 93 +++++++++++++++++++ 15 files changed, 222 insertions(+), 25 deletions(-) diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index a847f6072a6..710e3d2d710 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -86,7 +86,7 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | --- | --- | --- | | `adapter` | `string` | L'un des `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (ou alias `azure`). | | `baseUrl` | `string` | URL de base de l'API en amont. La plupart des points de terminaison fixes intégrés ignorent une valeur incompatible ; les préréglages de clés protégés contre les collisions préservent une ancienne destination personnalisée portant le même nom. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, models? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. Le nombre de requêtes par minute est converti en intervalle régulier ; `minIntervalMs` peut imposer un intervalle plus long. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont, par exemple `nvidia/llama-3.1-nemotron-ultra-253b-v1`, et ne peuvent qu’ajouter du délai. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP, Responses WebSocket et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP, Responses WebSocket et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. | | `responsesPath?` | `string` | Chemin de ressource relatif pour les requêtes d'authentification par clé `openai-responses`. Il doit commencer par `/` et ne contenir aucun schéma, requête ou fragment. | | `chatCompletionsPath?` | `string` | Chemin de ressource relatif pour les requêtes `openai-chat`, miroir de `responsesPath` et soumis aux mêmes règles de forme. Nécessaire lorsqu'un même service en amont sert Chat Completions et Responses sous des préfixes différents : un override wire par modèle change l'adaptateur sans toucher `baseUrl`, donc sans ce réglage une requête Chat activée serait envoyée vers la base Responses. L'exemple fourni est Z.AI. | | `upstreamWebsocket?` | `boolean` | Active le transport Responses WebSocket en amont pour les requêtes `openai-responses` (désactivé par défaut). N'est honoré que pour l'amont first-party `https://api.openai.com/v1` ; les points de terminaison des fournisseurs personnalisés utilisent toujours HTTP/SSE borné, car Bun ne peut pas appliquer de limite de taille aux messages WebSocket entrants avant d'avoir alloué le message complet. Pour le fournisseur canonique ChatGPT `openai`, l'omettre conserve le WebSocket en amont sur les tours éligibles, `false` envoie les tours en streaming via HTTP/SSE, et `true` est refusé ; avec `false`, le pilotage et l'injection natifs en cours de tour sont indisponibles. Ce champ est indépendant du réglage `websockets` côté client et ne change ni le point de terminaison ni les identifiants. Une base HTTP reste en SSE ; les chemins qui ne sont pas Responses et les requêtes `openai-chat` restent en HTTP. | diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index b3222d502f9..95c68025cb9 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -80,7 +80,7 @@ account を削除しても mapping は保持され、同じ id を再追加す | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai` (または別名 `azure`) のいずれか。 | | `baseUrl` | `string` |アップストリーム API のベース URL。ほとんどの組み込み固定エンドポイントは不一致を無視します。衝突安全キー プリセットは、古い同じ名前のカスタム宛先を保持します。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, models? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やす場合のみ有効です。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP、Responses WebSocket、明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP、Responses WebSocket、明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。 | | `responsesPath?` | `string` |キー認証 `openai-responses` リクエストの相対リソース パス。 `/` で始まり、スキーム、クエリ、またはフラグメントが含まれていない必要があります。 | | `chatCompletionsPath?` | `string` | `openai-chat` リクエストの相対リソース パス。 `responsesPath` の対となる設定で、同じ形式ルールが適用されます。1つのアップストリームが Chat Completions と Responses を異なるプレフィックスで提供する場合に必要です。モデルごとの wire override はアダプターのみを変更し `baseUrl` は変更しないため、この設定がないと有効化された Chat リクエストが Responses ベースへ送信されます。同梱例は Z.AI です。 | | `upstreamWebsocket?` | `boolean` | `openai-responses` リクエストで使用するアップストリーム Responses WebSocket トランスポート（既定値は無効）。ファーストパーティの `https://api.openai.com/v1` アップストリームでのみ有効です。カスタムプロバイダーのエンドポイントは常に制限付き HTTP/SSE を使用します。Bun はメッセージ全体を確保する前に受信 WebSocket メッセージのサイズ上限を適用できないためです。正規の ChatGPT `openai` プロバイダーでは、省略すると対象となるターンでアップストリーム WebSocket を使用し、`false` はストリーミングのターンを HTTP/SSE で送信し、`true` は拒否されます。`false` の間はネイティブのターン途中のステアリングとインジェクションを利用できません。このフィールドはクライアント側の `websockets` 設定とは独立しており、エンドポイントと認証情報のどちらも変更しません。HTTP のベース URL は SSE のままとなり、Responses 以外のパスと `openai-chat` リクエストは HTTP を使用します。 | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index c4eb5809204..71ff339c1fd 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -80,7 +80,7 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | --- | --- | --- | | `adapter` | `string` | `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` 중 하나이며, `azure`는 별칭입니다. | | `baseUrl` | `string` | 상위 API 기본 URL입니다. 대부분의 내장 고정 엔드포인트는 불일치를 무시합니다. 충돌 안전 키 프리셋은 같은 이름의 이전 사용자 지정 목적지를 보존합니다. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, models? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 더 늘릴 때만 적용됩니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP, Responses WebSocket, 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP, Responses WebSocket, 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. | | `responsesPath?` | `string` | 키 인증 `openai-responses` 요청의 상대 리소스 경로입니다. 반드시 `/`로 시작해야 하며 스킴, query, fragment를 포함하면 안 됩니다. | | `chatCompletionsPath?` | `string` | `openai-chat` 요청의 상대 리소스 경로로, `responsesPath`와 동일한 형식 규칙이 적용되는 대응 항목입니다. 하나의 업스트림이 Chat Completions와 Responses를 서로 다른 접두사로 제공할 때 필요합니다. 모델별 wire override는 어댑터만 바꾸고 `baseUrl`은 그대로 두므로, 이 설정이 없으면 옵트인된 Chat 요청이 Responses base로 전송됩니다. Z.AI가 제공되는 예시입니다. | | `upstreamWebsocket?` | `boolean` | `openai-responses` 요청에 대한 업스트림 Responses WebSocket 전송을 선택적으로 활성화합니다(기본값 `false`). 퍼스트파티 `https://api.openai.com/v1` 업스트림에서만 적용되며, 사용자 지정 공급자 엔드포인트는 항상 제한된 HTTP/SSE를 사용합니다. Bun은 전체 메시지를 할당하기 전에는 수신 WebSocket 메시지 크기 제한을 적용할 수 없기 때문입니다. 정식 ChatGPT `openai` 공급자에서는 생략하면 대상 턴에서 업스트림 WebSocket을 사용하고, `false`는 스트리밍 턴을 HTTP/SSE로 전송하며, `true`는 거부됩니다. `false`이면 네이티브 턴 중 스티어링과 주입을 사용할 수 없습니다. 이 필드는 클라이언트 측 `websockets` 설정과 독립적이며 엔드포인트와 자격 증명을 변경하지 않습니다. HTTP 기본 URL은 SSE를 유지하고, Responses가 아닌 경로와 `openai-chat` 요청은 HTTP를 사용합니다. | diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index e45d68b4bbb..e91d9b581dd 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -198,7 +198,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `baseUrl` | `string` | Upstream API base URL. Most built-in fixed endpoints ignore a mismatch; collision-safe key presets preserve an older same-named custom destination. | | `proxy?` | `string \| null` | Per-provider egress route. Omit it to inherit the global proxy decision; use `"direct"` or `null` to force direct egress; or provide an absolute `http://`, `https://`, `socks5://`, or `socks5h://` proxy URL. An empty string is rejected. | | `noProxy?` | `string \| string[]` | Destinations this provider reaches directly, using `NO_PROXY` host-pattern syntax. A match bypasses both this provider's own proxy and an inherited global proxy. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, models? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. Provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay. Queue waits do not consume the upstream response-header timeout. HTTP, Responses WebSocket, and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP, Responses WebSocket, and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. | | `upstreamHttpVersion?` | `"auto" \| "http1.1" \| "h1" \| "http2" \| "h2"` | Pin the HTTP version used for upstream requests to this provider. Defaults to `auto`, which lets Bun negotiate. An explicit pin requires an HTTPS target and fails locally when it cannot be honored. Set `http1.1` when a provider's HTTP/2 SSE stream stalls instead of delivering events — the symptom is a long-running streaming request that produces nothing and eventually times out. For Cursor, `http1.1`/`h1` selects its `RunSSE` + `BidiAppend` compatibility transport for inference and also pins live model discovery. Management `POST`/`PATCH` accept `null` to clear it back to `auto`. | | `responsesPath?` | `string` | Relative resource path for key-auth `openai-responses` requests. It must start with `/` and contain no scheme, query, or fragment. | | `chatCompletionsPath?` | `string` | Relative resource path for `openai-chat` requests, the mirror of `responsesPath` and subject to the same shape rules. Needed when one upstream serves Chat Completions and Responses under different prefixes: a per-model wire override changes the adapter and leaves `baseUrl` alone, so without this an opted-in Chat request would be sent to the Responses base. Z.AI is the shipped example. | @@ -294,6 +294,24 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity Provider registration and replacement (`POST /api/providers`) validate `responsesPath` and `chatCompletionsPath` before changing live configuration or disk state. `PATCH /api/providers?name=` merges the request body with the stored provider; updates touching fields beyond `disabled` — except `requestPacing`-only updates — validate the merged provider's paths the same way before saving, and an invalid retained path returns `400` with the configuration unchanged. The same path rules apply when loading a configuration file. +For example, this applies a provider-wide concurrency cap and a stricter cap to one exact model, without configuring an interval: + +```json +{ + "requestPacing": { + "enabled": true, + "maxConcurrentRequests": 8, + "models": { + "nvidia/llama-3.1-nemotron-ultra-253b-v1": { + "maxConcurrentRequests": 2 + } + } + } +} +``` + +The concurrency slot stays occupied until the upstream request finishes, including the final streamed response bytes. Requests above the applicable provider and model limits wait in the pacing queue; when a slot is released, the next eligible request is admitted. Queue waiting does not consume the upstream response-header timeout. + ### What a provider save keeps `POST /api/providers` with the name of an existing provider replaces the stored row with one built from the request. The dashboard's add/edit form cannot send every field, so the save keeps some stored fields the request omits. Five of them record how one upstream behaves: `preserveReasoningContentModels`, `requiresReasoningPlaceholderModels`, `foldDeveloperRoleToSystem`, `reasoningWireFormat` and `omitReasoningEffortWithToolsModels`. diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index db16f8502d3..51a640474ac 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -93,7 +93,7 @@ cross-route credential fallback не существует. Строки API GPT- | --- | --- | --- | | `adapter` | `string` | Один из `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (или alias `azure`). | | `baseUrl` | `string` | Базовый URL API upstream'а. Большинство built-in fixed-endpoint'ов игнорируют несовпадение; collision-safe key-preset'ы сохраняют старый custom destination с тем же именем. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, models? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может только увеличить задержку. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP, Responses WebSocket и явные вызовы адаптеров `fetchResponse`/`runTurn`. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP, Responses WebSocket и явные вызовы адаптеров `fetchResponse`/`runTurn`. | | `responsesPath?` | `string` | Relative resource path для key-auth запросов `openai-responses`. Должен начинаться с `/` и не может содержать scheme, query или fragment. | | `chatCompletionsPath?` | `string` | Relative resource path для запросов `openai-chat`, зеркало `responsesPath` с теми же правилами формы. Нужен, когда один upstream обслуживает Chat Completions и Responses под разными префиксами: per-model wire override меняет адаптер и не трогает `baseUrl`, поэтому без него включённый Chat-запрос ушёл бы в Responses base. Поставляемый пример — Z.AI. | | `upstreamWebsocket?` | `boolean` | Необязательный upstream Responses WebSocket для запросов `openai-responses` (по умолчанию `false`). Учитывается только для first-party upstream `https://api.openai.com/v1`; конечные точки пользовательских провайдеров всегда используют ограниченный HTTP/SSE, поскольку Bun не может применить ограничение размера входящего сообщения WebSocket до выделения памяти под всё сообщение. У канонического провайдера ChatGPT `openai` пропуск сохраняет upstream WebSocket для подходящих ходов, `false` отправляет потоковые ходы по HTTP/SSE, а `true` отклоняется; при `false` нативное управление и внедрение в середине хода недоступны. Это поле не зависит от клиентской настройки `websockets` и не меняет ни конечную точку, ни учётные данные. Для HTTP остаётся SSE; пути, не относящиеся к Responses, и запросы `openai-chat` остаются на HTTP. | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index 7f9565b7bbf..2480a5f2ca6 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -80,7 +80,7 @@ selector，而不是分配一个新名称。 | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或别名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API 基础 URL。大多数内置固定端点会忽略不匹配的值；具备冲突安全键的预设会保留一个更早、同名的自定义目标。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, models? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配且只能增加延迟。排队等待不计入响应头超时。覆盖 HTTP、Responses WebSocket 以及显式适配器 `fetchResponse`/`runTurn` 调用。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP、Responses WebSocket 以及显式适配器 `fetchResponse`/`runTurn` 调用。 | | `responsesPath?` | `string` | 用于 key-auth `openai-responses` 请求的相对资源路径。必须以 `/` 开头，且不能包含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | 用于 `openai-chat` 请求的相对资源路径，是 `responsesPath` 的对应项，适用相同的路径规则。当同一上游以不同前缀提供 Chat Completions 和 Responses 时需要此配置：按模型的 wire override 只更换适配器而不改动 `baseUrl`，否则已启用的 Chat 请求会被发送到 Responses base。随附示例为 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 为 `openai-responses` 请求选择性启用上游 Responses WebSocket 传输（默认 `false`）。仅对第一方 `https://api.openai.com/v1` 上游生效；自定义提供者端点始终使用有界 HTTP/SSE，因为 Bun 无法在分配完整消息之前对入站 WebSocket 消息实施大小限制。对于规范 ChatGPT `openai` 提供商，省略该字段会在符合条件的轮次使用上游 WebSocket，`false` 通过 HTTP/SSE 发送流式轮次，`true` 会被拒绝；设为 `false` 时，原生轮次中操控与注入不可用。该字段独立于客户端侧的 `websockets` 设置，且不改变端点或凭据。普通 HTTP 仍使用 SSE；非 Responses 路径和 `openai-chat` 请求仍使用 HTTP。 | diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index 9df50bc2bf2..eb8fb5cd79e 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -60,7 +60,7 @@ ocx models provider openrouter on | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或別名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API base URL。多數內建固定端點忽略不符；碰撞安全的金鑰預設保留較舊的同名自訂目的地。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, models? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對且只能增加延遲。排隊等待不計入回應標頭逾時。涵蓋 HTTP、Responses WebSocket 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP、Responses WebSocket 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。 | | `responsesPath?` | `string` | Key-auth `openai-responses` 請求的相對資源路徑。必須以 `/` 開頭且不含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | `openai-chat` 請求的相對資源路徑，為 `responsesPath` 的對應項，適用相同的路徑規則。當同一上游以不同前綴提供 Chat Completions 與 Responses 時需要此設定：按模型的 wire override 只更換適配器而不改動 `baseUrl`，否則已啟用的 Chat 請求會送往 Responses base。隨附範例為 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 為 `openai-responses` 請求選用上游 Responses WebSocket 傳輸（預設 `false`）。僅對第一方 `https://api.openai.com/v1` 上游生效；自訂供應商端點一律使用有界 HTTP/SSE，因為 Bun 無法在配置完整訊息之前對傳入 WebSocket 訊息套用大小限制。對於規範 ChatGPT `openai` 供應商，省略此欄位會在符合條件的回合使用上游 WebSocket，`false` 會以 HTTP/SSE 傳送串流回合，`true` 會被拒絕；設為 `false` 時，原生回合中操控與注入無法使用。此欄位獨立於用戶端 `websockets` 設定，且不會變更端點或認證資料。一般 HTTP 仍使用 SSE；非 Responses 路徑與 `openai-chat` 請求仍使用 HTTP。 | diff --git a/src/config/schema/leaf-validators.ts b/src/config/schema/leaf-validators.ts index fcd92f4fc17..1ba5049037e 100644 --- a/src/config/schema/leaf-validators.ts +++ b/src/config/schema/leaf-validators.ts @@ -97,18 +97,23 @@ const requestPacingRuleSchema = z.object({ // Keep the RPM-derived timer within the same one-hour bound as minIntervalMs. requestsPerMinute: z.number().min(1 / 60).max(60_000).optional(), minIntervalMs: z.number().int().min(1).max(3_600_000).optional(), -}).strict().refine(value => value.requestsPerMinute !== undefined || value.minIntervalMs !== undefined, { - message: "request pacing rules need requestsPerMinute or minIntervalMs", + maxConcurrentRequests: z.number().int().min(1).optional(), +}).strict().refine(value => value.requestsPerMinute !== undefined + || value.minIntervalMs !== undefined + || value.maxConcurrentRequests !== undefined, { + message: "request pacing rules need requestsPerMinute, minIntervalMs, or maxConcurrentRequests", }); const requestPacingSchema = z.object({ enabled: z.boolean(), requestsPerMinute: z.number().min(1 / 60).max(60_000).optional(), minIntervalMs: z.number().int().min(1).max(3_600_000).optional(), + maxConcurrentRequests: z.number().int().min(1).optional(), models: z.record(z.string().trim().min(1), requestPacingRuleSchema).optional(), }).strict().refine(value => value.enabled === false || value.requestsPerMinute !== undefined || value.minIntervalMs !== undefined + || value.maxConcurrentRequests !== undefined || (value.models !== undefined && Object.keys(value.models).length > 0), { message: "enabled request pacing needs a provider rule or model override", }); @@ -117,7 +122,7 @@ export function requestPacingConfigError(value: unknown): string | null { if (value === undefined) return null; const parsed = requestPacingSchema.safeParse(value); if (parsed.success) return null; - return "requestPacing must contain enabled and a valid requestsPerMinute/minIntervalMs provider rule or model overrides"; + return "requestPacing must contain enabled and a valid requestsPerMinute/minIntervalMs/maxConcurrentRequests provider rule or model overrides"; } /** diff --git a/src/providers/request-pacing.ts b/src/providers/request-pacing.ts index e38c62f3c09..e6181933a31 100644 --- a/src/providers/request-pacing.ts +++ b/src/providers/request-pacing.ts @@ -35,13 +35,17 @@ interface Waiter { modelIntervalMs: number; queuedAt: number; signal?: AbortSignal; - resolve: () => void; + providerMaxConcurrent: number; + modelMaxConcurrent: number; + resolve: (release: () => void) => void; reject: (reason: unknown) => void; abort?: () => void; } interface ProviderPacer { queue: Waiter[]; + inFlight: number; + modelInFlight: Map; providerNextStartAt: number; modelNextStartAt: Map; timer?: unknown; @@ -109,6 +113,33 @@ function requestPacingIntervals(provider: OcxProviderConfig, modelId?: string): }; } +function concurrencyAvailable(state: ProviderPacer, waiter: Waiter): boolean { + return state.inFlight < waiter.providerMaxConcurrent + && (state.modelInFlight.get(waiter.modelId ?? "") ?? 0) < waiter.modelMaxConcurrent; +} + +function acquireConcurrency(providerName: string, state: ProviderPacer, waiter: Waiter): () => void { + const model = waiter.modelId ?? ""; + state.inFlight += 1; + state.modelInFlight.set(model, (state.modelInFlight.get(model) ?? 0) + 1); + let released = false; + const release = () => { + if (released) return; + released = true; + waiter.signal?.removeEventListener("abort", release); + state.inFlight -= 1; + const remaining = (state.modelInFlight.get(model) ?? 1) - 1; + if (remaining > 0) state.modelInFlight.set(model, remaining); + else state.modelInFlight.delete(model); + if (state.timer) runtime.clearTimer(state.timer); + state.timer = undefined; + runQueue(providerName, state); + }; + waiter.signal?.addEventListener("abort", release, { once: true }); + if (waiter.signal?.aborted) release(); + return release; +} + function waiterReadyAt(state: ProviderPacer, modelId: string | undefined): number { return Math.max( state.providerNextStartAt, @@ -163,13 +194,14 @@ function runQueue(providerName: string, state: ProviderPacer): void { const providerReadyAt = Math.max(now, state.providerNextStartAt); const waiterIndex = state.queue.findIndex(waiter => { const modelReadyAt = waiter.modelId ? (state.modelNextStartAt.get(waiter.modelId) ?? 0) : 0; - return Math.max(providerReadyAt, modelReadyAt) <= now; + return concurrencyAvailable(state, waiter) && Math.max(providerReadyAt, modelReadyAt) <= now; }); if (waiterIndex < 0) { let earliestAt = Number.POSITIVE_INFINITY; for (const waiter of state.queue) { const modelReadyAt = waiter.modelId ? (state.modelNextStartAt.get(waiter.modelId) ?? 0) : 0; - const readyAt = Math.max(providerReadyAt, modelReadyAt); + const readyAt = concurrencyAvailable(state, waiter) + ? Math.max(providerReadyAt, modelReadyAt) : Number.POSITIVE_INFINITY; const expiresAt = waiter.queuedAt + maxQueueAgeMs; earliestAt = Math.min(earliestAt, readyAt, expiresAt); } @@ -190,22 +222,29 @@ function runQueue(providerName: string, state: ProviderPacer): void { if (waiter.modelId && waiter.modelIntervalMs > 0) { state.modelNextStartAt.set(waiter.modelId, startedAt + waiter.modelIntervalMs); } - waiter.resolve(); + waiter.resolve(acquireConcurrency(providerName, state, waiter)); runtime.enqueueMicrotask(() => runQueue(providerName, state)); } +/** Reserve a request start and capacity until the caller releases it or the signal aborts. */ export async function waitForProviderRequestSlot( providerName: string, provider: OcxProviderConfig, modelId?: string, signal?: AbortSignal, -): Promise { +): Promise<() => void> { const intervals = requestPacingIntervals(provider, modelId); - if (Math.max(intervals.providerIntervalMs, intervals.modelIntervalMs) <= 0) return; + const policy = provider.requestPacing?.enabled ? provider.requestPacing : undefined; + const providerMaxConcurrent = policy?.maxConcurrentRequests ?? Number.POSITIVE_INFINITY; + const modelMaxConcurrent = (modelId ? policy?.models?.[modelId]?.maxConcurrentRequests : undefined) + ?? Number.POSITIVE_INFINITY; + if (Math.max(intervals.providerIntervalMs, intervals.modelIntervalMs) <= 0 + && !Number.isFinite(providerMaxConcurrent) && !Number.isFinite(modelMaxConcurrent)) return () => {}; if (signal?.aborted) throw abortReason(signal); const state = pacers.get(providerName) ?? { - queue: [], providerNextStartAt: 0, modelNextStartAt: new Map(), + queue: [], inFlight: 0, modelInFlight: new Map(), + providerNextStartAt: 0, modelNextStartAt: new Map(), }; pacers.set(providerName, state); @@ -225,8 +264,11 @@ export async function waitForProviderRequestSlot( ); } - await new Promise((resolve, reject) => { - const waiter: Waiter = { modelId, ...intervals, queuedAt: runtime.now(), signal, resolve, reject }; + return new Promise<() => void>((resolve, reject) => { + const waiter: Waiter = { + modelId, ...intervals, providerMaxConcurrent, modelMaxConcurrent, + queuedAt: runtime.now(), signal, resolve, reject, + }; waiter.abort = () => { const index = state.queue.indexOf(waiter); if (index >= 0) state.queue.splice(index, 1); diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index 1c27951019a..1112d535a04 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -286,14 +286,14 @@ export function providerFetch( return httpFetch(input, init); }; let pacingSlotAcquired = options.pacingSlotAcquired === true; - const waitForPacing = (signal?: AbortSignal) => { + const waitForPacing = async (signal?: AbortSignal): Promise => { if (pacingSlotAcquired) { pacingSlotAcquired = false; - return Promise.resolve(); + return; + } + if (options.providerName) { + await waitForProviderRequestSlot(options.providerName, provider, options.modelId, signal); } - return options.providerName - ? waitForProviderRequestSlot(options.providerName, provider, options.modelId, signal) - : Promise.resolve(); }; const wrapped = async (input: Parameters[0], init?: RequestInit) => { await waitForPacing(init?.signal ?? undefined); diff --git a/src/types/provider.ts b/src/types/provider.ts index 7e52a323c5c..7c33d6b85f1 100644 --- a/src/types/provider.ts +++ b/src/types/provider.ts @@ -180,6 +180,8 @@ export interface RequestPacingRule { requestsPerMinute?: number; /** Minimum delay between request starts. The slower configured value wins. */ minIntervalMs?: number; + /** Maximum number of requests concurrently in flight. */ + maxConcurrentRequests?: number; } export interface ProviderRequestPacingConfig extends RequestPacingRule { diff --git a/structure/config.md b/structure/config.md index 32dc63be3ad..fae0c2a250e 100644 --- a/structure/config.md +++ b/structure/config.md @@ -97,6 +97,7 @@ merge cannot turn them into a valid config while discarding the original bytes. | --- | --- | --- | | Listener | `port`, `hostname` | The listener owns the port; `runtime-port.json` reports where it actually landed. | | Routing | `defaultProvider`, `providers`, per-provider `selectedModels`, `combos` | Explicit `provider/model` wins over `defaultProvider`; combo dispatch uses the selected target's existing capability ladder and does not create a second catalog authority. | +| Request pacing | `providers..requestPacing`, `requestPacing.models.` | Optional client-side request-start pacing supports interval limits and positive-integer `maxConcurrentRequests` caps. A provider or model rule may be concurrency-only; model entries target exact upstream IDs and can only add delay or narrow concurrency. | | Compaction routing | `compactionRouting.model`, optional `compactionRouting.reasoningEffort`, optional `compactionRouting.triggers` | Explicit Codex compaction metadata whose `compaction.trigger` is one the block names activates a request-local override; `triggers` defaults to `["manual"]`. See [Responses compaction](transports/responses-failover.md#compaction-routing-overrides). Invalid hand edits disable the block with a load warning without discarding providers; candidate writes reject invalid blocks. | | Catalog | `disabledModels`, `customModels`, `modelCacheTtlMs`, `providerContextCaps`, `contextCapValue`, per-provider `modelDisplayNames`, `codexAccountNamespaces`, `codexAccountPickerEnabled` | Catalog state is derived; config only records intent. Exact provider model display names are durable display only overlays. The picker flag is an explicit visibility override, while selector mappings remain the durable exact-routing contract. | | Retained state | `appOwnedMemoryBudgetMb` | Process-wide eviction target for app-owned logs, caches, blobs, and continuation payloads. Default 256 MiB, valid 64..4096; pinned state may temporarily exceed the target, but every pin-capable store has a finite local cap and their documented aggregate stays below `APP_OWNED_WORST_CASE_PINNED_BYTES` (512 MiB). Neither value caps RSS or native runtime memory. | @@ -555,7 +556,6 @@ The OpenCode launcher resolves the existing local management origin from the liv Provider `autoReviewModel` and `autoReviewModelOverrides` accept validated final-catalog selectors. Per-model keys preserve case and accept the existing raw/encoded slash equivalence. File-load degradation removes malformed optional selectors only; management writes reject malformed shapes. Omitted provider saves preserve selectors, explicit clears remove them, and raw editor candidates adopt normalized values before persistence and live replacement. See [catalog ownership](catalog.md#provider-scoped-approval-reviewer). Display-name validation retains prototype-shaped model IDs as data; reviewer-target map validation remains separate and rejects its reserved keys. - ## Explicit per-model capability declarations `modelCapabilities` on `src/types/provider.ts` stores exact model-ID entries with optional inputModalities, contextTier and video.processing axes. `src/config/provider-validation.ts` strictly validates writes and merges PATCH axes without sharing live objects; null map/model/axis/processing tombstones delete, while empty PATCH objects do nothing. Complete POST/PUT replacements reject tombstones. File reads retain valid axes; malformed explicit modalities restrict to text with a diagnostic. The two catalog writers receive explicit config and gather fingerprints include the map. This storage contract alone does not activate a context tier, advertise a larger window or enable video processing. diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 94dc5987050..57b2193f1f3 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -139,6 +139,14 @@ OAuth presets resolve discovery against the same canonical registry transport as before any adapter-specific transport override, so a stale configured `baseUrl` cannot receive an OAuth bearer token. +Provider request pacing in `src/providers/request-pacing.ts` combines start intervals with optional +`maxConcurrentRequests` limits. Provider capacity is shared across models; exact-model limits +apply in addition to that capacity. Admission reserves both counters atomically, and eligible +sibling models may bypass a saturated model lane. Releases are idempotent, wake queued requests, +and retain interval deadlines. Active aborts release capacity; dispatch owners release on response +body completion, cancellation, or failure. Capacity waits use the same bounded queue and retryable +queue-overload errors as interval waits. + ## TypeSafe JEV decision provider `src/providers/registry/entries-extended.ts` owns the canonical `jev` key preset at diff --git a/tests/server/management-provider-validation.test.ts b/tests/server/management-provider-validation.test.ts index 5a64e42f6f1..06eb65c39bf 100644 --- a/tests/server/management-provider-validation.test.ts +++ b/tests/server/management-provider-validation.test.ts @@ -1048,6 +1048,7 @@ describe("provider management validation", () => { enabled: true, requestsPerMinute: 38, minIntervalMs: 1_600, + maxConcurrentRequests: 4, models: { "deepseek-ai/deepseek-v4-flash-0731": { requestsPerMinute: 10 } }, }; @@ -1081,6 +1082,34 @@ describe("provider management validation", () => { }); expect(timerOverflow?.status).toBe(400); expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); + + const concurrencyOnly = await request("/api/providers?name=nvidia", { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: { + enabled: true, + maxConcurrentRequests: 2, + models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, + } }), + }); + expect(concurrencyOnly?.status).toBe(200); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual({ + enabled: true, + maxConcurrentRequests: 2, + models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, + }); + + const invalidConcurrency = await request("/api/providers?name=nvidia", { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: { enabled: true, maxConcurrentRequests: 0 } }), + }); + expect(invalidConcurrency?.status).toBe(400); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual({ + enabled: true, + maxConcurrentRequests: 2, + models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, + }); }); test("provider discovery status is additive and omitted before an attempt", async () => { diff --git a/tests/usage/request-pacing.test.ts b/tests/usage/request-pacing.test.ts index 2b776cd8381..bea2bfb6286 100644 --- a/tests/usage/request-pacing.test.ts +++ b/tests/usage/request-pacing.test.ts @@ -323,3 +323,96 @@ describe("provider request pacing queue", () => { expect(pacingWaited).toBe(1); }); }); + +describe("provider request concurrency", () => { + test("caps all models together and release is idempotent", async () => { + const configured = provider({ enabled: true, maxConcurrentRequests: 2 }); + const first = await waitForProviderRequestSlot("demo", configured, "a"); + const second = await waitForProviderRequestSlot("demo", configured, "b"); + const third = waitForProviderRequestSlot("demo", configured, "c"); + const fourth = waitForProviderRequestSlot("demo", configured, "d"); + expect(providerRequestPacingStatus("demo", configured).queued).toBe(2); + first(); + first(); + const releaseThird = await third; + expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); + second(); + const releaseFourth = await fourth; + releaseThird(); + releaseFourth(); + }); + + test("model limit tightens provider cap without blocking eligible siblings", async () => { + const configured = provider({ enabled: true, maxConcurrentRequests: 3, + models: { slow: { maxConcurrentRequests: 1 }, fast: { maxConcurrentRequests: 10 } } }); + const slow = await waitForProviderRequestSlot("demo", configured, "slow"); + const queuedSlow = waitForProviderRequestSlot("demo", configured, "slow"); + const fast = await waitForProviderRequestSlot("demo", configured, "fast"); + const anotherFast = await waitForProviderRequestSlot("demo", configured, "fast"); + const queuedFast = waitForProviderRequestSlot("demo", configured, "fast"); + expect(providerRequestPacingStatus("demo", configured).queued).toBe(2); + fast(); + const lastFast = await queuedFast; + expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); + slow(); + (await queuedSlow)(); + anotherFast(); + lastFast(); + }); + + test("model-only cap leaves other models and providers independent", async () => { + const configured = provider({ enabled: true, models: { slow: { maxConcurrentRequests: 1 } } }); + const first = await waitForProviderRequestSlot("demo", configured, "slow"); + const queued = waitForProviderRequestSlot("demo", configured, "slow"); + (await waitForProviderRequestSlot("demo", configured, "other"))(); + (await waitForProviderRequestSlot("other-provider", configured, "slow"))(); + expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); + first(); + (await queued)(); + }); + + test("aborting an active request frees exactly one slot", async () => { + const configured = provider({ enabled: true, maxConcurrentRequests: 1 }); + const controller = new AbortController(); + const release = await waitForProviderRequestSlot("demo", configured, "a", controller.signal); + const next = waitForProviderRequestSlot("demo", configured, "a"); + controller.abort(); + const releaseNext = await next; + release(); + const last = waitForProviderRequestSlot("demo", configured, "a"); + expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); + releaseNext(); + (await last)(); + }); + + test("concurrency wait expires without spinning and preserves active capacity", async () => { + const clock = fakePacingClock(); + setProviderRequestPacingRuntimeForTest(clock.runtime); + setProviderRequestPacingLimitsForTest({ maxQueueAgeMs: 25, maxQueueDepth: 1 }); + const configured = provider({ enabled: true, maxConcurrentRequests: 1 }); + const release = await waitForProviderRequestSlot("demo", configured); + const queued = waitForProviderRequestSlot("demo", configured); + await expect(waitForProviderRequestSlot("demo", configured)).rejects.toMatchObject({ reason: "queue_full" }); + expect(clock.pendingTimerCount()).toBe(1); + clock.advanceBy(25); + await expect(queued).rejects.toMatchObject({ reason: "queue_expired" }); + expect(clock.pendingTimerCount()).toBe(0); + const next = waitForProviderRequestSlot("demo", configured); + expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); + release(); + (await next)(); + }); + + test("releasing capacity still honors the start interval", async () => { + const clock = fakePacingClock(); + setProviderRequestPacingRuntimeForTest(clock.runtime); + const configured = provider({ enabled: true, maxConcurrentRequests: 1, minIntervalMs: 100 }); + const release = await waitForProviderRequestSlot("demo", configured); + const next = waitForProviderRequestSlot("demo", configured); + release(); + clock.advanceBy(99); + expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); + clock.advanceBy(1); + (await next)(); + }); +}); From a72deba8d4ed44f2a2d0f2d4ea0d0df5b24fdcca Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 02:43:16 +0900 Subject: [PATCH 02/19] fix(pacing): hold concurrency leases through physical response lifetime Track every physical HTTP send from admission through response-body completion, error, cancellation, and bounded abandonment. Preserve the first pre-acquired slot across adapter and runTurn paths, release before-dispatch failures, and move the capped management regression into a registered sibling. Co-authored-by: Brad Hallett <53977268+bradhallett@users.noreply.github.com> --- .../fr/reference/configuration/providers.md | 2 +- .../ja/reference/configuration/providers.md | 2 +- .../ko/reference/configuration/providers.md | 2 +- .../docs/reference/configuration/providers.md | 2 +- .../ru/reference/configuration/providers.md | 2 +- .../reference/configuration/providers.md | 2 +- .../reference/configuration/providers.md | 2 +- scripts/test-layout/layout.json | 1 + src/adapters/base.ts | 3 + src/adapters/kiro-retry.ts | 83 ++-- src/adapters/physical-send.ts | 13 +- src/images/loop.ts | 72 +-- src/providers/request-pacing.ts | 462 ++++++++++++++++-- src/server/responses/adapter-continuation.ts | 31 +- src/server/responses/adapter-dispatch.ts | 79 +-- src/server/responses/fetch-helpers.ts | 74 ++- src/server/responses/request-transport.ts | 33 +- src/server/responses/run-turn-execution.ts | 15 +- structure/providers-and-adapters.md | 9 +- tests/adapters/physical-send.test.ts | 22 +- tests/fixtures/test-layout-expected.json | 1 + ...management-provider-request-pacing.test.ts | 118 +++++ .../management-provider-validation.test.ts | 92 ---- tests/usage/request-pacing.test.ts | 119 ++++- 24 files changed, 899 insertions(+), 342 deletions(-) create mode 100644 tests/server/management-provider-request-pacing.test.ts diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index 710e3d2d710..0d95bbc1bc0 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -86,7 +86,7 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | --- | --- | --- | | `adapter` | `string` | L'un des `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (ou alias `azure`). | | `baseUrl` | `string` | URL de base de l'API en amont. La plupart des points de terminaison fixes intégrés ignorent une valeur incompatible ; les préréglages de clés protégés contre les collisions préservent une ancienne destination personnalisée portant le même nom. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP, Responses WebSocket et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. Avec une limite de concurrence, un tour Responses WebSocket canonique utilise HTTP/SSE afin de libérer la capacité à la fin, en cas d’erreur ou d’annulation du corps de réponse. | | `responsesPath?` | `string` | Chemin de ressource relatif pour les requêtes d'authentification par clé `openai-responses`. Il doit commencer par `/` et ne contenir aucun schéma, requête ou fragment. | | `chatCompletionsPath?` | `string` | Chemin de ressource relatif pour les requêtes `openai-chat`, miroir de `responsesPath` et soumis aux mêmes règles de forme. Nécessaire lorsqu'un même service en amont sert Chat Completions et Responses sous des préfixes différents : un override wire par modèle change l'adaptateur sans toucher `baseUrl`, donc sans ce réglage une requête Chat activée serait envoyée vers la base Responses. L'exemple fourni est Z.AI. | | `upstreamWebsocket?` | `boolean` | Active le transport Responses WebSocket en amont pour les requêtes `openai-responses` (désactivé par défaut). N'est honoré que pour l'amont first-party `https://api.openai.com/v1` ; les points de terminaison des fournisseurs personnalisés utilisent toujours HTTP/SSE borné, car Bun ne peut pas appliquer de limite de taille aux messages WebSocket entrants avant d'avoir alloué le message complet. Pour le fournisseur canonique ChatGPT `openai`, l'omettre conserve le WebSocket en amont sur les tours éligibles, `false` envoie les tours en streaming via HTTP/SSE, et `true` est refusé ; avec `false`, le pilotage et l'injection natifs en cours de tour sont indisponibles. Ce champ est indépendant du réglage `websockets` côté client et ne change ni le point de terminaison ni les identifiants. Une base HTTP reste en SSE ; les chemins qui ne sont pas Responses et les requêtes `openai-chat` restent en HTTP. | diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 95c68025cb9..5a83763c5eb 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -80,7 +80,7 @@ account を削除しても mapping は保持され、同じ id を再追加す | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai` (または別名 `azure`) のいずれか。 | | `baseUrl` | `string` |アップストリーム API のベース URL。ほとんどの組み込み固定エンドポイントは不一致を無視します。衝突安全キー プリセットは、古い同じ名前のカスタム宛先を保持します。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP、Responses WebSocket、明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP と明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。同時実行数の上限がある場合、標準の Responses WebSocket ターンは応答本文の完了・エラー・キャンセル時に枠を解放できるよう HTTP/SSE を使います。 | | `responsesPath?` | `string` |キー認証 `openai-responses` リクエストの相対リソース パス。 `/` で始まり、スキーム、クエリ、またはフラグメントが含まれていない必要があります。 | | `chatCompletionsPath?` | `string` | `openai-chat` リクエストの相対リソース パス。 `responsesPath` の対となる設定で、同じ形式ルールが適用されます。1つのアップストリームが Chat Completions と Responses を異なるプレフィックスで提供する場合に必要です。モデルごとの wire override はアダプターのみを変更し `baseUrl` は変更しないため、この設定がないと有効化された Chat リクエストが Responses ベースへ送信されます。同梱例は Z.AI です。 | | `upstreamWebsocket?` | `boolean` | `openai-responses` リクエストで使用するアップストリーム Responses WebSocket トランスポート（既定値は無効）。ファーストパーティの `https://api.openai.com/v1` アップストリームでのみ有効です。カスタムプロバイダーのエンドポイントは常に制限付き HTTP/SSE を使用します。Bun はメッセージ全体を確保する前に受信 WebSocket メッセージのサイズ上限を適用できないためです。正規の ChatGPT `openai` プロバイダーでは、省略すると対象となるターンでアップストリーム WebSocket を使用し、`false` はストリーミングのターンを HTTP/SSE で送信し、`true` は拒否されます。`false` の間はネイティブのターン途中のステアリングとインジェクションを利用できません。このフィールドはクライアント側の `websockets` 設定とは独立しており、エンドポイントと認証情報のどちらも変更しません。HTTP のベース URL は SSE のままとなり、Responses 以外のパスと `openai-chat` リクエストは HTTP を使用します。 | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index 71ff339c1fd..9dc6bbff29f 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -80,7 +80,7 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | --- | --- | --- | | `adapter` | `string` | `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` 중 하나이며, `azure`는 별칭입니다. | | `baseUrl` | `string` | 상위 API 기본 URL입니다. 대부분의 내장 고정 엔드포인트는 불일치를 무시합니다. 충돌 안전 키 프리셋은 같은 이름의 이전 사용자 지정 목적지를 보존합니다. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP, Responses WebSocket, 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP 및 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. 동시성 제한이 있으면 표준 Responses WebSocket 턴은 응답 본문의 완료·오류·취소 때 점유 슬롯을 해제할 수 있도록 HTTP/SSE를 사용합니다. | | `responsesPath?` | `string` | 키 인증 `openai-responses` 요청의 상대 리소스 경로입니다. 반드시 `/`로 시작해야 하며 스킴, query, fragment를 포함하면 안 됩니다. | | `chatCompletionsPath?` | `string` | `openai-chat` 요청의 상대 리소스 경로로, `responsesPath`와 동일한 형식 규칙이 적용되는 대응 항목입니다. 하나의 업스트림이 Chat Completions와 Responses를 서로 다른 접두사로 제공할 때 필요합니다. 모델별 wire override는 어댑터만 바꾸고 `baseUrl`은 그대로 두므로, 이 설정이 없으면 옵트인된 Chat 요청이 Responses base로 전송됩니다. Z.AI가 제공되는 예시입니다. | | `upstreamWebsocket?` | `boolean` | `openai-responses` 요청에 대한 업스트림 Responses WebSocket 전송을 선택적으로 활성화합니다(기본값 `false`). 퍼스트파티 `https://api.openai.com/v1` 업스트림에서만 적용되며, 사용자 지정 공급자 엔드포인트는 항상 제한된 HTTP/SSE를 사용합니다. Bun은 전체 메시지를 할당하기 전에는 수신 WebSocket 메시지 크기 제한을 적용할 수 없기 때문입니다. 정식 ChatGPT `openai` 공급자에서는 생략하면 대상 턴에서 업스트림 WebSocket을 사용하고, `false`는 스트리밍 턴을 HTTP/SSE로 전송하며, `true`는 거부됩니다. `false`이면 네이티브 턴 중 스티어링과 주입을 사용할 수 없습니다. 이 필드는 클라이언트 측 `websockets` 설정과 독립적이며 엔드포인트와 자격 증명을 변경하지 않습니다. HTTP 기본 URL은 SSE를 유지하고, Responses가 아닌 경로와 `openai-chat` 요청은 HTTP를 사용합니다. | diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index e91d9b581dd..57144f924e4 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -198,7 +198,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `baseUrl` | `string` | Upstream API base URL. Most built-in fixed endpoints ignore a mismatch; collision-safe key presets preserve an older same-named custom destination. | | `proxy?` | `string \| null` | Per-provider egress route. Omit it to inherit the global proxy decision; use `"direct"` or `null` to force direct egress; or provide an absolute `http://`, `https://`, `socks5://`, or `socks5h://` proxy URL. An empty string is rejected. | | `noProxy?` | `string \| string[]` | Destinations this provider reaches directly, using `NO_PROXY` host-pattern syntax. A match bypasses both this provider's own proxy and an inherited global proxy. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP, Responses WebSocket, and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. A concurrency-capped canonical Responses WebSocket turn uses HTTP/SSE so its lease can be released when the response body completes, errors, or is cancelled. | | `upstreamHttpVersion?` | `"auto" \| "http1.1" \| "h1" \| "http2" \| "h2"` | Pin the HTTP version used for upstream requests to this provider. Defaults to `auto`, which lets Bun negotiate. An explicit pin requires an HTTPS target and fails locally when it cannot be honored. Set `http1.1` when a provider's HTTP/2 SSE stream stalls instead of delivering events — the symptom is a long-running streaming request that produces nothing and eventually times out. For Cursor, `http1.1`/`h1` selects its `RunSSE` + `BidiAppend` compatibility transport for inference and also pins live model discovery. Management `POST`/`PATCH` accept `null` to clear it back to `auto`. | | `responsesPath?` | `string` | Relative resource path for key-auth `openai-responses` requests. It must start with `/` and contain no scheme, query, or fragment. | | `chatCompletionsPath?` | `string` | Relative resource path for `openai-chat` requests, the mirror of `responsesPath` and subject to the same shape rules. Needed when one upstream serves Chat Completions and Responses under different prefixes: a per-model wire override changes the adapter and leaves `baseUrl` alone, so without this an opted-in Chat request would be sent to the Responses base. Z.AI is the shipped example. | diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index 51a640474ac..62798fc78e6 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -93,7 +93,7 @@ cross-route credential fallback не существует. Строки API GPT- | --- | --- | --- | | `adapter` | `string` | Один из `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (или alias `azure`). | | `baseUrl` | `string` | Базовый URL API upstream'а. Большинство built-in fixed-endpoint'ов игнорируют несовпадение; collision-safe key-preset'ы сохраняют старый custom destination с тем же именем. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP, Responses WebSocket и явные вызовы адаптеров `fetchResponse`/`runTurn`. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP и явные вызовы адаптеров `fetchResponse`/`runTurn`. При ограничении параллелизма канонический запрос Responses WebSocket использует HTTP/SSE, чтобы освободить слот после завершения, ошибки или отмены тела ответа. | | `responsesPath?` | `string` | Relative resource path для key-auth запросов `openai-responses`. Должен начинаться с `/` и не может содержать scheme, query или fragment. | | `chatCompletionsPath?` | `string` | Relative resource path для запросов `openai-chat`, зеркало `responsesPath` с теми же правилами формы. Нужен, когда один upstream обслуживает Chat Completions и Responses под разными префиксами: per-model wire override меняет адаптер и не трогает `baseUrl`, поэтому без него включённый Chat-запрос ушёл бы в Responses base. Поставляемый пример — Z.AI. | | `upstreamWebsocket?` | `boolean` | Необязательный upstream Responses WebSocket для запросов `openai-responses` (по умолчанию `false`). Учитывается только для first-party upstream `https://api.openai.com/v1`; конечные точки пользовательских провайдеров всегда используют ограниченный HTTP/SSE, поскольку Bun не может применить ограничение размера входящего сообщения WebSocket до выделения памяти под всё сообщение. У канонического провайдера ChatGPT `openai` пропуск сохраняет upstream WebSocket для подходящих ходов, `false` отправляет потоковые ходы по HTTP/SSE, а `true` отклоняется; при `false` нативное управление и внедрение в середине хода недоступны. Это поле не зависит от клиентской настройки `websockets` и не меняет ни конечную точку, ни учётные данные. Для HTTP остаётся SSE; пути, не относящиеся к Responses, и запросы `openai-chat` остаются на HTTP. | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index 2480a5f2ca6..1cb588d964d 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -80,7 +80,7 @@ selector，而不是分配一个新名称。 | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或别名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API 基础 URL。大多数内置固定端点会忽略不匹配的值；具备冲突安全键的预设会保留一个更早、同名的自定义目标。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP、Responses WebSocket 以及显式适配器 `fetchResponse`/`runTurn` 调用。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP 以及显式适配器 `fetchResponse`/`runTurn` 调用。配置并发上限时，标准 Responses WebSocket 请求改用 HTTP/SSE，以便在响应体完成、出错或取消时释放并发名额。 | | `responsesPath?` | `string` | 用于 key-auth `openai-responses` 请求的相对资源路径。必须以 `/` 开头，且不能包含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | 用于 `openai-chat` 请求的相对资源路径，是 `responsesPath` 的对应项，适用相同的路径规则。当同一上游以不同前缀提供 Chat Completions 和 Responses 时需要此配置：按模型的 wire override 只更换适配器而不改动 `baseUrl`，否则已启用的 Chat 请求会被发送到 Responses base。随附示例为 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 为 `openai-responses` 请求选择性启用上游 Responses WebSocket 传输（默认 `false`）。仅对第一方 `https://api.openai.com/v1` 上游生效；自定义提供者端点始终使用有界 HTTP/SSE，因为 Bun 无法在分配完整消息之前对入站 WebSocket 消息实施大小限制。对于规范 ChatGPT `openai` 提供商，省略该字段会在符合条件的轮次使用上游 WebSocket，`false` 通过 HTTP/SSE 发送流式轮次，`true` 会被拒绝；设为 `false` 时，原生轮次中操控与注入不可用。该字段独立于客户端侧的 `websockets` 设置，且不改变端点或凭据。普通 HTTP 仍使用 SSE；非 Responses 路径和 `openai-chat` 请求仍使用 HTTP。 | diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index eb8fb5cd79e..780e40020c1 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -60,7 +60,7 @@ ocx models provider openrouter on | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或別名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API base URL。多數內建固定端點忽略不符；碰撞安全的金鑰預設保留較舊的同名自訂目的地。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP、Responses WebSocket 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。設定並行上限時，標準 Responses WebSocket 請求改用 HTTP/SSE，以便在回應本文完成、出錯或取消時釋放並行名額。 | | `responsesPath?` | `string` | Key-auth `openai-responses` 請求的相對資源路徑。必須以 `/` 開頭且不含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | `openai-chat` 請求的相對資源路徑，為 `responsesPath` 的對應項，適用相同的路徑規則。當同一上游以不同前綴提供 Chat Completions 與 Responses 時需要此設定：按模型的 wire override 只更換適配器而不改動 `baseUrl`，否則已啟用的 Chat 請求會送往 Responses base。隨附範例為 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 為 `openai-responses` 請求選用上游 Responses WebSocket 傳輸（預設 `false`）。僅對第一方 `https://api.openai.com/v1` 上游生效；自訂供應商端點一律使用有界 HTTP/SSE，因為 Bun 無法在配置完整訊息之前對傳入 WebSocket 訊息套用大小限制。對於規範 ChatGPT `openai` 供應商，省略此欄位會在符合條件的回合使用上游 WebSocket，`false` 會以 HTTP/SSE 傳送串流回合，`true` 會被拒絕；設為 `false` 時，原生回合中操控與注入無法使用。此欄位獨立於用戶端 `websockets` 設定，且不會變更端點或認證資料。一般 HTTP 仍使用 SSE；非 Responses 路徑與 `openai-chat` 請求仍使用 HTTP。 | diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index cc5dd7842a5..c6c7a6d0779 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1187,6 +1187,7 @@ "management-provider-atomicity.test.ts": "server", "management-provider-pinsless-validation.test.ts": "server", "management-provider-proto-override.test.ts": "server", + "management-provider-request-pacing.test.ts": "server", "management-provider-reasoning-lists.test.ts": "server", "management-provider-reset-replay.test.ts": "server", "management-provider-synthetic-max.test.ts": "server", diff --git a/src/adapters/base.ts b/src/adapters/base.ts index 0efce92e5b7..02fab18943e 100644 --- a/src/adapters/base.ts +++ b/src/adapters/base.ts @@ -3,12 +3,15 @@ import type { TranslatorBudget } from "../lib/translator-budget"; import type { RequestExecutionBudget } from "../lib/request-execution-budget"; import type { AttemptRecoveryKind, AttemptRecoveryWithheld } from "../usage/log"; import type { AdapterTierMetadata } from "../providers/fastwire"; +import type { ProviderRequestSlot } from "../providers/request-pacing"; /** Metadata about the caller's incoming request, for auth-forwarding adapters. */ export interface IncomingMeta { headers: Headers; translatorBudget: TranslatorBudget; abortSignal?: AbortSignal; + /** Lease acquired before the response is committed; the physical send transfers it to its body. */ + pacingSlot?: ProviderRequestSlot; /** * Provider-scoped fetch prepared by the Responses router. Stateful transports that emit more * than one physical HTTP request per logical turn must reuse it so every request participates in diff --git a/src/adapters/kiro-retry.ts b/src/adapters/kiro-retry.ts index d98a2a73e46..605c5c29b98 100644 --- a/src/adapters/kiro-retry.ts +++ b/src/adapters/kiro-retry.ts @@ -14,6 +14,7 @@ import { retryBackoffDelayMs, sleepWithAbort, } from "../lib/upstream-retry"; +import { releaseProviderRequestSlot, sendTrackingRequestSlot, type ProviderRequestSlot } from "../providers/request-pacing"; const RESET_ATTEMPTS = 3; const RESET_RETRY_BASE_MS = 150; @@ -167,49 +168,53 @@ async function fetchWithResetRecovery( for (let attempt = 0; attempt < RESET_ATTEMPTS; attempt++) { if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); const executor = (ctx.executor ?? globalThis.fetch) as typeof globalThis.fetch & { - waitForPacing?: (signal?: AbortSignal) => Promise; + waitForPacing?: (signal?: AbortSignal) => Promise; unpacedFetch?: typeof globalThis.fetch; }; - await executor.waitForPacing?.(ctx.abortSignal); - if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); - // Every physical send is admitted, not just the adapter entry. Kiro nests a throttle loop - // over this ladder and can run the ladder twice per throttle round, so counting one entry - // as one send hid up to eighteen upstream requests from the per-request cap (#4546). - const decision = ctx.sendBudget?.reserveDispatch({ sendClass: "transient", targetKey: url }); - if (decision && (!decision.allowed || !decision.permit.use())) { - throw new SendBudgetExhaustedError(url); - } - // Reported after admission and before dispatch, so a refused send is never counted and an - // admitted one is counted exactly once whichever way the fetch below settles. - notePhysicalSend(attempt > 0); + const slot = (await executor.waitForPacing?.(ctx.abortSignal)) || undefined; try { - const headers = new Headers(request.headers); - const recovered = attempt > 0; - if (recovered) headers.set("connection", "close"); - return await fetchWithAttemptDeadline(url, { - method: request.method, - headers, - body: request.body, - ...(recovered ? { keepalive: false } : {}), - }, timeoutMs, ctx.abortSignal, ctx.stream, (async (input, init) => { - try { - return await (executor.unpacedFetch ?? executor)(input, init); - } catch (error) { - if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); - const signal = init?.signal; - if (signal?.aborted && signal.reason instanceof Error && signal.reason.name === "TimeoutError") { - throw signal.reason; + if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); + // Every physical send is admitted, not just the adapter entry. Kiro nests a throttle loop + // over this ladder and can run the ladder twice per throttle round, so counting one entry + // as one send hid up to eighteen upstream requests from the per-request cap (#4546). + const decision = ctx.sendBudget?.reserveDispatch({ sendClass: "transient", targetKey: url }); + if (decision && (!decision.allowed || !decision.permit.use())) { + throw new SendBudgetExhaustedError(url); + } + // Reported after admission and before dispatch, so a refused send is never counted and an + // admitted one is counted exactly once whichever way the fetch below settles. + notePhysicalSend(attempt > 0); + try { + const headers = new Headers(request.headers); + const recovered = attempt > 0; + if (recovered) headers.set("connection", "close"); + return await sendTrackingRequestSlot(slot, () => fetchWithAttemptDeadline(url, { + method: request.method, + headers, + body: request.body, + ...(recovered ? { keepalive: false } : {}), + }, timeoutMs, ctx.abortSignal, ctx.stream, (async (input, init) => { + try { + return await (executor.unpacedFetch ?? executor)(input, init); + } catch (error) { + if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); + const signal = init?.signal; + if (signal?.aborted && signal.reason instanceof Error && signal.reason.name === "TimeoutError") { + throw signal.reason; + } + throw error; } - throw error; - } - }) as typeof globalThis.fetch); - } catch (error) { - if (ctx.abortSignal?.aborted || !isConnectionResetError(error) || attempt === RESET_ATTEMPTS - 1) throw error; - lastError = error; - await sleepWithAbort(retryBackoffDelayMs(attempt, { - baseDelayMs: RESET_RETRY_BASE_MS, - maxDelayMs: RESET_RETRY_MAX_MS, - }), ctx.abortSignal); + }) as typeof globalThis.fetch)); + } catch (error) { + if (ctx.abortSignal?.aborted || !isConnectionResetError(error) || attempt === RESET_ATTEMPTS - 1) throw error; + lastError = error; + await sleepWithAbort(retryBackoffDelayMs(attempt, { + baseDelayMs: RESET_RETRY_BASE_MS, + maxDelayMs: RESET_RETRY_MAX_MS, + }), ctx.abortSignal); + } + } finally { + releaseProviderRequestSlot(slot); } } throw lastError ?? new Error("Kiro fetch failed"); diff --git a/src/adapters/physical-send.ts b/src/adapters/physical-send.ts index 9f7c2a9f93b..669129019ee 100644 --- a/src/adapters/physical-send.ts +++ b/src/adapters/physical-send.ts @@ -2,9 +2,14 @@ import type { AdapterFetchContext } from "./base"; import type { SendClass } from "../lib/request-execution-budget"; import type { AttemptRecoveryKind } from "../usage/log"; import { abortError, SendBudgetExhaustedError } from "../lib/upstream-retry"; +import { + releaseProviderRequestSlot, + sendTrackingRequestSlot, + type ProviderRequestSlot, +} from "../providers/request-pacing"; type PacedFetch = typeof globalThis.fetch & { - waitForPacing?: (signal?: AbortSignal) => Promise; + waitForPacing?: (signal?: AbortSignal) => Promise; unpacedFetch?: typeof globalThis.fetch; }; @@ -37,13 +42,15 @@ export function createAdapterPhysicalSend(ctx: AdapterFetchContext = {}, fallbac ctx.onPhysicalSend?.({ ordinal, ...(options.recovery ? { recovery: options.recovery } : {}) }); return (executor.unpacedFetch ?? executor)(input, init); }) as typeof globalThis.fetch; + let slot: ProviderRequestSlot | undefined; try { - await executor.waitForPacing?.(ctx.abortSignal); + slot = (await executor.waitForPacing?.(ctx.abortSignal)) || undefined; if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); await options.beforeDispatch?.(); if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal); - return await options.dispatch(physicalExecutor); + return await sendTrackingRequestSlot(slot, () => options.dispatch(physicalExecutor)); } finally { + releaseProviderRequestSlot(slot); permit?.release(); } }; diff --git a/src/images/loop.ts b/src/images/loop.ts index dd12fbb9d48..dd66c198bb6 100644 --- a/src/images/loop.ts +++ b/src/images/loop.ts @@ -23,6 +23,7 @@ import { clearableDeadline, idleDeadline } from "../lib/abort"; import { readBoundedResponseBody } from "../lib/bounded-body"; import { applyUpstreamRecoveryInit, fetchWithResetRetry, prepareSameTarget429Wait } from "../lib/upstream-retry"; import { rateLimitRetryDelayMs } from "../providers/key-failover"; +import { releaseProviderRequestSlot, sendTrackingRequestSlot, type ProviderRequestSlot } from "../providers/request-pacing"; import { createTranslatorBudget, isTranslatorBudgetExceededError, @@ -309,7 +310,7 @@ export interface ImageBridgeDeps { /** Bind physical dispatch to this iteration's built request; pacing remains owned by the loop. */ fetchForRequest?: (request: AdapterRequest, parsed: OcxParsedRequest) => typeof globalThis.fetch; /** Reserve the routed provider's next request-start slot before each adapter dispatch. */ - waitForRequestSlot?: (signal?: AbortSignal) => Promise; + waitForRequestSlot?: (signal?: AbortSignal) => Promise; /** Raw adapter usage at the terminal event, pre wire-normalization (see bridgeToResponsesSSE onUsage). */ onUsage?: (usage: OcxUsage | undefined) => void; /** @@ -466,7 +467,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise internalAbort.abort("runTurn backlog exceeded"), }); @@ -514,6 +515,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise { if (accepting) { collectionError = err; @@ -528,6 +530,7 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise => { + const paceThenResetHeaderDeadline = async (): Promise => { headerDeadline.clear(); - await deps.waitForRequestSlot?.(signal); + const slot = (await deps.waitForRequestSlot?.(signal)) || undefined; headerDeadline = clearableDeadline(connectTimeoutMs, signal); + return slot; }; try { /** @@ -597,35 +601,43 @@ export async function runWithImageBridge(deps: ImageBridgeDeps): Promise requestAdapter.fetchResponse!(request, { + abortSignal: headerDeadline.signal, + timeoutMs: connectTimeoutMs, + returnRawErrors: true, + stream: true, + executor: requestFetch, + })); + } finally { + releaseProviderRequestSlot(slot); + } } else { response = await fetchWithResetRetry( async (retryRecovery) => { - await paceThenResetHeaderDeadline(); - // Record every helper-driven send (the callback runs for the first attempt and - // each connection-reset replay); preserve the caller's recovery kind - // (rate-limit-429 / key-429) when the retry layer supplies none. - deps.onAttemptSend?.(retryRecovery ?? recovery); - const h = new Headers(request.headers); - if (!h.has("accept-encoding")) h.set("accept-encoding", "identity"); - // Same reset-recovery parity as the web-search loop: the replay needs - // `keepalive: false` to abandon the pooled socket, because Bun has ignored the - // hop-by-hop header alone (oven-sh/bun#20492). - return requestFetch(request.url, applyUpstreamRecoveryInit({ - method: request.method, - redirect: "manual", - headers: h, - body: request.body, - signal: headerDeadline.signal, - }, retryRecovery)); + const slot = await paceThenResetHeaderDeadline(); + try { + // Record every helper-driven send (the callback runs for the first attempt and + // each connection-reset replay); preserve the caller's recovery kind + // (rate-limit-429 / key-429) when the retry layer supplies none. + deps.onAttemptSend?.(retryRecovery ?? recovery); + const h = new Headers(request.headers); + if (!h.has("accept-encoding")) h.set("accept-encoding", "identity"); + // Same reset-recovery parity as the web-search loop: the replay needs + // `keepalive: false` to abandon the pooled socket, because Bun has ignored the + // hop-by-hop header alone (oven-sh/bun#20492). + return await sendTrackingRequestSlot(slot, () => requestFetch(request.url, applyUpstreamRecoveryInit({ + method: request.method, + redirect: "manual", + headers: h, + body: request.body, + signal: headerDeadline.signal, + }, retryRecovery))); + } finally { + releaseProviderRequestSlot(slot); + } }, { replaySafe: true, abortSignal: headerDeadline.signal, label: "image-bridge-loop" }, ); diff --git a/src/providers/request-pacing.ts b/src/providers/request-pacing.ts index e6181933a31..1320d4cb35a 100644 --- a/src/providers/request-pacing.ts +++ b/src/providers/request-pacing.ts @@ -1,8 +1,34 @@ import type { OcxProviderConfig, RequestPacingRule } from "../types"; import type { GenerationContext } from "../lib/state-store-sweeper"; +import { carryReplayRefusal, isNonReplayableResponse, markResponseNonReplayable } from "../lib/upstream-retry"; +import { redactSecretString } from "../lib/redact"; export const REQUEST_PACING_MAX_QUEUE_DEPTH = 256; export const REQUEST_PACING_MAX_QUEUE_AGE_MS = 60_000; +/** + * A leased response body that is neither read nor cancelled for this long releases its + * lease and cancels the body: a caller that dropped a Response without touching its body + * must not hold a concurrency slot until process restart. The first pull or cancel swaps + * this deadline for the longer inactivity window below. + */ +export const REQUEST_PACING_UNCONSUMED_BODY_MS = 30_000; +/** + * Once a tracked body has been pulled at least once, this much silence reclaims its lease + * and cancels the body: a consumer that reads part of a response and then abandons it + * without cancel() (a clone-based peek that cancels only its own tee branch) must not + * hold a concurrency slot forever either. The window is far longer than the unconsumed + * deadline because a live consumer waiting on a slow upstream is indistinguishable from + * an abandoned one at this layer; providers keep SSE connections alive with keep-alives + * well inside this window. + */ +export const REQUEST_PACING_BODY_INACTIVITY_MS = 300_000; +/** + * Retry-After floor for a waiter refused while blocked by a concurrency cap rather than + * by a spacing interval. Interval readiness computes to zero under a pure cap, and the + * 1s minimum would invite clients to re-hit a saturated provider every second, which is + * the shared-account 429 churn this feature exists to prevent. + */ +export const REQUEST_PACING_CONCURRENCY_RETRY_AFTER_SECONDS = 5; let maxQueueDepth = REQUEST_PACING_MAX_QUEUE_DEPTH; let maxQueueAgeMs = REQUEST_PACING_MAX_QUEUE_AGE_MS; @@ -33,26 +59,64 @@ interface Waiter { modelId?: string; providerIntervalMs: number; modelIntervalMs: number; - queuedAt: number; - signal?: AbortSignal; providerMaxConcurrent: number; modelMaxConcurrent: number; - resolve: (release: () => void) => void; + queuedAt: number; + signal?: AbortSignal; + resolve: (slot: ProviderRequestSlot) => void; reject: (reason: unknown) => void; abort?: () => void; } interface ProviderPacer { queue: Waiter[]; - inFlight: number; - modelInFlight: Map; providerNextStartAt: number; modelNextStartAt: Map; + providerInFlight: number; + modelInFlight: Map; timer?: unknown; lastStartedAt?: number; lastModelId?: string; } +/** + * Lease returned by waitForProviderRequestSlot. Inert unless the provider (or the + * request model override) sets maxConcurrentRequests; then release() returns the + * concurrency slot when the upstream request finishes, whether that is body completion, + * body cancellation, or a send that never produced a response. + */ +export interface ProviderRequestSlot { + /** Provider identity for operator diagnostics: names the lease holder in deadline warnings. */ + readonly providerName?: string; + /** + * True only when this slot holds a concurrency lease whose release must follow the + * upstream body lifecycle. Interval-only slots stay inert so response objects keep + * their identity through the fetch path. + */ + readonly leased: boolean; + /** + * True once a tracked response body owns this lease, meaning body completion, body + * cancellation, or the unconsumed-body deadline will release it. Turn-end cleanup must + * then leave the release to that lifecycle instead of returning the lease early. + */ + readonly bodyTracked: boolean; + /** + * True once release() has run. A turn-scoped transport uses this to stop pacing by + * interval after the lease it relied on is gone (a send that threw, a body that + * closed) and re-acquire with concurrency, so the cap keeps counting its follow-ups. + */ + readonly released: boolean; + /** Idempotent. Safe to call from body completion, cancellation, and error paths alike. */ + release(): void; +} + +/** Internal: leased slots expose this to trackProviderRequestSlotBody when a body takes over the release. */ +interface BodyTrackableProviderRequestSlot extends ProviderRequestSlot { + markBodyTracked(): void; +} + +const inertProviderRequestSlot: ProviderRequestSlot = { leased: false, bodyTracked: false, released: false, release() {} }; + export interface RequestPacingRuntime { now: () => number; setTimer: (callback: () => void, delayMs: number) => unknown; @@ -65,6 +129,7 @@ export interface ProviderRequestPacingStatus { enabled: boolean; queued: number; nextSlotInMs: number; + inFlight?: number; lastStartedAt?: number; lastModelId?: string; } @@ -94,6 +159,15 @@ function normalizedInterval(rule: RequestPacingRule | undefined): number { return Math.max(rpmInterval, fixedInterval); } +function normalizedMaxConcurrent(rule: RequestPacingRule | undefined): number { + return typeof rule?.maxConcurrentRequests === "number" && rule.maxConcurrentRequests > 0 + // Floor so runtime-injected configs that bypass the integer schema cannot admit + // one request past the configured ceiling (a 2.5 cap must not let a third start); + // clamp sub-1 fractionals up to 1 so a positive cap never degrades into none. + ? Math.max(1, Math.floor(rule.maxConcurrentRequests)) + : 0; +} + export function requestPacingIntervalMs(provider: OcxProviderConfig, modelId?: string): number { const policy = provider.requestPacing; if (!policy?.enabled) return 0; @@ -104,40 +178,32 @@ export function requestPacingIntervalMs(provider: OcxProviderConfig, modelId?: s function requestPacingIntervals(provider: OcxProviderConfig, modelId?: string): { providerIntervalMs: number; modelIntervalMs: number; + providerMaxConcurrent: number; + modelMaxConcurrent: number; } { const policy = provider.requestPacing; - if (!policy?.enabled) return { providerIntervalMs: 0, modelIntervalMs: 0 }; + if (!policy?.enabled) { + return { providerIntervalMs: 0, modelIntervalMs: 0, providerMaxConcurrent: 0, modelMaxConcurrent: 0 }; + } return { providerIntervalMs: normalizedInterval(policy), modelIntervalMs: modelId ? normalizedInterval(policy.models?.[modelId]) : 0, + providerMaxConcurrent: normalizedMaxConcurrent(policy), + modelMaxConcurrent: modelId ? normalizedMaxConcurrent(policy.models?.[modelId]) : 0, }; } -function concurrencyAvailable(state: ProviderPacer, waiter: Waiter): boolean { - return state.inFlight < waiter.providerMaxConcurrent - && (state.modelInFlight.get(waiter.modelId ?? "") ?? 0) < waiter.modelMaxConcurrent; -} - -function acquireConcurrency(providerName: string, state: ProviderPacer, waiter: Waiter): () => void { - const model = waiter.modelId ?? ""; - state.inFlight += 1; - state.modelInFlight.set(model, (state.modelInFlight.get(model) ?? 0) + 1); - let released = false; - const release = () => { - if (released) return; - released = true; - waiter.signal?.removeEventListener("abort", release); - state.inFlight -= 1; - const remaining = (state.modelInFlight.get(model) ?? 1) - 1; - if (remaining > 0) state.modelInFlight.set(model, remaining); - else state.modelInFlight.delete(model); - if (state.timer) runtime.clearTimer(state.timer); - state.timer = undefined; - runQueue(providerName, state); - }; - waiter.signal?.addEventListener("abort", release, { once: true }); - if (waiter.signal?.aborted) release(); - return release; +/** + * Whether a concurrency cap applies to this provider/model, expressed as the looser of + * the two configured bounds. Enforcement gates the provider cap and the model cap + * independently (a model override of 1 inside a provider cap of 5 admits one), so the + * return value is a cap-presence signal — every current caller tests it for > 0 — and + * NOT an effective ceiling. A caller that needs the true ceiling must compute + * min(provider, model || Infinity) itself. + */ +export function requestPacingMaxConcurrentRequests(provider: OcxProviderConfig, modelId?: string): number { + const limits = requestPacingIntervals(provider, modelId); + return Math.max(limits.providerMaxConcurrent, limits.modelMaxConcurrent); } function waiterReadyAt(state: ProviderPacer, modelId: string | undefined): number { @@ -147,8 +213,65 @@ function waiterReadyAt(state: ProviderPacer, modelId: string | undefined): numbe ); } -function pacingRetryAfterSeconds(state: ProviderPacer, modelId: string | undefined, now: number): number { - return Math.max(1, Math.ceil(Math.max(0, waiterReadyAt(state, modelId) - now) / 1000)); +function pacingRetryAfterSeconds( + state: ProviderPacer, + modelId: string | undefined, + now: number, + concurrencyCapped = false, +): number { + const intervalSeconds = Math.max(1, Math.ceil(Math.max(0, waiterReadyAt(state, modelId) - now) / 1000)); + // A waiter blocked by an in-flight lease has no interval to report: when the wait ends + // in refusal, the honest answer is that the cap is saturated, not "retry every second". + return concurrencyCapped + ? Math.max(intervalSeconds, REQUEST_PACING_CONCURRENCY_RETRY_AFTER_SECONDS) + : intervalSeconds; +} + +function makeProviderRequestSlot( + providerName: string, + state: ProviderPacer, + waiter: Waiter, +): BodyTrackableProviderRequestSlot { + let released = false; + let bodyTracked = false; + const leased = waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0; + const slot: BodyTrackableProviderRequestSlot = { + providerName, + leased, + get bodyTracked() { + return bodyTracked; + }, + get released() { + return released; + }, + markBodyTracked() { + bodyTracked = true; + }, + release() { + if (released) return; + released = true; + waiter.signal?.removeEventListener("abort", releaseOnAbort); + if (waiter.providerMaxConcurrent > 0 && state.providerInFlight > 0) state.providerInFlight -= 1; + if (waiter.modelId && waiter.modelMaxConcurrent > 0) { + const current = state.modelInFlight.get(waiter.modelId) ?? 0; + if (current <= 1) state.modelInFlight.delete(waiter.modelId); + else state.modelInFlight.set(waiter.modelId, current - 1); + } + runtime.enqueueMicrotask(() => { + // A pending wake-up timer makes runQueue defer to it, but the lease that just + // returned may admit a waiter the timer was never scheduled for, so take over. + if (state.timer) { + runtime.clearTimer(state.timer); + state.timer = undefined; + } + runQueue(providerName, state); + }); + }, + }; + const releaseOnAbort = (): void => slot.release(); + waiter.signal?.addEventListener("abort", releaseOnAbort, { once: true }); + if (waiter.signal?.aborted) slot.release(); + return slot; } function rejectExpiredWaiters(providerName: string, state: ProviderPacer, now: number): void { @@ -160,7 +283,12 @@ function rejectExpiredWaiters(providerName: string, state: ProviderPacer, now: n waiter.reject(new RequestPacingQueueOverloadError( providerName, "queue_expired", - pacingRetryAfterSeconds(state, waiter.modelId, now), + pacingRetryAfterSeconds( + state, + waiter.modelId, + now, + waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0, + ), )); } } @@ -193,18 +321,27 @@ function runQueue(providerName: string, state: ProviderPacer): void { const providerReadyAt = Math.max(now, state.providerNextStartAt); const waiterIndex = state.queue.findIndex(waiter => { + if (waiter.providerMaxConcurrent > 0 && state.providerInFlight >= waiter.providerMaxConcurrent) return false; + if (waiter.modelId && waiter.modelMaxConcurrent > 0 + && (state.modelInFlight.get(waiter.modelId) ?? 0) >= waiter.modelMaxConcurrent) return false; const modelReadyAt = waiter.modelId ? (state.modelNextStartAt.get(waiter.modelId) ?? 0) : 0; - return concurrencyAvailable(state, waiter) && Math.max(providerReadyAt, modelReadyAt) <= now; + return Math.max(providerReadyAt, modelReadyAt) <= now; }); if (waiterIndex < 0) { let earliestAt = Number.POSITIVE_INFINITY; for (const waiter of state.queue) { const modelReadyAt = waiter.modelId ? (state.modelNextStartAt.get(waiter.modelId) ?? 0) : 0; - const readyAt = concurrencyAvailable(state, waiter) - ? Math.max(providerReadyAt, modelReadyAt) : Number.POSITIVE_INFINITY; + const readyAt = Math.max(providerReadyAt, modelReadyAt); const expiresAt = waiter.queuedAt + maxQueueAgeMs; - earliestAt = Math.min(earliestAt, readyAt, expiresAt); + // A waiter whose start time already passed is blocked on an in-flight lease; its release + // re-runs this queue through a microtask, so the timer only needs its expiry backstop. + // Scheduling for its readyAt (in the past) would spin the timer on every empty pass. + earliestAt = Math.min(earliestAt, readyAt <= now ? Number.POSITIVE_INFINITY : readyAt, expiresAt); } + // Unreachable for a non-empty queue: rejectExpiredWaiters above removed every waiter + // whose expiresAt passed, so each remaining one contributes a finite value. Kept as a + // belt-and-braces backstop rather than a case future readers should hunt for. + if (!Number.isFinite(earliestAt)) return; const delayMs = Math.max(0, earliestAt - now); state.timer = runtime.setTimer(() => { state.timer = undefined; @@ -222,29 +359,41 @@ function runQueue(providerName: string, state: ProviderPacer): void { if (waiter.modelId && waiter.modelIntervalMs > 0) { state.modelNextStartAt.set(waiter.modelId, startedAt + waiter.modelIntervalMs); } - waiter.resolve(acquireConcurrency(providerName, state, waiter)); + if (waiter.providerMaxConcurrent > 0) state.providerInFlight += 1; + if (waiter.modelId && waiter.modelMaxConcurrent > 0) { + state.modelInFlight.set(waiter.modelId, (state.modelInFlight.get(waiter.modelId) ?? 0) + 1); + } + waiter.resolve(makeProviderRequestSlot(providerName, state, waiter)); runtime.enqueueMicrotask(() => runQueue(providerName, state)); } -/** Reserve a request start and capacity until the caller releases it or the signal aborts. */ export async function waitForProviderRequestSlot( providerName: string, provider: OcxProviderConfig, modelId?: string, signal?: AbortSignal, -): Promise<() => void> { + options?: { concurrency?: boolean }, +): Promise { const intervals = requestPacingIntervals(provider, modelId); - const policy = provider.requestPacing?.enabled ? provider.requestPacing : undefined; - const providerMaxConcurrent = policy?.maxConcurrentRequests ?? Number.POSITIVE_INFINITY; - const modelMaxConcurrent = (modelId ? policy?.models?.[modelId]?.maxConcurrentRequests : undefined) - ?? Number.POSITIVE_INFINITY; - if (Math.max(intervals.providerIntervalMs, intervals.modelIntervalMs) <= 0 - && !Number.isFinite(providerMaxConcurrent) && !Number.isFinite(modelMaxConcurrent)) return () => {}; + // A turn-scoped transport sends several physical requests per logical turn (Cursor + // HTTP/1.1 RunSSE plus BidiAppends). One lease covers the turn; follow-up sends pace + // by interval only, or a follow-up would queue behind the lease its own turn holds. + const concurrency = options?.concurrency !== false; + const waiterIntervals = concurrency ? intervals : { + providerIntervalMs: intervals.providerIntervalMs, + modelIntervalMs: intervals.modelIntervalMs, + providerMaxConcurrent: 0, + modelMaxConcurrent: 0, + }; + const paced = Math.max(intervals.providerIntervalMs, intervals.modelIntervalMs) > 0 + || waiterIntervals.providerMaxConcurrent > 0 + || waiterIntervals.modelMaxConcurrent > 0; + if (!paced) return inertProviderRequestSlot; if (signal?.aborted) throw abortReason(signal); const state = pacers.get(providerName) ?? { - queue: [], inFlight: 0, modelInFlight: new Map(), - providerNextStartAt: 0, modelNextStartAt: new Map(), + queue: [], providerNextStartAt: 0, modelNextStartAt: new Map(), + providerInFlight: 0, modelInFlight: new Map(), }; pacers.set(providerName, state); @@ -260,14 +409,23 @@ export async function waitForProviderRequestSlot( throw new RequestPacingQueueOverloadError( providerName, "queue_full", - pacingRetryAfterSeconds(state, modelId, runtime.now()), + pacingRetryAfterSeconds( + state, + modelId, + runtime.now(), + waiterIntervals.providerMaxConcurrent > 0 || waiterIntervals.modelMaxConcurrent > 0, + ), ); } - return new Promise<() => void>((resolve, reject) => { + return await new Promise((resolve, reject) => { const waiter: Waiter = { - modelId, ...intervals, providerMaxConcurrent, modelMaxConcurrent, - queuedAt: runtime.now(), signal, resolve, reject, + modelId, + ...waiterIntervals, + queuedAt: runtime.now(), + signal, + resolve, + reject, }; waiter.abort = () => { const index = state.queue.indexOf(waiter); @@ -294,6 +452,192 @@ export async function waitForProviderRequestSlot( }); } +/** + * Tie a pacing slot lease to an upstream response body: the lease is released when the + * body completes, errors, or is cancelled by the consumer. A null body (204/304/HEAD) + * means the exchange is already finished, so the lease returns immediately. A body that + * is neither read nor cancelled for REQUEST_PACING_UNCONSUMED_BODY_MS releases its lease + * and cancels the body, so a dropped Response cannot hold a slot forever. The returned + * Response preserves status, statusText, headers, and the identity-based replay markers. + */ +export function trackProviderRequestSlotBody( + slot: ProviderRequestSlot | undefined, + response: Response, +): Response { + // An unleased slot has nothing to return on body close, and rewrapping the Response + // would break identity-based markers (the eager WS relay registry is a WeakSet). + if (!slot?.leased) return response; + // Slots built outside waitForProviderRequestSlot (test doubles) satisfy only the public + // ProviderRequestSlot shape; their bodies still own the release through the wrapper below. + (slot as Partial).markBodyTracked?.(); + if (!response.body) { + slot.release(); + return response; + } + const source = response.body; + let released = false; + // Set when a deadline callback cancelled the source: the truncation must surface as a + // stream error on the next pull, never as a clean EOF a relay would treat as success. + let expired = false; + let expiryTimer: unknown; + let expiryKind: "unconsumed" | "inactive" = "unconsumed"; + const clearExpiryTimer = (): void => { + if (expiryTimer === undefined) return; + runtime.clearTimer(expiryTimer); + expiryTimer = undefined; + }; + const release = (): void => { + if (released) return; + released = true; + clearExpiryTimer(); + slot.release(); + }; + const armExpiryTimer = (delayMs: number, kind: "unconsumed" | "inactive"): void => { + clearExpiryTimer(); + expiryKind = kind; + expiryTimer = runtime.setTimer(() => { + expiryTimer = undefined; + if (released) return; + expired = true; + release(); + // Cancel through the reader when one exists: after the first pull the source is + // locked to it, and cancelling a locked stream rejects without releasing the + // socket — the exact upstream leak this deadline exists to prevent. + void (reader ?? source).cancel().catch(() => {}); + const who = slot.providerName === undefined ? "" : ` for provider ${JSON.stringify(redactSecretString(slot.providerName))}`; + console.warn( + expiryKind === "inactive" + ? `[opencodex] requestPacing${who} released a concurrency lease after ` + + REQUEST_PACING_BODY_INACTIVITY_MS + + "ms of response body inactivity; the body was cancelled." + : `[opencodex] requestPacing${who} released a concurrency lease after ` + + REQUEST_PACING_UNCONSUMED_BODY_MS + + "ms because the provider response body was neither read nor cancelled; the body was cancelled.", + ); + }, delayMs); + }; + armExpiryTimer(REQUEST_PACING_UNCONSUMED_BODY_MS, "unconsumed"); + let reader: ReadableStreamDefaultReader | undefined; + let cancelled = false; + const tracked = new ReadableStream({ + pull: async controller => { + // A pull proves a consumer is attached, not that it will keep reading: re-arm a + // longer inactivity deadline instead of disarming. A body read once and then + // abandoned (a clone-based peek that cancels only its own tee branch) must still + // return its lease, while a live stream keeps pushing the deadline back per pull. + if (!released) armExpiryTimer(REQUEST_PACING_BODY_INACTIVITY_MS, "inactive"); + try { + // Inside the try: a source another reader already locked makes getReader() + // throw, and with the deadline disarmed above that failure must release the + // lease here instead of stranding it for the process lifetime. + reader ??= source.getReader(); + const { done, value } = await reader.read(); + // A consumer cancel while this read was pending resolves it (done or a late chunk); + // touching the cancelled controller would throw from the pull algorithm. + if (cancelled) return; + if (expired) { + // The deadline cancelled the source mid-stream; report the truncation as an + // error instead of letting the cancelled read's done flag close the stream. + controller.error(new Error( + "[opencodex] requestPacing cancelled the response body after its lease deadline", + )); + return; + } + if (done) { + controller.close(); + release(); + return; + } + controller.enqueue(value); + } catch (error) { + if (cancelled) return; + release(); + controller.error(error); + } + }, + cancel: reason => { + cancelled = true; + clearExpiryTimer(); + release(); + return (reader ?? source).cancel(reason); + }, + }, { + // A default-count stream pulls once at construction with no reader attached, which + // would mark the body consumed and disarm the deadline before any real consumer + // arrives. Zero capacity keeps pull consumer-driven: the first read arms nothing + // until a reader actually asks for bytes. + highWaterMark: 0, + }); + let wrapped: Response; + try { + wrapped = new Response(tracked, { + status: response.status, + statusText: response.statusText, + headers: response.headers, + }); + } catch { + // A non-conforming status (a proxy passing a raw 6xx through) or a body on a + // null-body status throws here after markBodyTracked, and boundary cleanup skips + // body-tracked slots: return the lease now instead of waiting out the deadline. + // The abandoned rewrap never locked the source (no pull ran), so hand back the + // ORIGINAL response with its body intact: rethrowing would make the google-http, + // command-code and mimo retry ladders replay a request whose response did arrive. + release(); + return response; + } + // Retry helpers mark the response they return from, and recovery decisions key on these + // identity markers; the rewrap must not make a non-replayable response look replayable. + if (isNonReplayableResponse(response)) markResponseNonReplayable(wrapped); + return carryReplayRefusal(response, wrapped); +} + +/** + * Return a lease at a turn or attempt boundary: a lease a tracked response body now owns + * is left to that body's lifecycle, and any other unconsumed lease is released. Idempotent, + * and a no-op for interval-only slots, so every boundary can call it unconditionally. + */ +export function releaseProviderRequestSlot(slot: ProviderRequestSlot | undefined): void { + if (!slot || slot.bodyTracked) return; + slot.release(); +} + +/** Transfer a physical send's lease to its response body, or release it on failure. */ +export async function sendTrackingRequestSlot( + slot: ProviderRequestSlot | undefined, + send: () => Promise, +): Promise { + try { + return trackProviderRequestSlotBody(slot, await send()); + } catch (error) { + slot?.release(); + throw error; + } +} + +/** + * Acquire one pacing lease and return it at the boundary. The send callback runs with + * the lease; this helper releases it unless a tracked response body has taken ownership. + * A send that throws (an abort, a send-budget refusal, a build failure) or returns + * without ever dispatching through the executor would otherwise strand the lease for + * the process lifetime, and a dispatched send's tracked body keeps its own release. + * Centralized so every dispatch boundary shares one copy of the acquire/release pairing + * and a future edit cannot fork the lease lifecycle. + */ +export async function withProviderRequestSlot( + providerName: string, + provider: OcxProviderConfig, + modelId: string | undefined, + signal: AbortSignal | undefined, + send: (pacingSlot: ProviderRequestSlot) => Promise, +): Promise { + const pacingSlot = await waitForProviderRequestSlot(providerName, provider, modelId, signal); + try { + return await send(pacingSlot); + } finally { + releaseProviderRequestSlot(pacingSlot); + } +} + export function providerRequestPacingStatus( providerName: string, provider: OcxProviderConfig, @@ -308,11 +652,23 @@ export function providerRequestPacingStatus( } if (Number.isFinite(earliestQueuedSlotAt)) nextSlotAt = earliestQueuedSlotAt; } + const providerConcurrencyCap = requestPacingMaxConcurrentRequests(provider); + // Gate on enabled so the status surface agrees with enforcement: a disabled requestPacing + // block must not report in-flight leases alongside enabled: false. + const anyConcurrencyCap = provider.requestPacing?.enabled === true && (providerConcurrencyCap > 0 + || Object.values(provider.requestPacing?.models ?? {}).some(rule => normalizedMaxConcurrent(rule) > 0)); return { provider: providerName, enabled: provider.requestPacing?.enabled === true, queued: state?.queue.length ?? 0, nextSlotInMs: Math.max(0, Math.ceil(nextSlotAt - now)), + // A provider cap counts every paced send in providerInFlight; model-only caps keep + // providerInFlight at zero, so report the per-model sum for those providers instead. + ...(anyConcurrencyCap ? { + inFlight: providerConcurrencyCap > 0 + ? state?.providerInFlight ?? 0 + : [...(state?.modelInFlight.values() ?? [])].reduce((sum, count) => sum + count, 0), + } : {}), ...(state?.lastStartedAt !== undefined ? { lastStartedAt: state.lastStartedAt } : {}), ...(state?.lastModelId ? { lastModelId: state.lastModelId } : {}), }; diff --git a/src/server/responses/adapter-continuation.ts b/src/server/responses/adapter-continuation.ts index c3d57e66294..519520b66a4 100644 --- a/src/server/responses/adapter-continuation.ts +++ b/src/server/responses/adapter-continuation.ts @@ -14,7 +14,7 @@ import { recordAttemptCredentialSource, } from "../request-log"; import { noteAttemptRecoveryWithheld } from "../request-log"; -import { waitForProviderRequestSlot } from "../../providers/request-pacing"; +import { withProviderRequestSlot } from "../../providers/request-pacing"; import { providerFetch, fetchWithHeaderTimeout, safeHostLabel } from "./fetch-helpers"; import { transientRetryPolicyFor, @@ -199,21 +199,22 @@ export function createAdapterContinuations( try { if (transportState.activeAdapter.fetchResponse) { transportState.noteRoutedAttemptSend(continuationEstimate, replayKind); - await waitForProviderRequestSlot(route.providerName, route.provider, nextParsed.modelId, upstream.signal); - return await transportState.activeAdapter.fetchResponse(builtContinuationRequest, { - abortSignal: upstream.signal, - timeoutMs: connectMs, + return await withProviderRequestSlot(route.providerName, route.provider, nextParsed.modelId, upstream.signal, pacingSlot => + transportState.activeAdapter.fetchResponse!(builtContinuationRequest, { + abortSignal: upstream.signal, + timeoutMs: connectMs, sendBudget: adapterDispatchBudget, - onPhysicalSend: send => noteAdapterPhysicalSend(continuationEstimate, send), - onRecoveryWithheld: noteAdapterRecoveryWithheld, - stream: nextParsed.stream, - executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { - pacingSlotAcquired: true, - dispatchOverride: oauthDispatch(builtContinuationRequest, nextParsed), - providerName: route.providerName, - modelId: nextParsed.modelId, - }), - }); + onPhysicalSend: send => noteAdapterPhysicalSend(continuationEstimate, send), + onRecoveryWithheld: noteAdapterRecoveryWithheld, + stream: nextParsed.stream, + executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { + pacingSlotAcquired: true, + pacingSlot, + dispatchOverride: oauthDispatch(builtContinuationRequest, nextParsed), + providerName: route.providerName, + modelId: nextParsed.modelId, + }), + })); } // Same #1851 scope guard as the initial send: transient-5xx retry only for direct // Google AI Studio; every other adapter keeps reset-only semantics here. diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index abb20f9940f..eed634a8df9 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -19,7 +19,7 @@ import { import { clientCancelledResponse, readDisplaySafeErrorText, normalizeUpstreamErrorText } from "./core-errors"; import { redactSecretString } from "../../lib/redact"; import { rewriteUpstreamPolicyRefusal } from "./policy-refusal"; -import { waitForProviderRequestSlot } from "../../providers/request-pacing"; +import { withProviderRequestSlot } from "../../providers/request-pacing"; import { providerFetch, fetchWithHeaderTimeout, safeHostLabel } from "./fetch-helpers"; import { transientRetryPolicyFor, @@ -296,21 +296,22 @@ export async function prepareAdapterExchange( try { if (transportState.activeAdapter.fetchResponse) { transportState.noteRoutedAttemptSend(inputTokenEstimate); - await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, upstream.signal); - upstreamResponse = await transportState.activeAdapter.fetchResponse(builtInitialRequest, { - abortSignal: upstream.signal, - timeoutMs: connectMs, - sendBudget: adapterDispatchBudget, - onPhysicalSend: send => noteAdapterPhysicalSend(inputTokenEstimate, send), - onRecoveryWithheld: noteAdapterRecoveryWithheld, - stream: parsed.stream, - executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { - pacingSlotAcquired: true, - dispatchOverride: oauthDispatch(builtInitialRequest), - providerName: route.providerName, - modelId: route.modelId, - }), - }); + upstreamResponse = await withProviderRequestSlot(route.providerName, route.provider, route.modelId, upstream.signal, pacingSlot => + transportState.activeAdapter.fetchResponse!(builtInitialRequest, { + abortSignal: upstream.signal, + timeoutMs: connectMs, + sendBudget: adapterDispatchBudget, + onPhysicalSend: send => noteAdapterPhysicalSend(inputTokenEstimate, send), + onRecoveryWithheld: noteAdapterRecoveryWithheld, + stream: parsed.stream, + executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { + pacingSlotAcquired: true, + pacingSlot, + dispatchOverride: oauthDispatch(builtInitialRequest), + providerName: route.providerName, + modelId: route.modelId, + }), + })); } else { // #1851 scope guard: transient-5xx retry on this generic adapter path is opt-in for // direct Google AI Studio only (Vertex/Antigravity use fetchResponse above). Other @@ -460,28 +461,30 @@ export async function prepareAdapterExchange( try { if (transportState.activeAdapter.fetchResponse) { transportState.noteRoutedAttemptSend(retryEstimate, recovery); - await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, upstream.signal); - // The dispatch boundary is HERE, not before the pacing wait: that wait can reject for - // an abort, a saturated queue, an expired slot or a removed provider, and none of - // those reach the wire. Confirming earlier would hold the charge for a send that the - // pacer refused. - onDispatch?.(); - return await transportState.activeAdapter.fetchResponse(retryRequest, { - abortSignal: upstream.signal, - timeoutMs: connectMs, - sendBudget: adapterDispatchBudget, - onPhysicalSend: send => { - noteAdapterPhysicalSend(retryEstimate, send); - chargeFastDowngradeWorkflowSend(); - }, - onRecoveryWithheld: noteAdapterRecoveryWithheld, - stream: parsed.stream, - executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { - pacingSlotAcquired: true, - dispatchOverride: oauthDispatch(retryRequest), - providerName: route.providerName, - modelId: route.modelId, - }), + return await withProviderRequestSlot(route.providerName, route.provider, route.modelId, upstream.signal, pacingSlot => { + // The dispatch boundary is HERE, not before the pacing wait: that wait can reject for + // an abort, a saturated queue, an expired slot or a removed provider, and none of + // those reach the wire. Confirming earlier would hold the charge for a send that the + // pacer refused. + onDispatch?.(); + return transportState.activeAdapter.fetchResponse!(retryRequest, { + abortSignal: upstream.signal, + timeoutMs: connectMs, + sendBudget: adapterDispatchBudget, + onPhysicalSend: send => { + noteAdapterPhysicalSend(retryEstimate, send); + chargeFastDowngradeWorkflowSend(); + }, + onRecoveryWithheld: noteAdapterRecoveryWithheld, + stream: parsed.stream, + executor: providerFetch(route.provider, options.codexWsRuntimeIdentity, { + pacingSlotAcquired: true, + pacingSlot, + dispatchOverride: oauthDispatch(retryRequest), + providerName: route.providerName, + modelId: route.modelId, + }), + }); }); } // #2643 review: this leg used to call fetchWithHeaderTimeout directly, so an diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index 1112d535a04..7bae07896f1 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -8,7 +8,12 @@ import { } from "./ws-upstream"; import type { OcxProviderConfig } from "../../types"; import type { WsData } from "../ws-bridge"; -import { waitForProviderRequestSlot } from "../../providers/request-pacing"; +import { + requestPacingMaxConcurrentRequests, + sendTrackingRequestSlot, + waitForProviderRequestSlot, + type ProviderRequestSlot, +} from "../../providers/request-pacing"; import { withUpstreamHttpVersion } from "../../lib/upstream-http-version"; import type { CodexWsQuotaObserver } from "./codex-ws-metadata"; import { configuredOutboundFetch } from "../../lib/proxy-env"; @@ -27,6 +32,7 @@ export { withUpstreamHttpVersion }; const egressWebsocketDowngradeWarned = new Set(); /** A provider name is configuration-controlled, so the notice set is bounded like any cache. */ const EGRESS_DOWNGRADE_NOTICE_LIMIT = 64; +const pacingWebsocketDowngradeWarned = new Set(); /** * Marks an init whose provider egress route an outer physical-send boundary already decided. * @@ -61,6 +67,7 @@ function warnEgressWebsocketDowngradeOnce(providerName: string, egress: string): /** Test seam: the downgrade notice is once per provider per process, not once per request. */ export function __resetEgressWebsocketDowngradeNotices(): void { egressWebsocketDowngradeWarned.clear(); + pacingWebsocketDowngradeWarned.clear(); } export function disableResponsesRequestTimeout(req: Request, server: Pick, "timeout"> | undefined): boolean { @@ -121,7 +128,7 @@ export function wantsFreshConnection( export interface PaceAwareFetch { - waitForPacing?: (signal?: AbortSignal) => Promise; + waitForPacing?: (signal?: AbortSignal) => Promise; unpacedFetch?: typeof globalThis.fetch; } @@ -187,6 +194,9 @@ export interface ProviderFetchOptions { modelId?: string; /** One pacing slot was acquired immediately before this fetch wrapper was created. */ pacingSlotAcquired?: boolean; + pacingSlot?: ProviderRequestSlot; + /** A turn transport may issue interval-paced follow-up sends while its lease is held. */ + turnScopedPacing?: boolean; /** Captured selected-account observer, attached before the native WS send. */ onCodexWsQuota?: CodexWsQuotaObserver; /** Synchronous admission at actual credential dispatch, after pacing/backoff. */ @@ -271,6 +281,14 @@ export function providerFetch( const upstreamWebsocket = provider.upstreamWebsocket; if (!options.httpOnly && typeof input === "string" && init && shouldUseCodexWsUpstream(input, init, runtime, upstreamWebsocket)) { + if (options.providerName && requestPacingMaxConcurrentRequests(provider, options.modelId) > 0) { + if (!pacingWebsocketDowngradeWarned.has(providerName) + && pacingWebsocketDowngradeWarned.size < EGRESS_DOWNGRADE_NOTICE_LIMIT) { + pacingWebsocketDowngradeWarned.add(providerName); + console.warn(`[opencodex] provider ${JSON.stringify(redactSecretString(providerName))} has a request concurrency cap; serving over HTTP/SSE.`); + } + return httpFetch(input, init); + } const egress = egressFor(input); if (providerEgressIsExplicit(egress)) { warnEgressWebsocketDowngradeOnce(providerName, describeProviderEgressForLog(egress)); @@ -281,23 +299,30 @@ export function providerFetch( // request over HTTP, and dropping the provider's `upstreamHttpVersion` // there would silently negotiate a transport the operator ruled out. return codexWsUpstreamFetch(input, init, httpFetch, runtime, options.onCodexWsQuota, options.beforeDispatch, options.nativeControl, - () => waitForPacing(init.signal ?? undefined)); + async () => { (await waitForPacing(init.signal ?? undefined))?.release(); }); } return httpFetch(input, init); }; let pacingSlotAcquired = options.pacingSlotAcquired === true; - const waitForPacing = async (signal?: AbortSignal): Promise => { + const waitForPacing = async (signal?: AbortSignal): Promise => { if (pacingSlotAcquired) { pacingSlotAcquired = false; - return; + if (!options.pacingSlot && requestPacingMaxConcurrentRequests(provider, options.modelId) > 0) { + throw new Error("A pre-acquired concurrency slot must be passed to providerFetch"); + } + return options.pacingSlot; } if (options.providerName) { - await waitForProviderRequestSlot(options.providerName, provider, options.modelId, signal); + return waitForProviderRequestSlot(options.providerName, provider, options.modelId, signal, + options.turnScopedPacing && options.pacingSlot?.leased && !options.pacingSlot.released + ? { concurrency: false } : undefined); + } + if (requestPacingMaxConcurrentRequests(provider, options.modelId) > 0) { + throw new Error("A provider name is required for request concurrency pacing"); } }; const wrapped = async (input: Parameters[0], init?: RequestInit) => { - await waitForPacing(init?.signal ?? undefined); - return unpaced(input, init); + return sendTrackingRequestSlot(await waitForPacing(init?.signal ?? undefined), () => unpaced(input, init)); }; // The returned wrapper forwards its init down to `dispatch`, which applies the route at the // physical send. Adapters that hand this executor back as `provider.fetch` (Cursor does) @@ -345,8 +370,7 @@ export function storedPoolReplayDispatchNotifier( { preconnect: unpacedSource.preconnect }, ) as ProviderFetch["unpacedFetch"]; const wrapped = async (input: Parameters[0], init?: RequestInit) => { - await executor.waitForPacing?.(init?.signal ?? undefined); - return unpaced!(input, init); + return sendTrackingRequestSlot(await executor.waitForPacing?.(init?.signal ?? undefined), () => unpaced!(input, init)); }; return Object.assign(wrapped, { preconnect: executor.preconnect, @@ -370,20 +394,21 @@ export async function fetchWithHeaderTimeout( _manualRedirect = false, ): Promise { const pacing = executor as ProviderFetch; - await pacing.waitForPacing?.(abortSignal); + const slot = await pacing.waitForPacing?.(abortSignal); const fetchExecutor = pacing.unpacedFetch ?? executor; - const timeout = new AbortController(); - const timer = setTimeout(() => { - if (!timeout.signal.aborted) timeout.abort(new DOMException("Timeout elapsed", "TimeoutError")); - }, timeoutMs); - const headers = new Headers(init.headers); - // Compressed SSE can be held until the decompressor has a complete block. Streaming calls - // default to identity for low-latency frame delivery, while an explicit caller choice wins. - if (preferIdentityEncoding && !headers.has("accept-encoding")) { - headers.set("accept-encoding", "identity"); - } + let timer: ReturnType | undefined; try { - return await fetchExecutor(url, { + const timeout = new AbortController(); + timer = setTimeout(() => { + if (!timeout.signal.aborted) timeout.abort(new DOMException("Timeout elapsed", "TimeoutError")); + }, timeoutMs); + const headers = new Headers(init.headers); + // Compressed SSE can be held until the decompressor has a complete block. Streaming calls + // default to identity for low-latency frame delivery, while an explicit caller choice wins. + if (preferIdentityEncoding && !headers.has("accept-encoding")) { + headers.set("accept-encoding", "identity"); + } + return await sendTrackingRequestSlot(slot, () => fetchExecutor(url, { ...init, headers, // Never replay provider credentials or request bodies to a redirect destination. @@ -391,8 +416,9 @@ export async function fetchWithHeaderTimeout( redirect: "manual", signal: AbortSignal.any([abortSignal, timeout.signal]), timeout: 0, - }); + })); } finally { - clearTimeout(timer); + if (timer) clearTimeout(timer); + if (!slot?.bodyTracked) slot?.release(); } } diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index 9ee1dab7f67..5d17fde3a4b 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -8,6 +8,7 @@ import { credentialGeneration, } from "../../oauth/store"; import type { ProviderAdapter, AdapterRequest } from "../../adapters/base"; +import { releaseProviderRequestSlot, waitForProviderRequestSlot, type ProviderRequestSlot } from "../../providers/request-pacing"; import type { AdapterEvent, OcxParsedRequest, OcxProviderConfig, OcxUsage } from "../../types"; import type { AnthropicAccountSelectionReason } from "../../oauth/anthropic-routing"; import { @@ -379,24 +380,32 @@ export async function prepareResponsesTransport( if (!run) throw new Error("Selected provider no longer supports this turn transport"); let sent = false; let refused = false; + let attemptSlot: ProviderRequestSlot | undefined; // Both main and image-loop callers already acquired the initial pacing slot. // Subsequent physical messages retain this adapter/credential and are paced normally. - const fetch = providerFetch(route.provider, options.codexWsRuntimeIdentity, { - providerName: route.providerName, modelId: route.modelId, pacingSlotAcquired: true, - beforeDispatch: () => { - if (sent) return; - if (!selectionIsCurrent(binding)) { - refused = true; - throw new Error("Account selection changed before the first turn dispatch"); - } - commitKeyAttemptSend(); - sent = true; - }, - }); try { + attemptSlot = attempt === 0 + ? incoming.pacingSlot ?? await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, incoming.abortSignal) + : await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, incoming.abortSignal); + const fetch = providerFetch(route.provider, options.codexWsRuntimeIdentity, { + providerName: route.providerName, modelId: route.modelId, pacingSlotAcquired: true, + pacingSlot: attemptSlot, + turnScopedPacing: true, + beforeDispatch: () => { + if (sent) return; + if (!selectionIsCurrent(binding)) { + refused = true; + throw new Error("Account selection changed before the first turn dispatch"); + } + commitKeyAttemptSend(); + sent = true; + }, + }); await run(requestParsed, { ...incoming, providerFetch: fetch }, event => { if (!refused) emit(event); }); } catch (error) { if (!refused) throw error; + } finally { + releaseProviderRequestSlot(attemptSlot); } if (!refused) return; // The adapter may map the guard's exception to an error event. Neither that diff --git a/src/server/responses/run-turn-execution.ts b/src/server/responses/run-turn-execution.ts index 5659c0970b9..8f6da738d7f 100644 --- a/src/server/responses/run-turn-execution.ts +++ b/src/server/responses/run-turn-execution.ts @@ -13,7 +13,7 @@ import { adapterResponseReachedServingTerminal, } from "./core-replay"; import { noteAttemptRecoveryWithheld, sealRequestAttemptIdentity, recordAttemptCredentialSource } from "../request-log"; -import { waitForProviderRequestSlot, RequestPacingQueueOverloadError } from "../../providers/request-pacing"; +import { releaseProviderRequestSlot, waitForProviderRequestSlot, RequestPacingQueueOverloadError, type ProviderRequestSlot } from "../../providers/request-pacing"; import type { AdapterEventQueue } from "../../adapters/run-turn-queue"; import type { AttemptRecoveryKind } from "../../usage/log"; import { providerFetch } from "./fetch-helpers"; @@ -187,8 +187,9 @@ export async function executeResponsesRunTurn( }; // Initial admission must settle before the streaming Response commits HTTP 200. // Let the outer Responses facade preserve the local retryable-429 contract. + let initialPacingSlot: ProviderRequestSlot; try { - await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); + initialPacingSlot = await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); } catch (error) { cleanupRunTurnAbort(); queue.close(); @@ -214,11 +215,13 @@ export async function executeResponsesRunTurn( // rounds carry the grown message history and adjusted tool list while // selection/replay binding stays on the request's own parsed object. turnParsed: PreparedResponsesRequest["parsed"] = parsed, + preacquiredSlot?: ProviderRequestSlot, ): Promise => { const attemptSeq = ++runTurnAttemptSeq; + let pacingSlot = preacquiredSlot; try { if (!pacingSlotAcquired) { - await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); + pacingSlot = await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); } await refreshRunTurnSelection(); // LOCAL PATCH (runturn-websearch): refreshRunTurnSelection binds route @@ -244,6 +247,8 @@ export async function executeResponsesRunTurn( // Cursor HTTP/1.1 consumes it for RunSSE; every BidiAppend and redial then waits on // the same provider queue through this stateful wrapper. pacingSlotAcquired: true, + pacingSlot, + turnScopedPacing: true, }, ); await transportState.runTurnAdapter.runTurn?.( @@ -253,6 +258,7 @@ export async function executeResponsesRunTurn( abortSignal: runTurnAbort.signal, translatorBudget, providerFetch: runTurnProviderFetch, + pacingSlot, // The only way the request budget reaches a transport the adapter owns. Without it // a Cursor turn's inner ladder was three physical sends the cap read as one. ...(adapterDispatchBudget ? { sendBudget: adapterDispatchBudget } : {}), @@ -304,6 +310,7 @@ export async function executeResponsesRunTurn( message: err instanceof Error ? err.message : String(err), }); } finally { + releaseProviderRequestSlot(pacingSlot); // Cursor assigns a stable conversation id inside runTurn on the first headerless // turn; backfill so Logs can filter/total that opening request (#330 / #522). if (!logCtx.conversationId && parsed._cursorConversationId) { @@ -316,7 +323,7 @@ export async function executeResponsesRunTurn( // synthetic web_search tool; later iterations get their own queue so the // search loop can buffer each turn's events before deciding to intercept. const wsFirstParsed = wsPlan ? runTurnWebSearchInitialParsed(parsed) : parsed; - const runTurn = async (): Promise => runTurnAttempt(queue, undefined, true, wsFirstParsed); + const runTurn = async (): Promise => runTurnAttempt(queue, undefined, true, wsFirstParsed, initialPacingSlot); const runTurnFailoverArmed = () => route.provider.authMode === "oauth" || !!(transportState.genericFailoverAccountId diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 57b2193f1f3..4a874b48e22 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -143,9 +143,12 @@ Provider request pacing in `src/providers/request-pacing.ts` combines start inte `maxConcurrentRequests` limits. Provider capacity is shared across models; exact-model limits apply in addition to that capacity. Admission reserves both counters atomically, and eligible sibling models may bypass a saturated model lane. Releases are idempotent, wake queued requests, -and retain interval deadlines. Active aborts release capacity; dispatch owners release on response -body completion, cancellation, or failure. Capacity waits use the same bounded queue and retryable -queue-overload errors as interval waits. +and retain interval deadlines. A lease follows each physical HTTP send through response-body +completion, error, or cancellation; failed dispatch and active abort also return it. Unconsumed +or inactive bodies are cancelled after a bounded deadline so a dropped response cannot hold +capacity indefinitely. A capped canonical Codex WebSocket turn uses HTTP/SSE because the socket +has no response-body lifecycle to return the lease. Capacity waits use the same bounded queue +and retryable queue-overload errors as interval waits. ## TypeSafe JEV decision provider diff --git a/tests/adapters/physical-send.test.ts b/tests/adapters/physical-send.test.ts index 1b76ce0e4a1..236f83ce055 100644 --- a/tests/adapters/physical-send.test.ts +++ b/tests/adapters/physical-send.test.ts @@ -1,10 +1,14 @@ -import { describe, expect, test } from "bun:test"; +import { afterEach, describe, expect, test } from "bun:test"; import { createAdapterPhysicalSend } from "../../src/adapters/physical-send"; import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; import { SendBudgetExhaustedError } from "../../src/lib/upstream-retry"; import { budgetOwner } from "../helpers/send-budget-owner"; +import { resetProviderRequestPacingForTest } from "../../src/providers/request-pacing"; +import { providerFetch } from "../../src/server/responses/fetch-helpers"; +import type { OcxProviderConfig } from "../../src/types"; const url = "https://adapter-fixture.invalid/inference"; +afterEach(() => resetProviderRequestPacingForTest()); /** * A credential hop has already reserved the replay it hands to the adapter, so the adapter's @@ -23,6 +27,22 @@ function prepaid() { } describe("adapter physical inference admission", () => { + test("returns a concurrency lease after body completion and a pre-dispatch refusal", async () => { + const configured = { + adapter: "openai-chat", baseUrl: "https://adapter-fixture.invalid", + requestPacing: { enabled: true, maxConcurrentRequests: 1 }, + fetch: Object.assign(async () => new Response("ok"), { preconnect() {} }) as typeof fetch, + } as OcxProviderConfig & { fetch: typeof fetch }; + const executor = providerFetch(configured, undefined, { providerName: "physical", modelId: "a" }); + const controller = new AbortController(); + const send = createAdapterPhysicalSend({ abortSignal: controller.signal }, executor); + await expect(send({ url, beforeDispatch: () => { throw new Error("refused"); }, + dispatch: physical => physical(url) })).rejects.toThrow("refused"); + const first = await send({ url, dispatch: physical => physical(url) }); + expect(await first.text()).toBe("ok"); + expect(await (await send({ url, dispatch: physical => physical(url) })).text()).toBe("ok"); + }); + test("a prepaid scope admits exactly one physical send and rejects replay before backoff", async () => { const { parent, scope, dispose } = prepaid(); let sends = 0, waits = 0, pacingSlots = 0; diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index a419094b1b4..96dcb47a84e 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1013,6 +1013,7 @@ "management-provider-atomicity.test.ts": "server", "management-provider-pinsless-validation.test.ts": "server", "management-provider-proto-override.test.ts": "server", + "management-provider-request-pacing.test.ts": "server", "management-provider-reasoning-lists.test.ts": "server", "management-provider-reset-replay.test.ts": "server", "management-provider-synthetic-max.test.ts": "server", diff --git a/tests/server/management-provider-request-pacing.test.ts b/tests/server/management-provider-request-pacing.test.ts new file mode 100644 index 00000000000..f42b0e3a722 --- /dev/null +++ b/tests/server/management-provider-request-pacing.test.ts @@ -0,0 +1,118 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { existsSync, mkdirSync, mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { loadConfig, saveConfig } from "../../src/config"; +import { handleManagementAPI } from "../../src/server/management-api"; +import type { OcxConfig } from "../../src/types"; +import { catalogConvergenceFactory } from "../helpers/catalog-convergence"; +import { ManagementRequest as Request } from "../helpers/management-auth"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const TEST_DIR = mkdtempSync(join(tmpdir(), "ocx-management-request-pacing-")); +const previousOpencodexHome = process.env.OPENCODEX_HOME; +let isolatedCodexHome: IsolatedCodexHome | null = null; +beforeEach(() => { isolatedCodexHome = installIsolatedCodexHome("ocx-pacing-home-"); }); +afterEach(() => { + if (previousOpencodexHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousOpencodexHome; + isolatedCodexHome?.restore(); + isolatedCodexHome = null; + if (existsSync(TEST_DIR)) removeTreeWithRetry(TEST_DIR); +}); + +describe("provider management request pacing", () => { + test("provider request pacing PATCH persists provider and model limits without catalog churn", async () => { + if (existsSync(TEST_DIR)) removeTreeWithRetry(TEST_DIR); + mkdirSync(TEST_DIR, { recursive: true }); + process.env.OPENCODEX_HOME = TEST_DIR; + const liveConfig: OcxConfig = { + port: 0, + hostname: "127.0.0.1", + defaultProvider: "nvidia", + providers: { + nvidia: { + adapter: "openai-chat", + baseUrl: "https://integrate.api.nvidia.com/v1", + apiKey: "sk-nvidia", + }, + }, + }; + saveConfig(liveConfig); + let catalogRefreshes = 0; + const request = async (path: string, init?: RequestInit) => { + const req = new Request(`http://127.0.0.1${path}`, init); + return handleManagementAPI(req, new URL(req.url), liveConfig, { + createManagementConvergeCodex: catalogConvergenceFactory(() => { catalogRefreshes += 1; }), + }); + }; + const policy = { + enabled: true, + requestsPerMinute: 38, + minIntervalMs: 1_600, + maxConcurrentRequests: 4, + models: { "deepseek-ai/deepseek-v4-flash-0731": { requestsPerMinute: 10 } }, + }; + + const saved = await request("/api/providers?name=nvidia", { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: policy }), + }); + expect(saved?.status).toBe(200); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); + expect(loadConfig().providers.nvidia?.requestPacing).toEqual(policy); + expect(catalogRefreshes).toBe(0); + + const providers = await request("/api/providers"); + expect((await providers?.json()).find((row: { name: string }) => row.name === "nvidia").requestPacing).toEqual(policy); + const status = await request("/api/provider-request-pacing?name=nvidia"); + expect(await status?.json()).toMatchObject({ provider: "nvidia", enabled: true, queued: 0, nextSlotInMs: 0 }); + + const invalid = await request("/api/providers?name=nvidia", { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: { enabled: true, requestsPerMinute: -1 } }), + }); + expect(invalid?.status).toBe(400); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); + + const timerOverflow = await request("/api/providers?name=nvidia", { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: { enabled: true, requestsPerMinute: 0.001 } }), + }); + expect(timerOverflow?.status).toBe(400); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); + + const concurrencyOnly = await request("/api/providers?name=nvidia", { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: { + enabled: true, + maxConcurrentRequests: 2, + models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, + } }), + }); + expect(concurrencyOnly?.status).toBe(200); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual({ + enabled: true, + maxConcurrentRequests: 2, + models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, + }); + + const invalidConcurrency = await request("/api/providers?name=nvidia", { + method: "PATCH", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: { enabled: true, maxConcurrentRequests: 0 } }), + }); + expect(invalidConcurrency?.status).toBe(400); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual({ + enabled: true, + maxConcurrentRequests: 2, + models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, + }); + }); + +}); diff --git a/tests/server/management-provider-validation.test.ts b/tests/server/management-provider-validation.test.ts index 06eb65c39bf..1e09a68bd0e 100644 --- a/tests/server/management-provider-validation.test.ts +++ b/tests/server/management-provider-validation.test.ts @@ -1020,98 +1020,6 @@ describe("provider management validation", () => { expect(error).toContain("[REDACTED]"); }); - test("provider request pacing PATCH persists provider and model limits without catalog churn", async () => { - if (existsSync(TEST_DIR)) removeTreeWithRetry(TEST_DIR); - mkdirSync(TEST_DIR, { recursive: true }); - process.env.OPENCODEX_HOME = TEST_DIR; - const liveConfig: OcxConfig = { - port: 0, - hostname: "127.0.0.1", - defaultProvider: "nvidia", - providers: { - nvidia: { - adapter: "openai-chat", - baseUrl: "https://integrate.api.nvidia.com/v1", - apiKey: "sk-nvidia", - }, - }, - }; - saveConfig(liveConfig); - let catalogRefreshes = 0; - const request = async (path: string, init?: RequestInit) => { - const req = new Request(`http://127.0.0.1${path}`, init); - return handleManagementAPI(req, new URL(req.url), liveConfig, { - createManagementConvergeCodex: catalogConvergenceFactory(() => { catalogRefreshes += 1; }), - }); - }; - const policy = { - enabled: true, - requestsPerMinute: 38, - minIntervalMs: 1_600, - maxConcurrentRequests: 4, - models: { "deepseek-ai/deepseek-v4-flash-0731": { requestsPerMinute: 10 } }, - }; - - const saved = await request("/api/providers?name=nvidia", { - method: "PATCH", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ requestPacing: policy }), - }); - expect(saved?.status).toBe(200); - expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); - expect(loadConfig().providers.nvidia?.requestPacing).toEqual(policy); - expect(catalogRefreshes).toBe(0); - - const providers = await request("/api/providers"); - expect((await providers?.json()).find((row: { name: string }) => row.name === "nvidia").requestPacing).toEqual(policy); - const status = await request("/api/provider-request-pacing?name=nvidia"); - expect(await status?.json()).toMatchObject({ provider: "nvidia", enabled: true, queued: 0, nextSlotInMs: 0 }); - - const invalid = await request("/api/providers?name=nvidia", { - method: "PATCH", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ requestPacing: { enabled: true, requestsPerMinute: -1 } }), - }); - expect(invalid?.status).toBe(400); - expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); - - const timerOverflow = await request("/api/providers?name=nvidia", { - method: "PATCH", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ requestPacing: { enabled: true, requestsPerMinute: 0.001 } }), - }); - expect(timerOverflow?.status).toBe(400); - expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); - - const concurrencyOnly = await request("/api/providers?name=nvidia", { - method: "PATCH", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ requestPacing: { - enabled: true, - maxConcurrentRequests: 2, - models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, - } }), - }); - expect(concurrencyOnly?.status).toBe(200); - expect(liveConfig.providers.nvidia?.requestPacing).toEqual({ - enabled: true, - maxConcurrentRequests: 2, - models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, - }); - - const invalidConcurrency = await request("/api/providers?name=nvidia", { - method: "PATCH", - headers: { "content-type": "application/json" }, - body: JSON.stringify({ requestPacing: { enabled: true, maxConcurrentRequests: 0 } }), - }); - expect(invalidConcurrency?.status).toBe(400); - expect(liveConfig.providers.nvidia?.requestPacing).toEqual({ - enabled: true, - maxConcurrentRequests: 2, - models: { "deepseek-ai/deepseek-v4-flash-0731": { maxConcurrentRequests: 1 } }, - }); - }); - test("provider discovery status is additive and omitted before an attempt", async () => { markProviderDiscoveryFailed("auth-broken", { reason: "http", httpStatus: 401 }); try { diff --git a/tests/usage/request-pacing.test.ts b/tests/usage/request-pacing.test.ts index bea2bfb6286..6a7280f2ffa 100644 --- a/tests/usage/request-pacing.test.ts +++ b/tests/usage/request-pacing.test.ts @@ -325,6 +325,83 @@ describe("provider request pacing queue", () => { }); describe("provider request concurrency", () => { + test("an active body holds capacity until cancellation", async () => { + let sends = 0; + const fetchImpl = Object.assign(async () => { + sends += 1; + return new Response(new ReadableStream({ start() {} })); + }, { preconnect() {} }) as typeof globalThis.fetch; + const configured = { ...provider({ enabled: true, maxConcurrentRequests: 1 }), fetch: fetchImpl }; + const executor = providerFetch(configured, undefined, { providerName: "demo", modelId: "a" }); + const first = await executor("https://example.test/first"); + const second = executor("https://example.test/second", { signal: AbortSignal.timeout(250) }); + await Bun.sleep(0); + expect(sends).toBe(1); + expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); + await first.body!.cancel(); + const resumed = await second; + expect(sends).toBe(2); + await resumed.body!.cancel(); + }); + + test("an errored body and failed send return their leases", async () => { + let streamController: ReadableStreamDefaultController | undefined; + let sends = 0; + const fetchImpl = Object.assign(async () => { + sends += 1; + if (sends === 2) throw new Error("send failed"); + return new Response(new ReadableStream({ + start(controller) { streamController = controller; }, + })); + }, { preconnect() {} }) as typeof globalThis.fetch; + const configured = { ...provider({ enabled: true, maxConcurrentRequests: 1 }), fetch: fetchImpl }; + const executor = providerFetch(configured, undefined, { providerName: "demo", modelId: "a" }); + const first = await executor("https://example.test/first"); + const reading = first.text(); + streamController!.error(new Error("body failed")); + await expect(reading).rejects.toThrow("body failed"); + await expect(executor("https://example.test/failed", { signal: AbortSignal.timeout(250) })).rejects.toThrow("send failed"); + const third = await executor("https://example.test/third", { signal: AbortSignal.timeout(250) }); + expect(sends).toBe(3); + await third.body!.cancel(); + }); + + test("a pre-acquired lease transfers to the response body", async () => { + const fetchImpl = Object.assign(async () => new Response("ok"), { preconnect() {} }) as typeof globalThis.fetch; + const configured = { ...provider({ enabled: true, maxConcurrentRequests: 1 }), fetch: fetchImpl }; + const pacingSlot = await waitForProviderRequestSlot("demo", configured, "a"); + const executor = providerFetch(configured, undefined, { + providerName: "demo", modelId: "a", pacingSlotAcquired: true, pacingSlot, + }); + expect(await (await executor("https://example.test/first")).text()).toBe("ok"); + expect(await (await executor("https://example.test/second", { signal: AbortSignal.timeout(250) })).text()).toBe("ok"); + }); + + test("completed streamed response returns capacity for the next physical send", async () => { + let sends = 0; + const fetchImpl = Object.assign(async () => { + sends += 1; + return new Response(new ReadableStream({ + start(controller) { + controller.enqueue(new TextEncoder().encode("ok")); + controller.close(); + }, + })); + }, { preconnect() {} }) as typeof globalThis.fetch; + const configured = { + ...provider({ enabled: true, maxConcurrentRequests: 1 }), + fetch: fetchImpl, + } as OcxProviderConfig & { fetch: typeof globalThis.fetch }; + const executor = providerFetch(configured, undefined, { providerName: "demo", modelId: "a" }); + const first = await fetchWithHeaderTimeout("https://example.test/v1/first", {}, new AbortController().signal, 1_000, false, executor); + expect(await first.text()).toBe("ok"); + + const secondSignal = AbortSignal.timeout(250); + const second = await fetchWithHeaderTimeout("https://example.test/v1/second", {}, secondSignal, 1_000, false, executor); + expect(await second.text()).toBe("ok"); + expect(sends).toBe(2); + }); + test("caps all models together and release is idempotent", async () => { const configured = provider({ enabled: true, maxConcurrentRequests: 2 }); const first = await waitForProviderRequestSlot("demo", configured, "a"); @@ -332,14 +409,14 @@ describe("provider request concurrency", () => { const third = waitForProviderRequestSlot("demo", configured, "c"); const fourth = waitForProviderRequestSlot("demo", configured, "d"); expect(providerRequestPacingStatus("demo", configured).queued).toBe(2); - first(); - first(); + first.release(); + first.release(); const releaseThird = await third; expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); - second(); + second.release(); const releaseFourth = await fourth; - releaseThird(); - releaseFourth(); + releaseThird.release(); + releaseFourth.release(); }); test("model limit tightens provider cap without blocking eligible siblings", async () => { @@ -351,24 +428,24 @@ describe("provider request concurrency", () => { const anotherFast = await waitForProviderRequestSlot("demo", configured, "fast"); const queuedFast = waitForProviderRequestSlot("demo", configured, "fast"); expect(providerRequestPacingStatus("demo", configured).queued).toBe(2); - fast(); + fast.release(); const lastFast = await queuedFast; expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); - slow(); - (await queuedSlow)(); - anotherFast(); - lastFast(); + slow.release(); + (await queuedSlow).release(); + anotherFast.release(); + lastFast.release(); }); test("model-only cap leaves other models and providers independent", async () => { const configured = provider({ enabled: true, models: { slow: { maxConcurrentRequests: 1 } } }); const first = await waitForProviderRequestSlot("demo", configured, "slow"); const queued = waitForProviderRequestSlot("demo", configured, "slow"); - (await waitForProviderRequestSlot("demo", configured, "other"))(); - (await waitForProviderRequestSlot("other-provider", configured, "slow"))(); + (await waitForProviderRequestSlot("demo", configured, "other")).release(); + (await waitForProviderRequestSlot("other-provider", configured, "slow")).release(); expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); - first(); - (await queued)(); + first.release(); + (await queued).release(); }); test("aborting an active request frees exactly one slot", async () => { @@ -378,11 +455,11 @@ describe("provider request concurrency", () => { const next = waitForProviderRequestSlot("demo", configured, "a"); controller.abort(); const releaseNext = await next; - release(); + release.release(); const last = waitForProviderRequestSlot("demo", configured, "a"); expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); - releaseNext(); - (await last)(); + releaseNext.release(); + (await last).release(); }); test("concurrency wait expires without spinning and preserves active capacity", async () => { @@ -399,8 +476,8 @@ describe("provider request concurrency", () => { expect(clock.pendingTimerCount()).toBe(0); const next = waitForProviderRequestSlot("demo", configured); expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); - release(); - (await next)(); + release.release(); + (await next).release(); }); test("releasing capacity still honors the start interval", async () => { @@ -409,10 +486,10 @@ describe("provider request concurrency", () => { const configured = provider({ enabled: true, maxConcurrentRequests: 1, minIntervalMs: 100 }); const release = await waitForProviderRequestSlot("demo", configured); const next = waitForProviderRequestSlot("demo", configured); - release(); + release.release(); clock.advanceBy(99); expect(providerRequestPacingStatus("demo", configured).queued).toBe(1); clock.advanceBy(1); - (await next)(); + (await next).release(); }); }); From 689bd219188da980685fef110956f18a2a3a2721 Mon Sep 17 00:00:00 2001 From: luvs01 <27862058+luvs01@users.noreply.github.com> Date: Sun, 27 Sep 2026 02:44:24 +0900 Subject: [PATCH 03/19] feat(compaction): recover failed routed compaction with an opt-in emergency model (#5919) Carried from #5919 as one squashed commit. Co-authored-by: luvs01 <27862058+luvs01@users.noreply.github.com> --- .../content/docs/guides/codex-integration.md | 36 ++ scripts/test-layout/layout.json | 3 + src/config/diagnostics.ts | 3 +- src/config/load-degrade.ts | 2 + src/config/schema/compaction-recovery.ts | 15 + src/config/schema/config-schema.ts | 2 + src/server/management/config-routes.ts | 14 +- src/server/responses/adapter-delivery.ts | 8 +- src/server/responses/adapter-dispatch.ts | 19 +- src/server/responses/compact.ts | 2 +- .../responses/compaction-recovery-policy.ts | 116 ++++++ src/server/responses/compaction-recovery.ts | 323 ++++++++++++++++ src/server/responses/core-combo-failure.ts | 1 + src/server/responses/core-options.ts | 14 +- src/server/responses/core.ts | 6 +- src/server/responses/request-send-budget.ts | 3 +- src/server/responses/run-turn-execution.ts | 8 +- src/types/config.ts | 2 + structure/config.md | 8 +- structure/gui-and-management-api.md | 5 + structure/transports/responses-failover.md | 17 + .../compaction-recovery-settings.test.ts | 67 ++++ tests/fixtures/test-layout-expected.json | 3 + tests/helpers/responses-core-source.ts | 2 + ...sponses-compaction-recovery-policy.test.ts | 102 +++++ .../responses-compaction-recovery.test.ts | 348 ++++++++++++++++++ 26 files changed, 1110 insertions(+), 19 deletions(-) create mode 100644 src/config/schema/compaction-recovery.ts create mode 100644 src/server/responses/compaction-recovery-policy.ts create mode 100644 src/server/responses/compaction-recovery.ts create mode 100644 tests/config/compaction-recovery-settings.test.ts create mode 100644 tests/responses/responses-compaction-recovery-policy.test.ts create mode 100644 tests/responses/responses-compaction-recovery.test.ts diff --git a/docs-site/src/content/docs/guides/codex-integration.md b/docs-site/src/content/docs/guides/codex-integration.md index 712d5d9036c..eb236fff474 100644 --- a/docs-site/src/content/docs/guides/codex-integration.md +++ b/docs-site/src/content/docs/guides/codex-integration.md @@ -477,6 +477,42 @@ While the mode is active, the realtime voice sideband override (`experimental_realtime_ws_base_url`) is not injected — the dedicated provider-table form cannot carry it — so Codex Desktop voice uses its native endpoint rather than the proxy. +### Emergency compaction model (opt-in) + +`compactionRecovery` leaves the initial compaction on the conversation's selected route. It +permits one emergency attempt only after a supported, pre-output compaction failure. It is +separate from `compactionRouting`, which chooses another model before compaction starts, and +from `codexClientCompaction`, which changes Codex's provider form. + +```json +{ + "compactionRecovery": { + "enabled": true, + "model": "provider/emergency-model", + "allowDevinInvalidArgument": false + } +} +``` + +Use an independently configured, authorized model with enough context for the failed input. +Enabling recovery permits that model's provider to receive the compaction history and charge +for the extra attempt when recovery runs; ordinary successful compactions incur no extra call. +The option is off when absent or disabled. The existing authenticated management API accepts +this block through `PUT /api/settings`; send `compactionRecovery: null` to remove it. A direct +file edit should follow the normal stopped-proxy configuration workflow. This setting does not +change sign-in, the conversation's ordinary model, Codex's provider ID, or the desktop composer. + +Recovery does not replay after cancellation, semantic output, tool side effects, an exhausted +send budget, or an authentication, admission or policy refusal. Generic `400` errors do not +enable fallback. The separately opted-in Devin `invalid_argument` case applies only to an +identified compaction failure from that adapter. The emergency attempt shares the original +request's send budget and never starts a second recovery attempt. + +Native encrypted compaction is outside this recovery path: its original error is retained. +There is no automatic local truncation mode. A response being accepted is not proof that a +long conversation retained its goals; verify the next turn on the original model before +treating an emergency summary as a recovered task. + ### Authless Codex Desktop (opt-in) In **Dashboard → Overview**, **Open Codex without signing in** controls this existing diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index c6c7a6d0779..0a69213201d 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -168,6 +168,9 @@ } }, "explicit": { + "responses-compaction-recovery.test.ts": "responses", + "compaction-recovery-settings.test.ts": "config", + "responses-compaction-recovery-policy.test.ts": "responses", "deepseek-artifact-tool-schema.test.ts": "providers", "client-config-export-output-limit.test.ts": "config", "openai-chat-serialized-tool-call-scaling.test.ts": "adapters/openai", diff --git a/src/config/diagnostics.ts b/src/config/diagnostics.ts index deb761dc2e8..ca7b899e3d6 100644 --- a/src/config/diagnostics.ts +++ b/src/config/diagnostics.ts @@ -2,6 +2,7 @@ import { createHash } from "node:crypto"; import { lstatSync, readFileSync } from "node:fs"; import { join } from "node:path"; import * as z from "zod/v4"; +import { compactionRecoveryConfigError } from "./schema/compaction-recovery"; import type { OcxConfig } from "../types"; import { configReasoningPinsConfigError } from "./provider-validation"; import { loopbackCompanionAllowed } from "../codex/loopback-target"; @@ -598,7 +599,7 @@ export function validateConfigCandidate(value: unknown): { ok: true; config: Ocx if (compactionRouting !== undefined && !compactionRoutingSchema.safeParse(compactionRouting).success) { return { ok: false, error: "schema_invalid: compactionRouting: requires a nonblank model, an optional valid reasoningEffort, and optional non-repeating triggers drawn from \"manual\" and \"auto\"" }; } - const boundaryError = configReasoningPinsConfigError(value) + const boundaryError = compactionRecoveryConfigError(value) ?? configReasoningPinsConfigError(value) ?? blankHostnameError(value) ?? claudeSubagentEffortError(value) ?? appOwnedMemoryBudgetError(value) diff --git a/src/config/load-degrade.ts b/src/config/load-degrade.ts index 412d011b2f3..aa8cc5e94c4 100644 --- a/src/config/load-degrade.ts +++ b/src/config/load-degrade.ts @@ -1,5 +1,6 @@ import { chmodSync, existsSync } from "node:fs"; import { join } from "node:path"; +import { compactionRecoveryConfigError } from "./schema/compaction-recovery"; import { modelPinnedEffortsConfigError, pinnedReasoningEffortConfigError, @@ -118,6 +119,7 @@ export function warnDegradedCompactionRouting(rawParsed: unknown, validated: Ocx * the ratchet only ever moves down: a per-block call there costs a line the file does not have. */ export function warnDegradedTopLevelOptIns(rawParsed: unknown, validated: OcxConfig): void { + if (compactionRecoveryConfigError(rawParsed)) console.warn("⚠️ invalid compactionRecovery disabled; the original compaction failure is preserved"); warnDegradedStreamMode(rawParsed, validated); warnDegradedCompactionRouting(rawParsed, validated); } diff --git a/src/config/schema/compaction-recovery.ts b/src/config/schema/compaction-recovery.ts new file mode 100644 index 00000000000..0282074b722 --- /dev/null +++ b/src/config/schema/compaction-recovery.ts @@ -0,0 +1,15 @@ +import * as z from "zod/v4"; + +/** Failure-only, opt-in recovery; it never chooses the initial compaction model. */ +export const compactionRecoverySchema = z.object({ + enabled: z.boolean(), + model: z.string().trim().min(1).max(512).regex(/^[^\s\u0000-\u0020\u007f-\u009f]+$/), + allowDevinInvalidArgument: z.boolean().optional(), +}).strict(); + +export function compactionRecoveryConfigError(value: unknown): string | null { + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const recovery = (value as Record).compactionRecovery; + return recovery === undefined || compactionRecoverySchema.safeParse(recovery).success + ? null : "schema_invalid: compactionRecovery: requires enabled, a nonblank model, and optional boolean allowDevinInvalidArgument"; +} diff --git a/src/config/schema/config-schema.ts b/src/config/schema/config-schema.ts index c881a269fc1..48460d3667e 100644 --- a/src/config/schema/config-schema.ts +++ b/src/config/schema/config-schema.ts @@ -1,4 +1,5 @@ import * as z from "zod/v4"; +import { compactionRecoverySchema } from "./compaction-recovery"; import { agentTaskRecoverySchema, catalogAutoRefreshSchema, @@ -156,6 +157,7 @@ export const configSchema = z.object({ providers: z.record(z.string(), providerConfigSchema), modelPinnedEfforts: modelPinnedEffortsSchema.optional(), compactionRouting: compactionRoutingSchema.optional().catch(undefined), + compactionRecovery: compactionRecoverySchema.optional().catch(undefined), defaultProvider: z.string().min(1).default("openai"), defaultModelAliases: z.boolean().optional(), // Malformed hand edits disable this opt-in projection without rejecting providers. diff --git a/src/server/management/config-routes.ts b/src/server/management/config-routes.ts index d439abb137c..1efba0c4b88 100644 --- a/src/server/management/config-routes.ts +++ b/src/server/management/config-routes.ts @@ -1,4 +1,5 @@ import { compactionRoutingSchema } from "../../config/schema/leaf-validators"; +import { compactionRecoverySchema } from "../../config/schema/compaction-recovery"; import { captureConfigTopLevelRollback } from "../../config/rebase-provenance"; import type { IntegrationClientId } from "../../integrations/registry"; import { randomUUID } from "node:crypto"; @@ -369,6 +370,7 @@ export async function handleConfigRoutes(ctx: ManagementContext): Promise { try { - yield* readResponseStreamWithInactivity( + for await (const event of readResponseStreamWithInactivity( upstreamResponse, upstream.signal, bodyInactivityMs, response => transportState.activeAdapter.parseStream(response, translatorBudget, logCtx.activeTierMetadata), - ); + )) { + options.onCompactionRecoveryAdapterEvent?.(event); + yield event; + } } catch (error) { if (error instanceof ResponseBodyInactivityError) { yield { @@ -197,6 +200,7 @@ export async function deliverAdapterResponse( bodyInactivityMs, response => transportState.activeAdapter.parseResponse!(response, translatorBudget, logCtx.activeTierMetadata), ); + for (const event of initialEvents) options.onCompactionRecoveryAdapterEvent?.(event); let guardedEvents: AdapterEvent[]; if (terminalGuardEnabled) { guardedEvents = []; diff --git a/src/server/responses/adapter-dispatch.ts b/src/server/responses/adapter-dispatch.ts index eed634a8df9..0e179def836 100644 --- a/src/server/responses/adapter-dispatch.ts +++ b/src/server/responses/adapter-dispatch.ts @@ -321,11 +321,18 @@ export async function prepareAdapterExchange( // legacy direct-Google exception is preserved exactly; every other adapter still keeps // reset-only semantics so combo failover hops on the first 5xx. const transientPolicy = transientRetryPolicyFor(route.provider); + const compactPrepaid = options.compactionRecoveryAttempted ? sendBudgetState.pendingHopPermit : undefined; + if (compactPrepaid) sendBudgetState.pendingHopPermit = undefined; + let compactPrepaidUsed = false; const fetchWithRetryPolicy = (route.provider.adapter === "google" || transientPolicy) ? fetchWithTransientRetry : fetchWithResetRetry; upstreamResponse = await fetchWithRetryPolicy( recovery => { + if (compactPrepaid && !compactPrepaidUsed) { + if (!compactPrepaid.use()) throw new SendBudgetExhaustedError(safeHostLabel(builtInitialRequest.url)); + compactPrepaidUsed = true; + } transportState.noteRoutedAttemptSend(inputTokenEstimate, recovery); return fetchWithHeaderTimeout(builtInitialRequest.url, applyUpstreamRecoveryInit({ method: builtInitialRequest.method, @@ -341,12 +348,15 @@ export async function prepareAdapterExchange( { abortSignal: upstream.signal, label: safeHostLabel(builtInitialRequest.url), - ...(transientPolicy + ...(transientPolicy || compactPrepaid // Draws the remainder, not the raw policy. A combo child inherits the parent's // holder but used to take a fresh full allowance on its own first send, so the // shared counter was inherited without ever being read as a limit. ? { - attempts: remainingTransientSendBudget(transientPolicy.attempts), + // The first emergency send is already paid for. Only retries consume the + // remaining allowance; treating the booking as unavailable blocks a cap of two. + attempts: Math.min(transientPolicy?.attempts ?? 1, + remainingTransientSendBudget(transientPolicy?.attempts ?? 1) + (compactPrepaid ? 1 : 0)), onSendsConsumed: noteTransientSends, } : {}), @@ -1105,6 +1115,11 @@ export async function prepareAdapterExchange( // material before it reaches the client-facing error surface. const upstreamRetryAfter = upstreamResponse.headers.get("retry-after"); const normalized = normalizeUpstreamErrorText(errorText, "unknown error"); + options.onCompactionRecoveryAdapterEvent?.({ + type: "error", status: upstreamResponse.status, + errorType: normalized.type, code: normalized.code, + message: "Structured upstream failure observed before client formatting", + }); const message = normalized.cyberPolicy ? normalized.message ?? (isCyberPolicyCode(normalized.code) ? CYBER_POLICY_FALLBACK_MESSAGE : normalized.safeText) diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index c9a3cc5e495..33e57939c9e 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -1433,7 +1433,7 @@ export async function handleResponsesCompact( // The routed compaction turn is a handoff inside the same logical request, so it draws the // REMAINDER. Minting here is what let a native attempt spend three sends and the routed // fallback spend four more. - const response = await handleResponses(internalReq, config, logCtx, { abortSignal: req.signal, turnAdmissionLease, sendBudget, compactionRoutingOverride: options.compactionRoutingOverride, ...(admission ? { admission } : {}) }); + const response = await handleResponses(internalReq, config, logCtx, { abortSignal: req.signal, turnAdmissionLease, sendBudget, compactionRecoveryKind: "compaction-v1", compactionRoutingOverride: options.compactionRoutingOverride, ...(admission ? { admission } : {}) }); if (!response.ok) return response; let json: { output?: unknown[]; status?: unknown; error?: unknown }; if (response.headers.get("content-type")?.includes("text/event-stream")) { diff --git a/src/server/responses/compaction-recovery-policy.ts b/src/server/responses/compaction-recovery-policy.ts new file mode 100644 index 00000000000..7d74e176496 --- /dev/null +++ b/src/server/responses/compaction-recovery-policy.ts @@ -0,0 +1,116 @@ +/** Pure eligibility policy. The caller owns admission, buffering, dispatch and the shared budget. */ +export interface CompactionRecoveryConfig { + enabled: true; + model: string; + allowDevinInvalidArgument?: boolean; +} + +/** Evidence must describe the failed attempt, never request text or a guessed error message. */ +export interface CompactionRecoveryEvidence { + requestKind: "ordinary" | "compaction-v1" | "compaction-v2"; + recoveryAttempts: number; + cancelled: boolean; + nonReplayable: boolean; + /** Any semantic output observed, including output retained privately before delivery. */ + partialOutput: boolean; + toolEffects: boolean; + remainingSends: number; + /** Canonical serving identities resolved by the caller, including provider/account identity. */ + originalModel: string; + fallbackModel: string; + /** Actual serving adapter/provider, not a prefix inferred from the requested selector. */ + provider: string; + httpStatus?: number; + /** A completed result is never retried. Output validation belongs to the caller. */ + responseStatus: "completed" | "failed" | "incomplete" | "unknown"; + errorCode?: string; + errorType?: string; + authenticationDenied: boolean; + policyDenied: boolean; + budgetDenied: boolean; + refusal: boolean; + upstreamFailure: boolean; +} + +export type CompactionRecoveryDecision = + | { recover: true; model: string; reason: "context-overflow" | "compaction-output" | "upstream-unavailable" | "devin-invalid-argument" } + | { recover: false; reason: "disabled" | "invalid-evidence" | "ordinary-request" | "already-attempted" | "cancelled" | "unsafe-replay" | "protected-failure" | "budget-exhausted" | "same-model" | "succeeded" | "unclassified-failure" }; + +const MODEL_LIMIT = 512; +const safeName = (value: unknown, limit: number): value is string => + typeof value === "string" && value.length > 0 && value.length <= limit + && value === value.trim() && !/[\u0000-\u0020\u007f-\u009f]/.test(value); + +/** Invalid persisted values disable recovery rather than widening it. No configurable retry count. */ +export function readCompactionRecoveryConfig(value: unknown): CompactionRecoveryConfig | null { + if (!value || typeof value !== "object" || Array.isArray(value)) return null; + const raw = value as Record; + if (Object.keys(raw).some(key => !["enabled", "model", "allowDevinInvalidArgument"].includes(key)) + || raw.enabled !== true || !safeName(raw.model, MODEL_LIMIT) + || (raw.allowDevinInvalidArgument !== undefined && typeof raw.allowDevinInvalidArgument !== "boolean")) return null; + return { + enabled: true, + model: raw.model, + ...(raw.allowDevinInvalidArgument !== undefined ? { allowDevinInvalidArgument: raw.allowDevinInvalidArgument } : {}), + }; +} + +// Structured tokens only. The normalizer must also supply the explicit denial flags above. +const PROTECTED_CODE = /(?:^|_)(?:auth|authentication|authorization|unauthenticated|unauthorized|permission|forbidden|policy|refusal|refused|budget|quota|billing|safety|content_filter|origin_rejected|admission|scope)(?:_|$)/; +const CONTEXT_CODES = new Set(["context_length_exceeded", "context_window_exceeded", "input_too_long"]); +const COMPACTION_CODES = new Set(["compaction_failed", "invalid_compaction_output", "empty_compaction_output"]); +const SERVER_CODES = new Set(["upstream_error", "upstream_server_error", "server_is_overloaded", "internal", "internal_error", "server_error", "unavailable", "service_unavailable", "gateway_timeout"]); + +/** Decides eligibility only; returning true neither spends nor grants another send. */ +export function decideCompactionRecovery( + value: unknown, + evidence: CompactionRecoveryEvidence, +): CompactionRecoveryDecision { + const config = readCompactionRecoveryConfig(value); + if (!config) return { recover: false, reason: "disabled" }; + const flags = [evidence.cancelled, evidence.nonReplayable, evidence.partialOutput, evidence.toolEffects, + evidence.authenticationDenied, evidence.policyDenied, evidence.budgetDenied, evidence.refusal, evidence.upstreamFailure]; + if (flags.some(flag => typeof flag !== "boolean") + || !Number.isSafeInteger(evidence.recoveryAttempts) || evidence.recoveryAttempts < 0 + || !Number.isSafeInteger(evidence.remainingSends) || evidence.remainingSends < 0 + || !safeName(evidence.originalModel, MODEL_LIMIT) || !safeName(evidence.fallbackModel, MODEL_LIMIT) + || !safeName(evidence.provider, 128) + || !["ordinary", "compaction-v1", "compaction-v2"].includes(evidence.requestKind) + || !["completed", "failed", "incomplete", "unknown"].includes(evidence.responseStatus) + || (evidence.httpStatus !== undefined && (!Number.isInteger(evidence.httpStatus) || evidence.httpStatus < 100 || evidence.httpStatus > 599)) + || [evidence.errorCode, evidence.errorType].some(code => code !== undefined && !safeName(code, 128))) { + return { recover: false, reason: "invalid-evidence" }; + } + if (evidence.requestKind === "ordinary") return { recover: false, reason: "ordinary-request" }; + if (evidence.recoveryAttempts !== 0) return { recover: false, reason: "already-attempted" }; + const status = evidence.httpStatus; + const codes = [evidence.errorCode, evidence.errorType].filter((code): code is string => code !== undefined); + if (evidence.cancelled || status === 499 || codes.some(code => code === "cancelled" || code === "canceled" || code === "client_cancelled")) { + return { recover: false, reason: "cancelled" }; + } + if (evidence.nonReplayable || evidence.partialOutput || evidence.toolEffects) return { recover: false, reason: "unsafe-replay" }; + if (evidence.authenticationDenied || evidence.policyDenied || evidence.budgetDenied || evidence.refusal + || status === 401 || status === 403 || status === 402 || status === 429 + || codes.some(code => PROTECTED_CODE.test(code.toLowerCase()) || code === "failed_precondition")) { + return { recover: false, reason: "protected-failure" }; + } + if (evidence.remainingSends === 0) return { recover: false, reason: "budget-exhausted" }; + if (evidence.originalModel === evidence.fallbackModel) return { recover: false, reason: "same-model" }; + if (evidence.responseStatus === "completed") return { recover: false, reason: "succeeded" }; + const failedHttp = status !== undefined && status >= 400; + if (!evidence.upstreamFailure || (!failedHttp && evidence.responseStatus !== "failed")) { + return { recover: false, reason: "unclassified-failure" }; + } + const code = evidence.errorCode; + const requestFailure = status === undefined || status === 200 || status === 400 || status === 413 || status === 422 || status >= 500; + if (requestFailure && code && CONTEXT_CODES.has(code)) return { recover: true, model: config.model, reason: "context-overflow" }; + if (requestFailure && code && COMPACTION_CODES.has(code)) return { recover: true, model: config.model, reason: "compaction-output" }; + if (config.allowDevinInvalidArgument === true && evidence.provider === "devin" && code === "invalid_argument" + && (evidence.responseStatus === "failed" || (status !== undefined && status >= 400 && status < 500))) { + return { recover: true, model: config.model, reason: "devin-invalid-argument" }; + } + if (status !== undefined && status >= 500 && (code === undefined || SERVER_CODES.has(code))) { + return { recover: true, model: config.model, reason: "upstream-unavailable" }; + } + return { recover: false, reason: "unclassified-failure" }; +} diff --git a/src/server/responses/compaction-recovery.ts b/src/server/responses/compaction-recovery.ts new file mode 100644 index 00000000000..d04cd08d8c3 --- /dev/null +++ b/src/server/responses/compaction-recovery.ts @@ -0,0 +1,323 @@ +import type { AdapterEvent, OcxConfig } from "../../types"; +import { routeConcreteModel, type RouteResult } from "../../router"; +import { copyPlainData } from "../../lib/plain-data"; +import { jsonUtf8Bytes } from "../../lib/json-byte-size"; +import type { TranslatorBudget } from "../../lib/translator-budget"; +import { readBoundedResponseBytes } from "../../lib/bounded-body"; +import { isRequestExecutionBudget, type RequestExecutionBudget, type SingleUseDispatchPermit } from "../../lib/request-execution-budget"; +import { isNonReplayableResponse, isNonReplayableUpstreamCode, markResponseNonReplayable, TRANSIENT_RETRY_MAX_ATTEMPTS } from "../../lib/upstream-retry"; +import { isCyberPolicyCode, isTerminalRefusalCode } from "../../lib/errors"; +import { isCanonicalOpenAiForwardProvider, supportsNativeResponsesCompactEndpoint } from "../../providers/openai-tiers"; +import { bridgeToResponsesSSE, formatErrorResponse } from "../../bridge"; +import { buildCompactV1Output, decodeCompactionSummary, encodeCompactionSummary, extractCompactUserMessages } from "../../responses/compaction"; +import { finishRequestAttempt, usageFromResponsesPayload, type RequestLogContext } from "../request-log"; +import { linkRequestSessionLane } from "../request-log-conversation"; +import { isNativePassthroughSseResponse, markNativePassthroughSseResponse, isEagerRelaySseResponse, markEagerRelaySseResponse } from "../relay"; +import type { HandleResponsesOptions } from "./core-options"; +import { consumeComboFailure, createChildPassthroughCallbackGate } from "./core-combo-failure"; +import { preflightComboStreamResponse } from "./combo-stream-preflight"; +import { conversationCarriesUploadedFiles } from "./account-change-state"; +import { selfContainedResponsesBody } from "./reset-replay"; +import { decideCompactionRecovery, readCompactionRecoveryConfig } from "./compaction-recovery-policy"; + +type Options = HandleResponsesOptions & { translatorBudget: TranslatorBudget }; +type Dispatch = (req: Request, config: OcxConfig, log: RequestLogContext, options: Options) => Promise; +const MAX_BYTES = 32 * 1024 * 1024; +const RETAINED_USER_CHARS = 80_000; +const record = (value: unknown): value is Record => !!value && typeof value === "object" && !Array.isArray(value); +const token = (value: unknown): string | undefined => typeof value === "string" && /^[a-zA-Z0-9_-]{1,128}$/.test(value) ? value : undefined; + +function identity(route: RouteResult): string { + return JSON.stringify([route.providerName, route.modelId, route.codexAccountMode ?? "", route.codexAccountNamespace ?? ""]); +} + +function physicalSends(log: RequestLogContext): number { + // activeAttempt normally also belongs to attempts: count each receipt exactly once. + const attempts = new Set([...(log.attempts ?? []), ...(log.activeAttempt ? [log.activeAttempt] : [])]); + return [...attempts].reduce((sum, attempt) => sum + Math.max(0, attempt.sendCount), 0); +} + +/** Reconcile only this leg's physical receipts; never charge already-booked adapter sends again. */ +function settlePhysicalSends(log: RequestLogContext, budget: RequestExecutionBudget, beforeSends: number, beforeUsed: number, reported: number, permit?: SingleUseDispatchPermit): number { + const sent = Math.max(0, physicalSends(log) - beforeSends); + // A legacy fetch leg does not claim the hop through adapterDispatchBudget. Settle its + // prepaid booking explicitly; an adapter-owned leg already claimed it, making this a no-op. + if (sent > 0) permit?.assumeCharge(); + else permit?.release(); + // An external report may settle a prepaid booking without changing used. Its explicit + // receipt outranks the numeric delta, or the same source send would be charged twice. + const unreported = Math.max(0, sent - Math.max(reported, Math.max(0, budget.used - beforeUsed))); + if (unreported > 0) budget.used += unreported; + return sent; +} + +function portableBody(body: Record): boolean { + if (!Array.isArray(body.input) || body.store === true || conversationCarriesUploadedFiles(body)) return false; + // Native ciphertext cannot be summarized by another provider. Never silently replace it with a note. + if (body.input.some(item => record(item) && ["compaction", "compaction_summary", "context_compaction"].includes(String(item.type)) + && typeof item.encrypted_content === "string" && !item.encrypted_content.startsWith("ocx1:"))) return false; + const input = body.input.filter(item => !record(item) || item.type !== "compaction_trigger"); + return selfContainedResponsesBody({ ...body, store: false, input }); +} + +function routed(route: RouteResult): boolean { + return !route.combo && route.routeKind !== "policy" && route.routeReason !== "default-provider" + && !isCanonicalOpenAiForwardProvider(route.provider) + && !supportsNativeResponsesCompactEndpoint(route.providerName, route.provider); +} + +/** One reader, bounded bytes, and an exact replacement body; never clone a live stream. */ +async function bufferedJson(response: Response, signal: AbortSignal): Promise<{ response: Response; json?: Record }> { + const bytes = await readBoundedResponseBytes(response, { signal, maxBytes: MAX_BYTES, inactivityTimeoutMs: 300_000 }); + if (bytes.oversized) return { response: formatErrorResponse(502, "translation_buffer_limit", "Compaction recovery response exceeded its byte limit") }; + const headers = new Headers(response.headers); + headers.delete("content-length"); + headers.delete("content-encoding"); + const replacement = new Response(bytes.bytes, { status: response.status, statusText: response.statusText, headers }); + if (isNonReplayableResponse(response)) markResponseNonReplayable(replacement); + try { + const json: unknown = JSON.parse(new TextDecoder("utf-8", { fatal: true }).decode(bytes.bytes)); + return { response: replacement, ...(record(json) ? { json } : {}) }; + } catch { return { response: replacement }; } +} + +/** + * Opt-in recovery for routed compaction only. Native compact/ciphertext, stored continuations, + * policy/combo routes and hosted tools retain their original failure. This owns no credentials. + */ +export async function runWithCompactionRecovery( + req: Request, config: OcxConfig, logCtx: RequestLogContext, options: Options, dispatch: Dispatch, +): Promise { + const recovery = readCompactionRecoveryConfig(config.compactionRecovery); + if (!recovery || options.compactionRecoveryAttempted || options.comboAttempt || (options.inboundWire && options.inboundWire !== "responses")) { + return dispatch(req, config, logCtx, options); + } + let snapshot: Record | undefined; + let snapshotBytes = 0; + let sourceRoute: RouteResult | undefined; + let partialOutput = false; + let replayUnsafe = false; + let adapterError: Extract | undefined; + let sourceFailure: Response | undefined; + let recoveryPermit: SingleUseDispatchPermit | undefined; + let restoreSourceLog: ((attemptStatus?: number) => void) | undefined; + let sourceReportedSends = 0; + const gate = createChildPassthroughCallbackGate(options); + const signal = options.abortSignal ?? req.signal; + const spentBefore = options.sendBudget?.used ?? 0; + const sendsBefore = physicalSends(logCtx); + const firstOptions: Options = { + ...options, + onCompactionRecoverySendsReported(count) { + sourceReportedSends += count; + options.onCompactionRecoverySendsReported?.(count); + }, + onRequestBodyParsed(body) { + options.onRequestBodyParsed?.(body); + if (!record(body) || !Array.isArray(body.input) || typeof body.model !== "string" + || !body.input.some(item => record(item) && item.type === "compaction_trigger") || !portableBody(body)) return; + const users = extractCompactUserMessages(body.input); + if ((users.at(-1)?.length ?? 0) > RETAINED_USER_CHARS) return; + try { + const bytes = jsonUtf8Bytes(body, MAX_BYTES); + const reservation = options.translatorBudget.reserveTransient(bytes, { kind: "request_copies" }); + try { + const copy = copyPlainData(body); + if (!copy.ok) return; + snapshot = copy.value; + snapshotBytes = bytes; + reservation.commitRetained(); + } finally { reservation.release(); } + } catch { /* Optional recovery cannot reject an otherwise valid original request. */ } + }, + onCompactionRecoveryRoute(route) { + options.onCompactionRecoveryRoute?.(route); + if (snapshot && routed(route)) sourceRoute = { ...route }; + }, + onCompactionRecoveryAdapterEvent(event) { + options.onCompactionRecoveryAdapterEvent?.(event); + if (!snapshot) return; + if (event.type === "heartbeat") replayUnsafe ||= event.replayUnsafe === true; + else if (event.type === "error") adapterError = event; + else if (event.type !== "done") partialOutput = true; + }, + onResponseComplete: model => snapshot ? gate.onResponseComplete(model) : options.onResponseComplete?.(model), + onNativePassthroughTerminal: status => snapshot ? gate.onTerminal(status) : options.onNativePassthroughTerminal?.(status), + onNativePassthroughCancel: () => snapshot ? gate.onCancel() : options.onNativePassthroughCancel?.(), + }; + try { + let response = await dispatch(req, config, logCtx, firstOptions); + const keep = (value: Response) => { gate.commit(); return value; }; + if (!snapshot || !sourceRoute || signal.aborted || req.signal.aborted || isNonReplayableResponse(response)) return keep(response); + let target: RouteResult; + try { target = routeConcreteModel(config, recovery.model); } catch { return keep(response); } + if (!routed(target) || identity(sourceRoute) === identity(target)) return keep(response); + const originalModel = firstOptions.compactionRoutingOverride?.sourceModel ?? String(snapshot.model); + // Use the established protocol commit boundary. For runTurn streams the direct event + // observer additionally preserves side-effect heartbeats that the bridge does not publish. + if (response.ok && response.headers.get("content-type")?.includes("text/event-stream")) { + const native = isNativePassthroughSseResponse(response); + const eager = isEagerRelaySseResponse(response); + const preflight = await preflightComboStreamResponse(response, logCtx); + response = preflight.response; + if (preflight.kind !== "failed") { + if (native) markNativePassthroughSseResponse(response); + if (eager) markEagerRelaySseResponse(response); + return keep(response); + } + } else if (response.ok) { + const buffered = await bufferedJson(response, signal); + response = buffered.response; + const json = buffered.json; + if (!json || json.status !== "failed" || (Array.isArray(json.output) && json.output.length > 0)) return keep(response); + // HTTP 200 can carry a failed terminal. The original structured error remains intact. + response = Response.json({ error: json.error, response: json }, { status: adapterError?.status ?? 502 }); + } + if (response.ok || replayUnsafe || partialOutput || signal.aborted || req.signal.aborted) { + if (replayUnsafe) markResponseNonReplayable(response); + return keep(response); + } + const failure = await consumeComboFailure(response, signal); + response = failure.response; + const code = token(adapterError?.code) ?? token(failure.upstreamCode); + const errorType = token(adapterError?.errorType) ?? token(failure.upstreamType); + const budget = options.sendBudget; + // Reset-only fetch legs report their physical receipt but historically leave used alone. + // Reconcile only a failed, eligible compaction; successful/disabled requests stay unchanged. + const sourceSends = budget && isRequestExecutionBudget(budget) + ? settlePhysicalSends(logCtx, budget, sendsBefore, spentBefore, sourceReportedSends) : 0; + const decision = decideCompactionRecovery(recovery, { + requestKind: options.compactionRecoveryKind ?? "compaction-v2", recoveryAttempts: 0, + cancelled: signal.aborted || req.signal.aborted, nonReplayable: !!failure.nonReplayable || isNonReplayableUpstreamCode(code), + partialOutput, toolEffects: replayUnsafe, + remainingSends: budget && isRequestExecutionBudget(budget) ? budget.remainingBaseSends(TRANSIENT_RETRY_MAX_ATTEMPTS) : 0, + originalModel: identity(sourceRoute), fallbackModel: identity(target), provider: sourceRoute.provider.adapter, + httpStatus: adapterError?.status ?? response.status, responseStatus: "failed", errorCode: code, errorType, + authenticationDenied: response.status === 401 || response.status === 403, + policyDenied: isCyberPolicyCode(code), budgetDenied: code === "translation_buffer_limit", + refusal: isTerminalRefusalCode(code), upstreamFailure: sourceSends > 0, + }); + if (!decision.recover || !budget || !isRequestExecutionBudget(budget)) return keep(response); + const fallbackBeforeSends = physicalSends(logCtx); + const fallbackBeforeUsed = budget.used; + const reservation = budget.reserveDispatch({ sendClass: "combo-failover", targetKey: `compaction:${identity(target)}`, countedExternally: true, replaySafe: true }); + if (!reservation.allowed) return keep(response); + recoveryPermit = reservation.permit; + sourceFailure = response; + gate.discard(); + // Finish the first physical attempt while retaining its receipt in attempts[]. + if (logCtx.activeAttempt) finishRequestAttempt(logCtx.activeAttempt, response.status, + Math.max(0, Date.now() - (logCtx.activeAttemptStartedAt ?? Date.now())), logCtx.activeAttempt.usage ?? logCtx.usage); + // Snapshot the original log fields before the fallback rewrites them: a failed fallback + // returns the original failure, so the log must keep describing that failure, not the + // fallback's model, provider, route decision or terminal error. + const SOURCE_LOG_FIELDS = [ + "model", "provider", "providerAdapter", "requestedAlias", "servedModel", "wireModel", + "resolvedModel", "routeDecision", "tierOutcome", "activeTierMetadata", "usage", + "usageFromBridge", "upstreamError", "terminalHttpStatus", "terminalErrorCode", + "terminalIncompleteReason", "errorCode", + ] as const; + const sourceLog: Partial> = {}; + const logFields = logCtx as unknown as Record; + for (const field of SOURCE_LOG_FIELDS) sourceLog[field] = logCtx[field]; + restoreSourceLog = (attemptStatus?: number) => { + if (logCtx.activeAttempt) finishRequestAttempt(logCtx.activeAttempt, attemptStatus ?? sourceFailure!.status, + Math.max(0, Date.now() - (logCtx.activeAttemptStartedAt ?? Date.now())), logCtx.activeAttempt.usage ?? logCtx.usage); + delete logCtx.activeAttempt; + delete logCtx.activeAttemptStartedAt; + for (const field of SOURCE_LOG_FIELDS) { + if (sourceLog[field] === undefined) delete logFields[field]; + else logFields[field] = sourceLog[field]; + } + }; + delete logCtx.activeAttempt; + delete logCtx.activeAttemptStartedAt; + delete logCtx.usage; + delete logCtx.usageFromBridge; + delete logCtx.upstreamError; + delete logCtx.terminalHttpStatus; + delete logCtx.terminalErrorCode; + delete logCtx.terminalIncompleteReason; + const headers = new Headers(req.headers); + headers.delete("authorization"); + headers.delete("chatgpt-account-id"); + headers.delete("content-length"); + headers.delete("content-encoding"); + headers.set("content-type", "application/json"); + const nextBody = { ...snapshot, model: decision.model, stream: false, store: false }; + const bytes = jsonUtf8Bytes(nextBody, MAX_BYTES); + const serialization = options.translatorBudget.reserveTransient(bytes, { kind: "request_copies" }); + let fallback: Response; + let fallbackReportedSends = 0; + try { + const child = new Request(req.url, { method: "POST", headers, body: JSON.stringify(nextBody), signal: req.signal }); + linkRequestSessionLane(req, child); + fallback = await dispatch(child, config, logCtx, { + ...options, compactionRecoveryAttempted: true, compactionRecoveryPermit: recoveryPermit, + compactionRoutingOverride: { sourceModel: originalModel }, + onRequestBodyRead: undefined, onRequestBodyParsed: undefined, + onCompactionRecoveryRoute: undefined, onCompactionRecoveryAdapterEvent: undefined, + onCompactionRecoverySendsReported: count => { + fallbackReportedSends += count; + options.onCompactionRecoverySendsReported?.(count); + }, + onResponseComplete: undefined, onNativePassthroughTerminal: undefined, onNativePassthroughCancel: undefined, + }); + } finally { + settlePhysicalSends(logCtx, budget, fallbackBeforeSends, fallbackBeforeUsed, fallbackReportedSends, recoveryPermit); + serialization.release(); + } + if (signal.aborted || req.signal.aborted) { + void fallback.body?.cancel().catch(() => undefined); + return formatErrorResponse(499, "client_cancelled", "Client cancelled compact request"); + } + if (!fallback.ok) { + void fallback.body?.cancel().catch(() => undefined); + restoreSourceLog?.(fallback.status); + return response; + } + const completed = await bufferedJson(fallback, signal); + const json = completed.json; + const items = json && Array.isArray(json.output) ? json.output : []; + const compactions = items.filter(value => record(value) && value.type === "compaction"); + const permitted = items.every(value => record(value) && (value.type === "compaction" || value.type === "reasoning")); + const item = permitted && compactions.length === 1 ? compactions[0] as Record : undefined; + const summary = item && typeof item.encrypted_content === "string" ? decodeCompactionSummary(item.encrypted_content) : null; + if (json?.status !== "completed" || !summary?.trim()) { + void completed.response.body?.cancel().catch(() => undefined); + restoreSourceLog?.(completed.response.status); + return response; + } + // v1 unpacks this text through buildCompactV1Output, which already re-adds retained user + // messages as items; embedding them here too would duplicate the same text in the output. + const preserved = options.compactionRecoveryKind === "compaction-v1" ? summary : (() => { + const retained = buildCompactV1Output(extractCompactUserMessages(snapshot.input), summary).slice(0, -1); + const userText = extractCompactUserMessages(retained).map((text, index) => `User message ${index + 1}:\n${text}`).join("\n\n"); + return `${summary}\n\nRetained original user messages (verbatim; preserve their goals and constraints):\n${userText}`; + })(); + item!.encrypted_content = encodeCompactionSummary(preserved); + json.model = originalModel; + void completed.response.body?.cancel().catch(() => undefined); + void response.body?.cancel().catch(() => undefined); + if (snapshot.stream === true) { + const usage = usageFromResponsesPayload(json.usage); + async function* events(): AsyncGenerator { + yield { type: "text_delta", text: preserved }; + yield { type: "done", ...(usage ? { usage } : {}) }; + } + return new Response(bridgeToResponsesSSE(events(), originalModel, undefined, undefined, undefined, undefined, 2_000, + { compaction: true, translatorBudget: options.translatorBudget, onCompletedResponse: () => options.onResponseComplete?.(originalModel) }), { headers: { "content-type": "text/event-stream" } }); + } + options.onResponseComplete?.(originalModel); + return Response.json(json); + } catch (error) { + gate.discard(); + if (signal.aborted || req.signal.aborted) return formatErrorResponse(499, "client_cancelled", "Client cancelled compact request"); + if (sourceFailure) { restoreSourceLog?.(); return sourceFailure; } + throw error; + } finally { + if (snapshotBytes) options.translatorBudget.releaseRetained(snapshotBytes, { kind: "request_copies" }); + recoveryPermit?.release(); + snapshot = undefined; + } +} diff --git a/src/server/responses/core-combo-failure.ts b/src/server/responses/core-combo-failure.ts index e222c2ccaea..18e9f6e5c6a 100644 --- a/src/server/responses/core-combo-failure.ts +++ b/src/server/responses/core-combo-failure.ts @@ -117,6 +117,7 @@ export async function consumeComboFailure( ...(nonReplayable ? { nonReplayable: true } : {}), classificationText, ...(normalizedUpstreamCode !== undefined ? { upstreamCode: normalizedUpstreamCode } : {}), + ...(upstreamType !== undefined ? { upstreamType } : {}), ...(!cyberFailure && cooldownRetryAfter !== undefined ? { retryAfter: cooldownRetryAfter } : {}), // The EFFECTIVE classification decides, not the raw status. An upstream that wraps a quota // refusal in a 5xx still carries `x-codex-*-reset-at`, and gating on 402/429 alone threw diff --git a/src/server/responses/core-options.ts b/src/server/responses/core-options.ts index 47c2c982c40..a8cc1e06835 100644 --- a/src/server/responses/core-options.ts +++ b/src/server/responses/core-options.ts @@ -1,5 +1,7 @@ import type { NativeResponseControl } from "./native-response-control"; -import type { OcxUsage, OcxProviderContinuationState, OcxConfig } from "../../types"; +import type { AdapterEvent, OcxUsage, OcxProviderContinuationState, OcxConfig } from "../../types"; +import type { RouteResult } from "../../router"; +import type { SingleUseDispatchPermit } from "../../lib/request-execution-budget"; import type { CodexAuthPolicyConfig, CodexAuthContext } from "../../codex/auth-context"; import type { AdmissionLease } from "../../lib/admission"; import type { DataPlaneAdmission } from "../auth-cors"; @@ -22,6 +24,8 @@ export interface ConsumedComboFailure { classificationText: string; /** Structured upstream `error.code` when present in the failure body. */ upstreamCode?: string; + /** Complete structured provider type, retained for conservative recovery classification. */ + upstreamType?: string; /** Valid numeric/date value used only for cooldown calculation. */ retryAfter?: string; /** Upstream Codex quota-window reset timestamps used for combo cooldowns. */ @@ -52,6 +56,14 @@ export interface ClientEncoderOption { } export interface HandleResponsesOptions { + /** Internal routed-compaction recovery: one logical request, one emergency target. */ + compactionRecoveryAttempted?: boolean; + compactionRecoveryPermit?: SingleUseDispatchPermit; + compactionRecoveryKind?: "compaction-v1" | "compaction-v2"; + onCompactionRecoveryRoute?: (route: RouteResult) => void; + onCompactionRecoveryAdapterEvent?: (event: AdapterEvent) => void; + /** Physical-send reports already delivered to the shared used setter, including booking settlement. */ + onCompactionRecoverySendsReported?: (count: number) => void; /** Internal Claude replay identity; consumed only by the final canonical Go transport. */ claudeGoAffinity?: { sessionLane?: string }; /** Validated Claude metadata identity; projected only into final canonical attempt headers. */ diff --git a/src/server/responses/core.ts b/src/server/responses/core.ts index 635d5e37c17..168c28c6a30 100644 --- a/src/server/responses/core.ts +++ b/src/server/responses/core.ts @@ -27,6 +27,7 @@ import { createAdapterContinuations } from "./adapter-continuation"; import { deliverAdapterResponse } from "./adapter-delivery"; import { releaseUpstreamHostAdmission } from "../../codex/upstream-host-health"; import { releaseCodexAuthContextProbeLease } from "../../codex/auth-context"; +import { runWithCompactionRecovery } from "./compaction-recovery"; /** Public Responses entry and compatibility exports. Implementations live with their owners. */ @@ -43,7 +44,7 @@ export async function handleResponses( const ownsBudget = options.translatorBudget === undefined; const translatorBudget = options.translatorBudget ?? createTranslatorBudget(); try { - const response = await handleResponsesInner(req, config, logCtx, { + const response = await runWithCompactionRecovery(req, config, logCtx, { ...options, openAiSidecarAuth: options.openAiSidecarAuth === undefined ? captureExplicitOpenAiCallerAuth(req.headers, config) : options.openAiSidecarAuth, @@ -57,7 +58,7 @@ export async function handleResponses( translatorBudget, // Once at ingress, spend observer included: a combo child inherits the parent's holder. sendBudget: options.sendBudget ?? createInferenceSendBudget(req, logCtx), - }); + }, handleResponsesInner); return ownsBudget ? finalizeOwnedTranslatorBudget(response, translatorBudget) : response; } catch (error) { if (ownsBudget) translatorBudget.dispose(); @@ -101,6 +102,7 @@ async function handleResponsesInner( if (requestState instanceof Response) return requestState; const transportState = await prepareResponsesTransport(requestContext, admissionState, requestState); if (transportState instanceof Response) return transportState; + options.onCompactionRecoveryRoute?.(requestState.route); const sidecarState = await prepareResponsesSidecarAuth(requestContext, requestState, transportState); if (sidecarState instanceof Response) return sidecarState; const responseEffects = createResponsesEffects( diff --git a/src/server/responses/request-send-budget.ts b/src/server/responses/request-send-budget.ts index 4f27131b94d..b8bc18e4fd9 100644 --- a/src/server/responses/request-send-budget.ts +++ b/src/server/responses/request-send-budget.ts @@ -60,6 +60,7 @@ export function createResponsesSendBudget( const noteTransientSends = (used: number): void => { const charged = Math.max(0, used); sendBudget.used += charged; + options.onCompactionRecoverySendsReported?.(charged); chargeWorkflowSends(workflowRootId, charged); }; // Refused before any dispatch, and deliberately not by evicting the root's ledger entry: @@ -156,7 +157,7 @@ export function createResponsesSendBudget( * refused and the request would answer with a synthetic 502 in place of the real 429 the hop * was recovering from. */ - let pendingHopPermit: SingleUseDispatchPermit | undefined; + let pendingHopPermit: SingleUseDispatchPermit | undefined = options.compactionRecoveryPermit; /** * The budget an adapter's OWN dispatch ladder reserves against. * diff --git a/src/server/responses/run-turn-execution.ts b/src/server/responses/run-turn-execution.ts index 8f6da738d7f..109151355bf 100644 --- a/src/server/responses/run-turn-execution.ts +++ b/src/server/responses/run-turn-execution.ts @@ -219,6 +219,10 @@ export async function executeResponsesRunTurn( ): Promise => { const attemptSeq = ++runTurnAttemptSeq; let pacingSlot = preacquiredSlot; + const emit = (event: AdapterEvent) => { + options.onCompactionRecoveryAdapterEvent?.(event); + targetQueue.push(event); + }; try { if (!pacingSlotAcquired) { pacingSlot = await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); @@ -271,7 +275,7 @@ export async function executeResponsesRunTurn( ), onRecoveryWithheld: noteAdapterRecoveryWithheld, }, - targetQueue.push, + emit, ); // LOCAL PATCH (runturn-websearch): adapters may write conversation/ // continuation state onto the object they received; merge it back so @@ -285,7 +289,7 @@ export async function executeResponsesRunTurn( Object.assign(parsed, routeState); } } catch (err) { - targetQueue.push(err instanceof RequestPacingQueueOverloadError + emit(err instanceof RequestPacingQueueOverloadError ? { type: "error", status: 429, diff --git a/src/types/config.ts b/src/types/config.ts index ad59d6f9f65..8c89c4dd824 100644 --- a/src/types/config.ts +++ b/src/types/config.ts @@ -722,6 +722,8 @@ export interface OcxConfig { /** Compaction triggers this override covers; omission means `["manual"]`. */ triggers?: ("manual" | "auto")[]; }; + /** Opt-in failure-only recovery; never replaces the initial compaction model. */ + compactionRecovery?: { enabled: boolean; model: string; allowDevinInvalidArgument?: boolean }; /** * Models hidden from Codex discovery without blocking direct proxy calls. Routed provider ids * are excluded from the catalog + /v1/models entirely. Account-qualified native ids hide only diff --git a/structure/config.md b/structure/config.md index fae0c2a250e..cd857e21390 100644 --- a/structure/config.md +++ b/structure/config.md @@ -21,13 +21,13 @@ Native main reauthentication follows the [CLI JSON output contract](runtime.md#n The Codex restart command follows the [CLI restart scope contract](runtime.md#cli-codex-restart-scope). -`src/cli/account-orca-import.ts` exposes an explicit-source, preview-first local import command. -Apply adds pool configuration under the shared mutation lock; the -[source-owned credential contract](codex-home.md#orca-source-owned-account-import) governs -deduplication and credential storage separately from Codex config injection. +`src/cli/account-orca-import.ts` exposes an explicit-source, preview-first local import command. Apply adds pool configuration under the shared mutation lock; +the [source-owned credential contract](codex-home.md#orca-source-owned-account-import) governs deduplication and credential storage separately from Codex config injection. ## Config surface +`src/config/schema/compaction-recovery.ts` strictly validates opt-in `compactionRecovery`; invalid disk values disable it with a warning, while candidate writes reject them. The [failure-only contract](transports/responses-failover.md) leaves provider identity, accounts and client compaction unchanged. + Google providers may persist `googleToolSchemaPolicy` as `compatible` or `reject-lossy`. `ocx provider add --google-tool-schema-policy` is one authoring path and is accepted only when the effective adapter is `google`. Omission remains absent in `config.json`; the adapter resolves it to diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 6f9d8bd7495..abbc0f25439 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -22,6 +22,11 @@ linked sources after the quota await to reject revoked or rotated captures. ## Compact desktop usage +`GET /api/settings` and successful `PUT /api/settings` report `compactionRecovery` as a block or +null. The write accepts the strict failure-recovery schema or null to remove it, and restores the +prior field if persistence fails. Changing this field alone does not converge catalogs or inject +Codex configuration. `tests/config/compaction-recovery-settings.test.ts` covers that boundary. + The standalone `/#/tray` GUI route presents local usage and account limits without the full dashboard navigation. It reuses the existing API session and fetch wrapper; it has no Tauri IPC capability. Companion settings control its sections and chart. Account diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 8d4cbb7c3b2..78c16d7034c 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -1,5 +1,22 @@ # Responses Failover And Replay +`src/server/responses/compaction-recovery-policy.ts` is a pure eligibility policy, not a dispatcher. +It requires explicit configuration and normalized attempt evidence, preserves ordinary requests, +and refuses cancellation, committed semantic output, tool effects, protected failures, exhausted +send budgets and repeated recovery. Its Devin `invalid_argument` exception is separately opted in; +an opaque HTTP 400 alone never grants replay. The caller owns canonical target resolution, output +validation and shared-budget reservation. `compaction-recovery.ts` connects this policy to +self-contained routed v1/v2 compaction in `core.ts` and `compact.ts`; normal and successful +requests keep their original route. One configured emergency target shares the original send +and translation budgets. Physical-send receipts and explicit retry-helper reports reconcile legacy +fetch sends without double charging external reservations; one prepaid emergency permit is shared +with adapter dispatch, and only additional retries draw from the remainder. Adapter observers retain partial-output and structured denial evidence +before response projection. Native encrypted compaction, uploaded files, stored continuations, +and policy/combo routes are excluded. Emergency output must contain one readable portable +compaction item; recent original user messages are retained verbatim, and recovery failure keeps +the original failure. `tests/responses/responses-compaction-recovery-policy.test.ts` and +`tests/responses/responses-compaction-recovery.test.ts` pin these boundaries. + Retry, replay, and combo failover on the Responses data plane: upstream reset retry, the ambiguous-resend gate and replay boundary, combo quota fallback and commit boundaries, compaction routing overrides, and output headroom. The endpoint and dispatch rules they build on are in diff --git a/tests/config/compaction-recovery-settings.test.ts b/tests/config/compaction-recovery-settings.test.ts new file mode 100644 index 00000000000..b5cff096e47 --- /dev/null +++ b/tests/config/compaction-recovery-settings.test.ts @@ -0,0 +1,67 @@ +import { describe, expect, test } from "bun:test"; +import { getDefaultConfig, validateConfigCandidate } from "../../src/config"; +import { configSchema } from "../../src/config/schema/config-schema"; +import { compactionRecoverySchema } from "../../src/config/schema/compaction-recovery"; +import { handleConfigRoutes } from "../../src/server/management/config-routes"; +import type { ManagementContext } from "../../src/server/management/context"; +import type { OcxConfig } from "../../src/types"; + +const enabled = { enabled: true, model: "emergency/model", allowDevinInvalidArgument: true }; +function harness(failSave = false) { + const config = getDefaultConfig(); + let saves = 0; + const call = async (body: unknown) => { + const url = new URL("http://localhost/api/settings"); + return handleConfigRoutes({ + url, req: new Request(url, { method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify(body) }), + config, version: "test", deps: { saveConfigPreservingClaudeCode: (_c: OcxConfig) => { + saves++; if (failSave) throw new Error("fixture-save-failed"); + } }, + } as unknown as ManagementContext); + }; + return { config, call, saves: () => saves }; +} + +describe("compaction failure recovery settings", () => { + test("absent is off; an explicit valid configuration survives parsing", () => { + expect(getDefaultConfig().compactionRecovery).toBeUndefined(); + const result = validateConfigCandidate({ ...getDefaultConfig(), compactionRecovery: enabled }); + expect(result.ok).toBe(true); + if (result.ok) expect(result.config.compactionRecovery).toEqual(enabled); + expect(compactionRecoverySchema.parse({ enabled: false, model: "emergency/model" }).enabled).toBe(false); + }); + test("invalid hand edits degrade to off but mutation validation rejects them", () => { + for (const value of [{ ...enabled, retryCount: 99 }, { ...enabled, enabled: "true" }, { ...enabled, model: "" }, { ...enabled, model: "x\ny" }, { ...enabled, allowDevinInvalidArgument: 1 }]) { + const input = { ...getDefaultConfig(), compactionRecovery: value }; + expect(configSchema.parse(input).compactionRecovery).toBeUndefined(); + expect(validateConfigCandidate(input).ok).toBe(false); + } + }); + test("management writes and clears recovery without toggling routing or login mode", async () => { + const h = harness(); + h.config.compactionRouting = { model: "original/manual" }; + const res = await h.call({ compactionRecovery: enabled }); + expect(res?.status).toBe(200); + expect(h.config.compactionRecovery).toEqual(enabled); + expect(h.config.compactionRouting).toEqual({ model: "original/manual" }); + expect(h.config.codexDesktopAuthless).toBeUndefined(); + expect(h.config.codexClientCompaction).toBeUndefined(); + expect((await res!.json() as any).compactionRecovery).toEqual(enabled); + expect((await h.call({ compactionRecovery: null }))?.status).toBe(200); + expect(h.config.compactionRecovery).toBeUndefined(); + expect(h.saves()).toBe(2); + }); + test("rejected management configuration never reaches persistence", async () => { + const h = harness(); + expect((await h.call({ compactionRecovery: { ...enabled, enabled: 1 } }))?.status).toBe(400); + expect(h.saves()).toBe(0); + expect(h.config.compactionRecovery).toBeUndefined(); + }); + test("a save failure restores the prior recovery field", async () => { + const h = harness(true); + const original = { enabled: false, model: "kept/model" }; + h.config.compactionRecovery = original; + await expect(h.call({ compactionRecovery: enabled })).rejects.toThrow("fixture-save-failed"); + expect(h.config.compactionRecovery).toEqual(original); + }); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 96dcb47a84e..d7480980b9a 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,7 @@ { + "responses-compaction-recovery.test.ts": "responses", + "compaction-recovery-settings.test.ts": "config", + "responses-compaction-recovery-policy.test.ts": "responses", "deepseek-artifact-tool-schema.test.ts": "providers", "client-config-export-output-limit.test.ts": "config", "openai-chat-serialized-tool-call-scaling.test.ts": "adapters/openai", diff --git a/tests/helpers/responses-core-source.ts b/tests/helpers/responses-core-source.ts index 2d3805ea0b7..7460b64ada9 100644 --- a/tests/helpers/responses-core-source.ts +++ b/tests/helpers/responses-core-source.ts @@ -32,6 +32,8 @@ export const RESPONSES_CORE_MODULES = [ "request-prepare.ts", "shadow-target-availability.ts", "compaction-routing.ts", + "compaction-recovery.ts", + "compaction-recovery-policy.ts", "request-transport.ts", "request-sidecar-auth.ts", "response-effects.ts", diff --git a/tests/responses/responses-compaction-recovery-policy.test.ts b/tests/responses/responses-compaction-recovery-policy.test.ts new file mode 100644 index 00000000000..87c7e2734a6 --- /dev/null +++ b/tests/responses/responses-compaction-recovery-policy.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, test } from "bun:test"; +import { decideCompactionRecovery, readCompactionRecoveryConfig, type CompactionRecoveryEvidence } from "../../src/server/responses/compaction-recovery-policy"; + +const config = { enabled: true, model: "emergency-alias", allowDevinInvalidArgument: true }; +const failure = (patch: Partial = {}): CompactionRecoveryEvidence => ({ + requestKind: "compaction-v2", recoveryAttempts: 0, cancelled: false, nonReplayable: false, + partialOutput: false, toolEffects: false, remainingSends: 1, + originalModel: "devin/main/swe-2", fallbackModel: "google/main/gemini", provider: "devin", + httpStatus: 400, responseStatus: "failed", errorCode: "invalid_argument", + authenticationDenied: false, policyDenied: false, budgetDenied: false, refusal: false, + upstreamFailure: true, ...patch, +}); + +describe("compaction emergency recovery policy", () => { + test("accepts the explicit Devin exception on v1 and HTTP-200 failed v2", () => { + expect(decideCompactionRecovery(config, failure({ requestKind: "compaction-v1" }))).toEqual({ + recover: true, model: "emergency-alias", reason: "devin-invalid-argument", + }); + expect(decideCompactionRecovery(config, failure({ httpStatus: 200 }))).toMatchObject({ recover: true }); + }); + + test("successful compaction and ordinary generation never request fallback", () => { + expect(decideCompactionRecovery(config, failure({ httpStatus: 200, responseStatus: "completed" }))).toEqual({ recover: false, reason: "succeeded" }); + expect(decideCompactionRecovery(config, failure({ requestKind: "ordinary" }))).toEqual({ recover: false, reason: "ordinary-request" }); + }); + + test("the exception needs both opt-ins, exact serving provider and structured code", () => { + for (const candidate of [undefined, null, {}, { ...config, enabled: false }]) { + expect(decideCompactionRecovery(candidate, failure()).recover).toBe(false); + } + for (const patch of [ + { provider: "google" }, { errorCode: "invalid_request_error" }, { errorCode: undefined }, + { errorCode: "message_contains_invalid_argument" }, { upstreamFailure: false }, + { httpStatus: 200, responseStatus: "unknown" as const }, + { httpStatus: 502, responseStatus: "unknown" as const }, + ]) expect(decideCompactionRecovery(config, failure(patch)).recover).toBe(false); + expect(decideCompactionRecovery({ enabled: true, model: "google/gemini" }, failure()).recover).toBe(false); + }); + + test("caller cancellation and semantic output or tool effects stop replay", () => { + for (const patch of [{ cancelled: true }, { httpStatus: 499 }, { errorCode: "client_cancelled" }]) { + expect(decideCompactionRecovery(config, failure(patch))).toEqual({ recover: false, reason: "cancelled" }); + } + for (const patch of [{ nonReplayable: true }, { partialOutput: true }, { toolEffects: true }]) { + expect(decideCompactionRecovery(config, failure(patch))).toEqual({ recover: false, reason: "unsafe-replay" }); + } + }); + + test("auth, policy, refusal and budget evidence outrank the provider exception", () => { + for (const patch of [ + { authenticationDenied: true }, { policyDenied: true }, { budgetDenied: true }, { refusal: true }, + { httpStatus: 401 }, { httpStatus: 403 }, { httpStatus: 402 }, { httpStatus: 429 }, + ...["authentication_error", "permission_denied", "origin_rejected", "cyber_policy_violation", + "content_filter", "request_send_budget_exhausted", "failed_precondition", "admission_model_denied"] + .map(errorType => ({ errorType })), + ]) expect(decideCompactionRecovery(config, failure(patch))).toEqual({ recover: false, reason: "protected-failure" }); + }); + + test("same canonical target, recursive recovery and exhausted allowance stop dispatch", () => { + expect(decideCompactionRecovery(config, failure({ fallbackModel: "devin/main/swe-2" }))).toEqual({ recover: false, reason: "same-model" }); + for (const recoveryAttempts of [1, 2, 100]) { + expect(decideCompactionRecovery(config, failure({ recoveryAttempts }))).toEqual({ recover: false, reason: "already-attempted" }); + } + expect(decideCompactionRecovery(config, failure({ remainingSends: 0 }))).toEqual({ recover: false, reason: "budget-exhausted" }); + }); + + test("recognizes typed overflow and compact-output failures without a Devin exception", () => { + const configured = { enabled: true, model: "google/gemini" }; + expect(decideCompactionRecovery(configured, failure({ provider: "google", errorCode: "context_length_exceeded" }))).toMatchObject({ recover: true, reason: "context-overflow" }); + expect(decideCompactionRecovery(configured, failure({ httpStatus: 200, errorCode: "invalid_compaction_output" }))).toMatchObject({ recover: true, reason: "compaction-output" }); + expect(decideCompactionRecovery(configured, failure({ httpStatus: 503, errorCode: "unavailable" }))).toMatchObject({ recover: true, reason: "upstream-unavailable" }); + // classifyError normalizes codeless 5xx bodies and overloads before the evidence reaches us. + expect(decideCompactionRecovery(configured, failure({ httpStatus: 500, errorCode: "upstream_server_error" }))).toMatchObject({ recover: true, reason: "upstream-unavailable" }); + expect(decideCompactionRecovery(configured, failure({ httpStatus: 503, errorCode: "server_is_overloaded" }))).toMatchObject({ recover: true, reason: "upstream-unavailable" }); + expect(decideCompactionRecovery(configured, failure({ httpStatus: 503, errorCode: "unrecognized_failure" })).recover).toBe(false); + }); + + test("missing or malformed evidence cannot silently grant a send", () => { + for (const patch of [ + { recoveryAttempts: NaN }, { remainingSends: Infinity }, { remainingSends: -1 }, + { partialOutput: undefined }, { toolEffects: undefined }, { originalModel: "" }, + { fallbackModel: "bad\nmodel" }, { httpStatus: 99 }, { errorCode: "x".repeat(129) }, + ]) expect(decideCompactionRecovery(config, failure(patch as Partial))).toEqual({ recover: false, reason: "invalid-evidence" }); + }); + + test("configuration is bounded, detached and requires explicit booleans", () => { + expect(readCompactionRecoveryConfig(config)).toEqual(config); + expect(readCompactionRecoveryConfig(config)).not.toBe(config); + for (const model of ["", " x", "x ", "x\ny", "x".repeat(513)]) { + expect(readCompactionRecoveryConfig({ ...config, model })).toBeNull(); + } + expect(readCompactionRecoveryConfig({ ...config, allowDevinInvalidArgument: "true" })).toBeNull(); + expect(readCompactionRecoveryConfig({ ...config, retries: 10 })).toBeNull(); + }); + + test("decision is pure and does not consume the caller's shared send allowance", () => { + const evidence = Object.freeze(failure()); + expect(decideCompactionRecovery(Object.freeze(config), evidence).recover).toBe(true); + expect(evidence.remainingSends).toBe(1); + expect(evidence.recoveryAttempts).toBe(0); + }); +}); diff --git a/tests/responses/responses-compaction-recovery.test.ts b/tests/responses/responses-compaction-recovery.test.ts new file mode 100644 index 00000000000..a60f3af0105 --- /dev/null +++ b/tests/responses/responses-compaction-recovery.test.ts @@ -0,0 +1,348 @@ +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { getDefaultConfig } from "../../src/config"; +import { handleResponses, handleResponsesCompact } from "../../src/server/responses"; +import { decodeCompactionSummary } from "../../src/responses/compaction"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import { createTranslatorBudget } from "../../src/lib/translator-budget"; +import { jsonUtf8Bytes } from "../../src/lib/json-byte-size"; +import type { AdapterEvent, OcxConfig, OcxParsedRequest } from "../../src/types"; +import type { RequestLogContext } from "../../src/server/request-log"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; + +const originalFetch = globalThis.fetch; +const sourceError: AdapterEvent = { type: "error", status: 400, code: "invalid_argument", message: "Source rejected compact fixture" }; +let sourceEvents: AdapterEvent[]; +let fallbackEvents: AdapterEvent[]; +let calls: Array<{ model: string; parsed: OcxParsedRequest }>; +let releaseSpend: (() => void) | undefined; +let restoreFactory: (() => void) | undefined; +let restoreChatFactory: (() => void) | undefined; +let abortOnSource: AbortController | undefined; + +function settings(): OcxConfig { + return { + ...getDefaultConfig(), defaultProvider: "source", + providers: { + source: { adapter: "devin", authMode: "key", apiKey: "fixture-only", baseUrl: "https://source.example" }, + emergency: { adapter: "devin", authMode: "key", apiKey: "fixture-only", baseUrl: "https://emergency.example" }, + }, + compactionRecovery: { enabled: true, model: "emergency/rescue", allowDevinInvalidArgument: true }, + }; +} + +function body(stream = false, compact = true): Record { + return { + model: "source/swe-2", stream, store: false, max_output_tokens: 512, + input: [ + { type: "message", role: "user", content: "Remember marker ALPHA-729." }, + { type: "message", role: "assistant", content: "Recorded." }, + { type: "message", role: "user", content: "Latest goal: finish the report, preserve the marker." }, + ...(compact ? [{ type: "compaction_trigger" }] : []), + ], + }; +} + +function request(payload = body(), path = "responses", signal?: AbortSignal): Request { + return new Request(`http://localhost/v1/${path}`, { + method: "POST", headers: { "content-type": "application/json", session_id: "recovery-fixture" }, + body: JSON.stringify(payload), signal, + }); +} + +beforeEach(() => { + releaseSpend = acquireOwnedSpendHome(); + calls = []; + sourceEvents = [sourceError]; + fallbackEvents = [ + { type: "thinking_delta", thinking: "Prepare the handoff." }, + { type: "text_delta", text: "Work is pending; resume the report." }, + { type: "done", usage: { inputTokens: 4, outputTokens: 6, totalTokens: 10 } }, + ]; + abortOnSource = undefined; + globalThis.fetch = (async () => { throw new Error("Unexpected network request in recovery fixture"); }) as typeof fetch; + const factory = spyOn(ADAPTER_REGISTRY.devin, "create").mockImplementation((_provider, context) => ({ + name: "devin", + reportsPhysicalSends: true, + buildRequest() { throw new Error("runTurn fixture must not build HTTP requests"); }, + async *parseStream() { throw new Error("runTurn fixture must not parse HTTP responses"); }, + async runTurn(parsed, incoming, emit) { + const send = incoming.sendBudget?.reserveDispatch({ sendClass: "initial", targetKey: `${context.providerId}/${parsed.modelId}` }); + if (send && (!send.allowed || !send.permit.use())) { + emit({ type: "error", status: 429, code: "request_send_budget_exhausted", message: "Fixture shared send allowance exhausted" }); + return; + } + incoming.onPhysicalSend?.({ ordinal: 1 }); + calls.push({ model: parsed.modelId, parsed: structuredClone(parsed) }); + const isSource = context.providerId === "source"; + for (const event of isSource ? sourceEvents : fallbackEvents) emit(event); + if (isSource) abortOnSource?.abort(); + }, + })); + restoreFactory = () => factory.mockRestore(); +}); + +afterEach(() => { + releaseSpend?.(); + releaseSpend = undefined; + restoreFactory?.(); + restoreFactory = undefined; + restoreChatFactory?.(); + restoreChatFactory = undefined; + globalThis.fetch = originalFetch; +}); + +describe("routed compaction emergency integration", () => { + test("source success and ordinary requests never use the emergency model", async () => { + sourceEvents = [{ type: "text_delta", text: "Source summary" }, { type: "done" }]; + for (const compact of [true, false]) { + const response = await handleResponses(request(body(false, compact)), settings(), { model: "", provider: "" }); + expect((await response.json()).status).toBe("completed"); + } + expect(calls.map(call => call.model)).toEqual(["swe-2", "swe-2"]); + }); + + test.each([false, true])("v2 failed terminal recovers once and retains user goals (stream=%s)", async stream => { + const config = settings(); + const before = structuredClone(config); + const completed: string[] = []; + const log: RequestLogContext = { model: "", provider: "" }; + const response = await handleResponses(request(body(stream)), config, log, { onResponseComplete: model => completed.push(model) }); + let summary: string | null; + if (stream) { + const text = await response.text(); + expect(text).not.toContain("Source rejected compact fixture"); + const terminal = text.split("\n").filter(line => line.startsWith("data: ") && line !== "data: [DONE]") + .map(line => JSON.parse(line.slice(6))).find(event => event.type === "response.completed"); + summary = decodeCompactionSummary(terminal.response.output.find((item: { type: string }) => item.type === "compaction").encrypted_content); + expect(terminal.response.usage.total_tokens).toBe(10); + } else { + const json = await response.json(); + expect(json.status).toBe("completed"); + summary = decodeCompactionSummary(json.output.find((item: { type: string }) => item.type === "compaction").encrypted_content); + } + expect(summary).toContain("ALPHA-729"); + expect(summary).toContain("Latest goal: finish the report"); + expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); + expect(calls[1]!.parsed.options.maxOutputTokens).toBe(512); + expect(calls[1]!.parsed.context.tools).toBeUndefined(); + expect(completed).toEqual(["source/swe-2"]); + expect(log.provider).toBe("emergency"); + expect(config).toEqual(before); + }); + + test("routed v1 returns replacement history retaining original user text once", async () => { + const response = await handleResponsesCompact(request(body(false, false), "responses/compact"), settings(), { model: "", provider: "" }); + expect(response.status).toBe(200); + const json = await response.json(); + const output = JSON.stringify(json.output); + expect(output).toContain("ALPHA-729"); + expect(output).toContain("Latest goal: finish the report"); + // Retained messages belong to v1 output items; embedding them in the summary duplicates the text. + expect(output).not.toContain("Retained original user messages"); + expect(output.split("ALPHA-729").length - 1).toBe(1); + expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); + }); + + test("a failed fallback returns and logs the original failure", async () => { + fallbackEvents = [{ type: "error", status: 503, code: "server_is_overloaded", message: "Emergency overloaded fixture" }]; + const log: RequestLogContext = { model: "", provider: "" }; + const response = await handleResponses(request(body(false)), settings(), log); + expect(response.status).toBe(400); + expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); + // The returned failure is the source's, so the log must describe it too — not the fallback's. + expect(log.provider).toBe("source"); + expect(log.model).toBe("swe-2"); + expect(log.requestedAlias).toBe("source/swe-2"); + expect(log.activeAttempt).toBeUndefined(); + // The fallback's own failed attempt stays recorded; its wire status was a failed 200 terminal. + expect(log.attempts?.map(attempt => attempt.status)).toEqual([400, 200]); + }); + + test("an existing unconditional override keeps its original logical model on recovery", async () => { + const config = settings(); + config.compactionRouting = { model: "source/swe-2" }; + const payload = { ...body(), model: "source/normal", client_metadata: { + "x-codex-turn-metadata": JSON.stringify({ request_kind: "compaction", compaction: { trigger: "manual" } }), + } }; + const completed: string[] = []; + const response = await handleResponses(request(payload), config, { model: "", provider: "" }, { onResponseComplete: model => completed.push(model) }); + expect((await response.json()).model).toBe("source/normal"); + expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); + expect(completed).toEqual(["source/normal"]); + }); + + test.each([false, true])("fetch adapter hidden text then failure cannot replay (stream=%s)", async stream => { + const config = settings(); + config.providers.source = { adapter: "openai-chat", authMode: "key", apiKey: "fixture-only", baseUrl: "https://source.example/v1" }; + const events: AdapterEvent[] = [{ type: "text_delta", text: "Private partial compact text" }, { type: "error", status: 500, errorType: "upstream_error", message: "Source failed after partial text" }]; + const factory = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation(() => ({ + name: "openai-chat", buildRequest() { return { url: "https://source.example/v1/chat/completions", method: "POST", headers: {}, body: "{}" }; }, + async *parseStream() { yield* events; }, async parseResponse() { return events; }, + })); + restoreChatFactory = () => factory.mockRestore(); + let fetches = 0; + globalThis.fetch = (async () => { fetches++; return Response.json({ fixture: true }); }) as typeof fetch; + const response = await handleResponses(request(body(stream)), config, { model: "", provider: "" }); + expect(await response.text()).toContain("Source failed after partial text"); + expect(fetches).toBe(1); + expect(calls).toHaveLength(0); + }); + + test("fetch HTTP 500 authentication type survives client formatting and forbids recovery", async () => { + const config = settings(); + config.providers.source = { adapter: "openai-chat", authMode: "key", apiKey: "fixture-only", baseUrl: "https://source.example/v1" }; + let fetches = 0; + globalThis.fetch = (async () => { + fetches++; + return Response.json({ error: { type: "authentication_error", message: "denied" } }, { status: 500 }); + }) as typeof fetch; + const response = await handleResponses(request(), config, { model: "", provider: "" }); + expect(response.status).toBe(500); + await response.text(); + expect(fetches).toBe(1); + expect(calls).toHaveLength(0); + }); + + test.each([[1, false], [2, false], [2, true]] as const)("fetch source and fetch emergency share cap=%s transient=%s", async (cap, transient) => { + const config = settings(); + config.providers.source = { adapter: "openai-chat", authMode: "key", apiKey: "fixture-only", baseUrl: "https://source.example/v1" }; + config.providers.emergency = { adapter: "openai-chat", authMode: "key", apiKey: "fixture-only", baseUrl: "https://emergency.example/v1", ...(transient ? { transientRetryOn5xx: { attempts: 3 } } : {}) }; + const requests: string[] = []; + globalThis.fetch = (async (input: unknown) => { + const url = String(input); + requests.push(url); + return url.includes("source.example") + ? Response.json({ error: { type: "invalid_request_error", code: "context_length_exceeded", message: "Source input context is full" } }, { status: 400 }) + : Response.json({ choices: [{ message: { role: "assistant", content: "Resume the report." }, finish_reason: "stop" }], usage: { prompt_tokens: 4, completion_tokens: 6, total_tokens: 10 } }); + }) as typeof fetch; + const budget = createRequestExecutionBudget({ maxTotalModelSends: cap, baseSendAllowance: cap, finalRecoveryAllowance: 0, maxAlternateTargetSends: 1, maxTargetTransitions: 1 }); + const response = await handleResponses(request(), config, { model: "", provider: "" }, { sendBudget: budget }); + const json = await response.json(); + expect(requests).toHaveLength(cap); + expect(budget.used).toBe(cap); + if (cap === 2) { + expect(json.status).toBe("completed"); + expect(decodeCompactionSummary(json.output.find((item: { type: string }) => item.type === "compaction").encrypted_content)).toContain("ALPHA-729"); + } else expect(json.error.code).toBe("context_length_exceeded"); + }); + + test.each([false, true])("externally booked source settles once (transient=%s)", async transient => { + const config = settings(); + config.providers.source = { adapter: "openai-chat", authMode: "key", apiKey: "fixture-only", baseUrl: "https://source.example/v1", ...(transient ? { transientRetryOn5xx: { attempts: 3 } } : {}) }; + let sourceRequests = 0; + globalThis.fetch = (async () => { + sourceRequests++; + return Response.json({ error: { code: "context_length_exceeded", message: "Source context is full" } }, { status: 400 }); + }) as typeof fetch; + const budget = createRequestExecutionBudget({ maxTotalModelSends: 2, baseSendAllowance: 2, finalRecoveryAllowance: 0, maxAlternateTargetSends: 1, maxTargetTransitions: 1 }); + const reservation = budget.reserveDispatch({ sendClass: "initial", targetKey: "source/swe-2", countedExternally: true }); + expect(reservation.allowed).toBe(true); + const response = await handleResponses(request(), config, { model: "", provider: "" }, { sendBudget: budget }); + expect((await response.json()).status).toBe("completed"); + expect(sourceRequests).toBe(1); + expect(calls.map(call => call.model)).toEqual(["rescue"]); + expect(budget.used).toBe(2); + }); + + test("runTurn emergency consumes its prepaid permit once rather than taking another send", async () => { + const budget = createRequestExecutionBudget({ maxTotalModelSends: 2, baseSendAllowance: 2, finalRecoveryAllowance: 0, maxAlternateTargetSends: 1, maxTargetTransitions: 1 }); + const response = await handleResponses(request(), settings(), { model: "", provider: "" }, { sendBudget: budget }); + expect((await response.json()).status).toBe("completed"); + expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); + expect(budget.used).toBe(2); + }); + + test("emergency transient 5xx retry cannot exceed the shared cap", async () => { + const config = settings(); + config.providers.emergency = { adapter: "openai-chat", authMode: "key", apiKey: "fixture-only", baseUrl: "https://emergency.example/v1", transientRetryOn5xx: { attempts: 3 } }; + let emergencyRequests = 0; + globalThis.fetch = (async () => { + emergencyRequests++; + return Response.json({ error: { code: "server_error", message: "Emergency unavailable" } }, { status: 500 }); + }) as typeof fetch; + const budget = createRequestExecutionBudget({ maxTotalModelSends: 4, baseSendAllowance: 4, finalRecoveryAllowance: 0, maxAlternateTargetSends: 1, maxTargetTransitions: 1 }); + const response = await handleResponses(request(), config, { model: "", provider: "" }, { sendBudget: budget }); + expect(await response.text()).toContain("Source rejected compact fixture"); + expect(emergencyRequests).toBe(3); + expect(1 + emergencyRequests).toBeLessThanOrEqual(4); + expect(budget.used).toBe(1 + emergencyRequests); + }); + + test("an emergency rejected before sending refunds the unused reservation", async () => { + const config = settings(); + config.providers.emergency = { adapter: "openai-chat", authMode: "key", baseUrl: "https://emergency.example/v1" }; + const budget = createRequestExecutionBudget(); + const response = await handleResponses(request(), config, { model: "", provider: "" }, { sendBudget: budget }); + expect(await response.text()).toContain("Source rejected compact fixture"); + expect(calls.map(call => call.model)).toEqual(["swe-2"]); + expect(budget.used).toBe(1); + expect(budget.alternateTargetSends).toBe(0); + }); + + test.each(["disabled", "generic-400", "policy", "auth", "partial", "side-effect", "same-model", "opaque", "continuation"])("keeps source failure: %s", async variant => { + const config = settings(); + const payload = body(true); + if (variant === "disabled") config.compactionRecovery!.allowDevinInvalidArgument = false; + if (variant === "generic-400") sourceEvents = [{ ...sourceError, code: "invalid_request_error" } as AdapterEvent]; + if (variant === "policy") sourceEvents = [{ ...sourceError, code: "cyber_policy" } as AdapterEvent]; + if (variant === "auth") sourceEvents = [{ ...sourceError, status: 403, code: "permission_denied" } as AdapterEvent]; + if (variant === "partial") sourceEvents = [{ type: "text_delta", text: "Partial source text" }, sourceError]; + if (variant === "side-effect") sourceEvents = [{ type: "heartbeat", replayUnsafe: true }, sourceError]; + if (variant === "same-model") config.compactionRecovery!.model = "source/swe-2"; + if (variant === "opaque") (payload.input as unknown[]).unshift({ type: "compaction", encrypted_content: "native-opaque-fixture" }); + if (variant === "continuation") payload.previous_response_id = "missing-fixture"; + const response = await handleResponses(request(payload), config, { model: "", provider: "" }); + await response.text(); + expect(calls.filter(call => call.model === "rescue")).toHaveLength(0); + if (variant !== "continuation") expect(calls.map(call => call.model)).toEqual(["swe-2"]); + }); + + test.each(["error", "empty", "truncated"])("failed emergency %s preserves the source error without recursive recovery", async outcome => { + fallbackEvents = outcome === "error" ? [{ ...sourceError, message: "Different emergency failure" } as AdapterEvent] + : outcome === "empty" ? [{ type: "done" }] + : [{ type: "text_delta", text: "Truncated summary" }, { type: "done", stopReason: "max_tokens" }]; + const response = await handleResponses(request(), settings(), { model: "", provider: "" }); + const text = await response.text(); + expect(text).toContain("Source rejected compact fixture"); + expect(text).not.toContain("Different emergency failure"); + expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); + }); + + test("cancellation after source error does not dispatch emergency", async () => { + abortOnSource = new AbortController(); + const response = await handleResponses(request(body(), "responses", abortOnSource.signal), settings(), { model: "", provider: "" }); + await response.text(); + expect(calls.map(call => call.model)).toEqual(["swe-2"]); + }); + + test("one shared send budget blocks emergency when the source consumes the allowance", async () => { + const sendBudget = createRequestExecutionBudget({ maxTotalModelSends: 1, baseSendAllowance: 1, finalRecoveryAllowance: 0, maxAlternateTargetSends: 0, maxTargetTransitions: 0 }); + const translatorBudget = createTranslatorBudget(); + try { + const response = await handleResponses(request(), settings(), { model: "", provider: "" }, { sendBudget, translatorBudget }); + await response.text(); + expect(calls.map(call => call.model)).toEqual(["swe-2"]); + expect(sendBudget.used).toBe(1); + // The ingress-owned body observation remains until its caller disposes the shared budget; + // the additional recovery snapshot has already released its separate retained charge. + expect(translatorBudget.snapshot().currentBytes).toBe(jsonUtf8Bytes(body())); + } finally { translatorBudget.dispose(); } + expect(translatorBudget.snapshot().currentBytes).toBe(0); + }); + + test("canonical native v1 stays on its existing compact path and never invokes routed recovery", async () => { + const config = settings(); + config.providers["openai-apikey"] = { adapter: "openai-responses", authMode: "key", baseUrl: "https://api.openai.com/v1", apiKey: "fixture-only" }; + const urls: string[] = []; + globalThis.fetch = (async (input: unknown) => { + urls.push(String(input)); + return Response.json({ error: { code: "invalid_argument", message: "Native compact fixture failure" } }, { status: 400 }); + }) as typeof fetch; + const response = await handleResponsesCompact(request({ ...body(false, false), model: "openai-apikey/gpt-4.1" }, "responses/compact"), config, { model: "", provider: "" }); + expect(response.status).toBe(400); + await response.text(); + expect(urls).toEqual(["https://api.openai.com/v1/responses/compact"]); + expect(calls).toHaveLength(0); + }); +}); From d5e8e7ef6800f47c6752ceefc1ff3eac750b0d48 Mon Sep 17 00:00:00 2001 From: halysondev Date: Sun, 27 Sep 2026 02:45:33 +0900 Subject: [PATCH 04/19] feat(plugins): load local plugins and expose an upstream rewrite slot (#5896) Carried from #5896 as one squashed commit. Co-authored-by: halysondev --- docs-site/astro.config.mjs | 1 + .../src/content/docs/guides/local-plugins.md | 104 ++++++++ scripts/test-layout/layout.json | 2 + src/cli/index.ts | 1 + src/plugins/loader.ts | 247 ++++++++++++++++++ src/plugins/upstream-hooks.ts | 120 +++++++++ src/server/responses/codex-ws-pool.ts | 9 +- src/server/responses/codex-ws-request.ts | 5 +- src/server/responses/fetch-helpers.ts | 32 ++- src/server/responses/ws-upstream.ts | 48 +++- structure/INDEX.md | 2 + structure/manifest.json | 9 + structure/ops/plugins.md | 68 +++++ tests/fixtures/test-layout-expected.json | 2 + tests/lib/plugin-loader.test.ts | 235 +++++++++++++++++ tests/lib/plugin-upstream-hooks.test.ts | 204 +++++++++++++++ .../responses-fetch-helpers-boundary.test.ts | 2 + 17 files changed, 1077 insertions(+), 14 deletions(-) create mode 100644 docs-site/src/content/docs/guides/local-plugins.md create mode 100644 src/plugins/loader.ts create mode 100644 src/plugins/upstream-hooks.ts create mode 100644 structure/ops/plugins.md create mode 100644 tests/lib/plugin-loader.test.ts create mode 100644 tests/lib/plugin-upstream-hooks.test.ts diff --git a/docs-site/astro.config.mjs b/docs-site/astro.config.mjs index fc4162f8879..628a59942d5 100644 --- a/docs-site/astro.config.mjs +++ b/docs-site/astro.config.mjs @@ -110,6 +110,7 @@ export default defineConfig({ { label: "Integrations", translations: { fr: "Intégrations", ko: "연동", "zh-CN": "集成", "zh-TW": "整合", ru: "Интеграции", ja: "連携", tr: "Entegrasyonlar" }, slug: "guides/integrations" }, { label: "MiniMax clients", translations: { fr: "Clients MiniMax", ko: "MiniMax 클라이언트", "zh-CN": "MiniMax 客户端", "zh-TW": "MiniMax 客戶端", ru: "Клиенты MiniMax", ja: "MiniMax クライアント", tr: "MiniMax İstemcileri" }, slug: "guides/minimax" }, { label: "Sidecars: Web Search & Vision", translations: { fr: "Services auxiliaires : recherche web et vision", ko: "사이드카: 웹 검색 & 비전", "zh-CN": "边车：网络搜索与视觉", "zh-TW": "邊車：網路搜尋與視覺", ru: "Сайдкары: веб-поиск и зрение", ja: "サイドカー: ウェブ検索 & ビジョン", tr: "Sidecar'lar: Web Arama ve Görme" }, slug: "guides/sidecars" }, + { label: "Local Plugins", translations: { fr: "Plugins locaux", ko: "로컬 플러그인", "zh-CN": "本地插件", "zh-TW": "本機外掛", ru: "Локальные плагины", ja: "ローカルプラグイン", tr: "Yerel Eklentiler" }, slug: "guides/local-plugins" }, { label: "Image Bridge", translations: { fr: "Pont d’images", ko: "이미지 브릿지", "zh-CN": "图像桥接", "zh-TW": "圖像橋接", ru: "Image Bridge", ja: "画像ブリッジ", tr: "Image Bridge" }, slug: "guides/image-bridge" }, { label: "Video Bridge", translations: { fr: "Pont vidéo", ko: "비디오 브릿지", "zh-CN": "视频桥接", "zh-TW": "影片橋接", ru: "Video Bridge", ja: "動画ブリッジ", tr: "Video Bridge" }, slug: "guides/video-bridge" }, { label: "Web Dashboard", translations: { fr: "Tableau de bord web", ko: "웹 대시보드", "zh-CN": "网页控制台", "zh-TW": "網頁儀表板", ru: "Веб-дашборд", ja: "ウェブダッシュボード", tr: "Web Kontrol Paneli" }, slug: "guides/web-dashboard" }, diff --git a/docs-site/src/content/docs/guides/local-plugins.md b/docs-site/src/content/docs/guides/local-plugins.md new file mode 100644 index 00000000000..a7864b7d4c1 --- /dev/null +++ b/docs-site/src/content/docs/guides/local-plugins.md @@ -0,0 +1,104 @@ +--- +title: Local Plugins +description: Load your own code into the proxy at startup to rewrite provider sends, for example to put a local compression proxy in front of providers. +--- + +A local plugin is a TypeScript or JavaScript file that `ocx start` loads before the proxy begins +serving. It can see every provider send just before it leaves the process and redirect it or add +headers — enough to place a local sidecar (a compression proxy, a recorder) in front of providers +without changing opencodex itself. + +Plugins are local to one install. opencodex does not download, update or sign them. + +## Where plugins live + +Put plugin files in `plugins/` inside the opencodex home (`~/.opencodex/plugins/`, or +`$OPENCODEX_HOME/plugins/` when that variable is set): + +```text +~/.opencodex/plugins/ + my-sidecar.ts +``` + +- Files ending in `.ts`, `.js` or `.mjs` are loaded in name order. +- Names starting with `.` or `_`, and `*.d.ts`, are ignored — rename a plugin to `_my-sidecar.ts` + to switch it off. +- The directory is optional. Without it nothing is loaded. +- A plugin runs inside the proxy with your credentials, so opencodex refuses a plugin file or a + `plugins/` directory that is owned by another user or writable by group or others, and refuses + symbolic links. Every directory above `plugins/`, up to `/`, must also be owned by you or root and + not writable by group or others, unless it is sticky like `/tmp`. Fix permissions with + `chmod go-w ~/.opencodex/plugins ~/.opencodex/plugins/*`; on systems whose default umask is + `002`, check the parent directories too. +- On Windows these owner and permission checks are not performed; only regular files are loaded. + Keep the `plugins/` directory writable by your account only. + +Restart the proxy after adding, changing or removing a plugin (`ocx service restart`, or stop and +start `ocx start`). Each loaded plugin prints a `Plugin loaded: ` line at startup; a skipped +plugin prints the reason. + +To start once without plugins, set `OCX_PLUGINS=0`. + +## Writing a plugin + +A plugin default-exports an object with an optional `name` and a `setup` function. `setup` receives a +context. An asynchronous `setup` has five seconds to finish; plugins run in the proxy's own thread, so +a `setup` that blocks synchronously cannot be interrupted and delays startup until it returns. A +`setup` that times out is not stopped either: servers or timers it already started keep running, so +start long-lived resources only after the work that can fail. + +```ts +interface UpstreamTarget { + url: string; // absolute upstream URL; assign a new one to redirect + headers: Headers; // outbound headers, including credentials — never log them + readonly transport: "http" | "websocket"; +} + +export default { + name: "my-sidecar", + setup(ctx: { + log(message: string): void; + registerUpstreamRewriter(rewrite: (target: UpstreamTarget) => void): void; + onShutdown(teardown: () => void): void; + }) { + ctx.registerUpstreamRewriter(target => { + const upstream = new URL(target.url); + if (!upstream.pathname.endsWith("/chat/completions")) return; + target.url = `http://127.0.0.1:9000${upstream.pathname}${upstream.search}`; + target.headers.set("x-original-origin", upstream.origin); + }); + }, +}; +``` + +The context also carries `name`, `configDir` (the opencodex home) and `pluginDir`. + +Plugins cannot import opencodex modules — in the packaged binary they are not on disk. Declare the +small interfaces you need locally, as above. + +## How rewrites behave + +- The rewriter runs synchronously on every provider send over HTTP and on the Codex WebSocket + connection, after opencodex has picked the transport. Keep it fast; do network checks (health + probes) in the background and read a cached result in the rewriter. +- A send redirected to a loopback address (`127.0.0.1`, `::1`, `localhost`) connects directly, over + HTTP and over the Codex WebSocket, ignoring provider proxies and `HTTP_PROXY`: a proxy elsewhere + cannot reach this machine's loopback. Any other destination follows the normal egress settings + (including `NO_PROXY`), evaluated against the rewritten URL, on both transports. +- The Codex WebSocket rewriter runs for every turn, before an idle pooled socket is reused, and a + socket is only reused for the same destination and the same rewritten headers, apart from the two + per-turn headers `x-codex-turn-state` and `x-codex-turn-metadata`. Those travel inside each + request frame and may differ between exchanges on one socket; changes a rewriter makes to them + are discarded. A plugin that starts or stops redirecting takes + effect on the next turn. +- It runs after opencodex has chosen the provider, account and route, so it does not change routing, + account selection, retries or request logs. +- A redirected send goes to the host you chose. That host sees the request exactly as the provider + would, credentials included. +- If a rewriter throws, opencodex undoes that rewriter's edits to the send and disables it for the + rest of the process. Edits made by rewriters that ran before it are kept, so the send goes out as + those left it (unmodified when it is the only plugin). If `setup` throws or times out, the plugin is skipped, + anything it registered is removed, and later registration attempts from it are ignored; other + plugins and the proxy start normally. +- A plugin directory that exists but cannot be read (for example, wrong permissions) is reported at + startup rather than treated as empty. diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 0a69213201d..234f1d261d0 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -176,6 +176,8 @@ "openai-chat-serialized-tool-call-scaling.test.ts": "adapters/openai", "openai-chat-tool-call-id-remint.test.ts": "adapters/openai", "coding-agent-json-lines-scaling.test.ts": "providers", + "plugin-loader.test.ts": "lib", + "plugin-upstream-hooks.test.ts": "lib", "usage-snapshot-digest-reuse.test.ts": "usage", "release-desktop-scripts.test.ts": "ci-workflows", "installed-gate-drivers.test.ts": "ci-workflows", diff --git a/src/cli/index.ts b/src/cli/index.ts index 827dbd6701b..11a8c9a2523 100755 --- a/src/cli/index.ts +++ b/src/cli/index.ts @@ -529,6 +529,7 @@ async function handleStart(options: { block?: boolean } = {}) { const readinessGate = createReadinessGate(); const localAttestationSecret = createLocalAttestationSecret(); const config = loadConfig(); + await (await import("../plugins/loader")).loadAndReportOcxPlugins(); let server: ReturnType; for (let attempt = 0; ; attempt++) { try { diff --git a/src/plugins/loader.ts b/src/plugins/loader.ts new file mode 100644 index 00000000000..aaa4e27a4cc --- /dev/null +++ b/src/plugins/loader.ts @@ -0,0 +1,247 @@ +/** + * Local plugin loader. + * + * `ocx start` imports every `*.ts`, `*.js` and `*.mjs` file in `$OPENCODEX_HOME/plugins/` + * before the server binds, so a plugin's hooks are in place for the first request. A missing + * directory means no plugins and no work. `OCX_PLUGINS=0` disables loading for one start. + * + * A plugin is a module whose default export is `{ name?, setup(ctx) }`. It runs in the proxy + * process with the operator's credentials, so the loader accepts only files owned by the + * current user that no other user can write — the same trust boundary as `config.json`. + * Plugins cannot import ocx internals (a compiled binary keeps them inside `$bunfs`); they + * receive everything they may use through `OcxPluginContext`. + * + * Failures are contained: a plugin that throws, times out or has the wrong shape is reported + * and skipped, its context stops accepting registrations, and the remaining plugins and the + * proxy start normally. The setup deadline bounds setup that yields to the event loop; plugins + * run in the proxy's own thread, so synchronous work that never yields cannot be interrupted. + */ + +import { lstatSync, readdirSync, realpathSync } from "node:fs"; +import { basename, dirname, join } from "node:path"; +import { pathToFileURL } from "node:url"; +import { getConfigDir } from "../config/paths"; +import { registerOptionalShutdownHook } from "../lib/optional-shutdown-hooks"; +import { registerUpstreamRewriter, type UpstreamRewriter } from "./upstream-hooks"; + +export type { UpstreamRewriter, UpstreamTarget, UpstreamTransport } from "./upstream-hooks"; + +export interface OcxPluginContext { + /** The plugin's own name, as reported in logs. */ + readonly name: string; + /** `$OPENCODEX_HOME`, for plugins that keep state next to the proxy's. */ + readonly configDir: string; + /** `$OPENCODEX_HOME/plugins`. */ + readonly pluginDir: string; + log(message: string): void; + /** See `src/plugins/upstream-hooks.ts`. Called synchronously on every provider send. */ + registerUpstreamRewriter(rewrite: UpstreamRewriter): void; + /** Runs once when the proxy shuts down. Must not throw or block. */ + onShutdown(teardown: () => void): void; +} + +export interface OcxPlugin { + name?: string; + setup(context: OcxPluginContext): void | Promise; +} + +export interface PluginLoadResult { + file: string; + name: string; + loaded: boolean; + error?: string; +} + +const PLUGIN_EXTENSIONS = [".ts", ".js", ".mjs"]; +const SETUP_TIMEOUT_MS = 5_000; + +export function pluginDirectory(): string { + return join(getConfigDir(), "plugins"); +} + +/** A missing directory is "no plugins"; any other read failure propagates to be reported. */ +function listPluginFiles(dir: string): string[] { + let entries: string[]; + try { + entries = readdirSync(dir); + } catch (error) { + if ((error as NodeJS.ErrnoException).code === "ENOENT") return []; + throw error; + } + return entries + .filter(entry => !entry.startsWith(".") && !entry.startsWith("_") && !entry.endsWith(".d.ts")) + .filter(entry => PLUGIN_EXTENSIONS.some(extension => entry.endsWith(extension))) + .sort() + .map(entry => join(dir, entry)); +} + +/** + * Null when `path` is safe to trust, otherwise the reason it is refused. `lstat` is used so a + * symbolic link is judged as a link — and refused — rather than as the file it points to: a + * link to a file you own would otherwise pass the owner and mode checks. Windows has no + * POSIX owner or mode bits, so there only the file type is checked. + */ +function trustError(path: string, kind: "file" | "directory"): string | null { + let stats: ReturnType; + try { + stats = lstatSync(path); + } catch (error) { + return error instanceof Error ? error.message : String(error); + } + if (stats.isSymbolicLink()) return "is a symbolic link"; + if (kind === "file" ? !stats.isFile() : !stats.isDirectory()) return `not a regular ${kind}`; + if (process.platform === "win32") return null; + const uid = process.getuid?.(); + if (uid !== undefined && stats.uid !== uid) return "owned by another user"; + if ((stats.mode & 0o022) !== 0) return "writable by group or others (chmod go-w)"; + return null; +} + +/** Null when the file is safe to execute, otherwise the reason it is refused. */ +export function pluginFileTrustError(file: string): string | null { + return trustError(file, "file"); +} + +/** + * A directory another user can write lets them add or swap plugin files, so its owner and + * mode are checked like each file's. + */ +export function pluginDirectoryTrustError(dir: string): string | null { + return trustError(dir, "directory"); +} + +/** + * Every directory above the (resolved) plugin directory, up to `/`, must be owned by the user + * or root and not writable by group or others unless it is sticky (like `/tmp`), where others + * cannot rename or replace entries they do not own. With no writable component on the path, no + * other user can swap what the loader checked for something else before it is imported + * (OpenSSH StrictModes applies the same rule). POSIX only. + */ +export function pluginAncestorsTrustError(realDir: string): string | null { + if (process.platform === "win32") return null; + const uid = process.getuid?.(); + let current = dirname(realDir); + for (;;) { + let stats: ReturnType; + try { + stats = lstatSync(current); + } catch (error) { + return `${current}: ${error instanceof Error ? error.message : String(error)}`; + } + if (uid !== undefined && stats.uid !== uid && stats.uid !== 0) return `${current} is owned by another user`; + if ((stats.mode & 0o022) !== 0 && (stats.mode & 0o1000) === 0) { + return `${current} is writable by group or others`; + } + const parent = dirname(current); + if (parent === current) return null; + current = parent; + } +} + +function isPlugin(value: unknown): value is OcxPlugin { + return typeof value === "object" && value !== null && typeof (value as OcxPlugin).setup === "function"; +} + +async function withTimeout(work: Promise, ms: number, label: string): Promise { + let timer: ReturnType | undefined; + const timeout = new Promise((_, reject) => { + timer = setTimeout(() => reject(new Error(`${label} did not finish within ${ms}ms`)), ms); + }); + try { + return await Promise.race([work, timeout]); + } finally { + clearTimeout(timer); + } +} + +export interface LoadOcxPluginsOptions { + /** Deadline for a setup that yields; see the module comment. */ + setupTimeoutMs?: number; +} + +export async function loadOcxPlugins( + dir = pluginDirectory(), + options: LoadOcxPluginsOptions = {}, +): Promise { + if (process.env["OCX_PLUGINS"] === "0") return []; + let files: string[]; + try { + files = listPluginFiles(dir); + } catch (error) { + return [{ file: dir, name: "plugins directory", loaded: false, error: error instanceof Error ? error.message : String(error) }]; + } + if (files.length === 0) return []; + const dirRefused = pluginDirectoryTrustError(dir); + if (dirRefused) return [{ file: dir, name: "plugins directory", loaded: false, error: `refused: ${dirRefused}` }]; + // Check and import through the resolved path, so both refer to the same components. + let realDir: string; + try { + realDir = realpathSync(dir); + } catch (error) { + return [{ file: dir, name: "plugins directory", loaded: false, error: error instanceof Error ? error.message : String(error) }]; + } + const ancestorRefused = pluginAncestorsTrustError(realDir); + if (ancestorRefused) return [{ file: dir, name: "plugins directory", loaded: false, error: `refused: ${ancestorRefused}` }]; + const results: PluginLoadResult[] = []; + for (const file of files.map(listed => join(realDir, basename(listed)))) { + const fallbackName = basename(file).replace(/\.(ts|js|mjs)$/, ""); + const refused = pluginFileTrustError(file); + if (refused) { + results.push({ file, name: fallbackName, loaded: false, error: `refused: ${refused}` }); + continue; + } + const unregister: Array<() => void> = []; + // Closed when setup fails or times out: a setup that resumes later must not register. + let active = true; + let shutdownCount = 0; + const whileActive = (name: string, register: () => () => void): void => { + if (!active) { + console.error(`[plugin:${name}] registration after a failed setup was ignored`); + return; + } + unregister.push(register()); + }; + try { + const module = await import(pathToFileURL(file).href) as { default?: unknown; plugin?: unknown }; + const plugin = module.default ?? module.plugin; + if (!isPlugin(plugin)) throw new Error("default export must be { name?, setup(context) }"); + const name = typeof plugin.name === "string" && plugin.name.trim() ? plugin.name.trim() : fallbackName; + const context: OcxPluginContext = { + name, + configDir: getConfigDir(), + pluginDir: dir, + log: message => console.log(`[plugin:${name}] ${message}`), + registerUpstreamRewriter: rewrite => whileActive(name, () => registerUpstreamRewriter(name, rewrite)), + // Keyed by file and registration, not name: two plugins may share a display name, and + // one plugin may register several teardowns. + onShutdown: teardown => whileActive(name, () => registerOptionalShutdownHook(`plugin:${file}#${++shutdownCount}`, teardown)), + }; + const timeoutMs = options.setupTimeoutMs ?? SETUP_TIMEOUT_MS; + await withTimeout(Promise.resolve(plugin.setup(context)), timeoutMs, `plugin "${name}" setup`); + results.push({ file, name, loaded: true }); + } catch (error) { + // A half-initialised plugin must not leave hooks behind, now or later. + active = false; + for (const undo of unregister) undo(); + results.push({ + file, + name: fallbackName, + loaded: false, + error: error instanceof Error ? error.message : String(error), + }); + } + } + return results; +} + +/** `ocx start` entry: load and print one line per plugin. Never throws. */ +export async function loadAndReportOcxPlugins(): Promise { + try { + for (const result of await loadOcxPlugins()) { + if (result.loaded) console.log(`🔌 Plugin loaded: ${result.name}`); + else console.error(`⚠️ Plugin ${result.name} skipped: ${result.error}`); + } + } catch (error) { + console.error(`⚠️ Plugin loading failed: ${error instanceof Error ? error.message : String(error)}`); + } +} diff --git a/src/plugins/upstream-hooks.ts b/src/plugins/upstream-hooks.ts new file mode 100644 index 00000000000..77436d1b015 --- /dev/null +++ b/src/plugins/upstream-hooks.ts @@ -0,0 +1,120 @@ +/** + * Upstream rewrite slot for local plugins. + * + * A plugin loaded by `src/plugins/loader.ts` may register a rewriter that sees every + * provider send at the physical boundary, after the transport was chosen: HTTP in + * `sendWithConnectionPolicy` and the Codex WebSocket dial in `CodexWsSession`. The rewriter + * may replace the URL and add or change headers — enough to put a local sidecar (a + * compression proxy, a recorder) in front of the provider without the core knowing it exists. + * + * This module imports nothing, so the request path pays one array-length check when no + * plugin is installed. A rewriter that throws is disabled for the rest of the process and + * the send continues unmodified: a broken plugin must never take the proxy down with it. + */ + +export type UpstreamTransport = "http" | "websocket"; + +export interface UpstreamTarget { + /** Absolute upstream URL. A rewriter may assign a new one. */ + url: string; + /** Mutable outbound headers. Credentials are present; a rewriter must not log them. */ + headers: Headers; + readonly transport: UpstreamTransport; +} + +export type UpstreamRewriter = (target: UpstreamTarget) => void; + +interface Registration { + readonly name: string; + readonly rewrite: UpstreamRewriter; + disabled: boolean; +} + +const registrations: Registration[] = []; + +export function registerUpstreamRewriter(name: string, rewrite: UpstreamRewriter): () => void { + const registration: Registration = { name, rewrite, disabled: false }; + registrations.push(registration); + return () => { + const index = registrations.indexOf(registration); + if (index >= 0) registrations.splice(index, 1); + }; +} + +export function hasUpstreamRewriters(): boolean { + return registrations.length > 0; +} + +/** + * Run every active rewriter over one send. Returns the input untouched (same objects) when + * no rewriter is registered, so the common path allocates nothing. + */ +export function rewriteUpstream( + url: string, + headers: H, + transport: UpstreamTransport, +): { url: string; headers: H | Headers } { + if (registrations.length === 0) return { url, headers }; + const target: UpstreamTarget = { url, headers: new Headers(headers), transport }; + for (const registration of registrations) { + if (registration.disabled) continue; + // A rewriter that edits the target and then throws must not leave a half-rewritten send + // for the next rewriter or the network. + const urlBefore = target.url; + const headersBefore = new Headers(target.headers); + try { + registration.rewrite(target); + } catch (error) { + target.url = urlBefore; + target.headers = headersBefore; + registration.disabled = true; + const reason = error instanceof Error ? error.message : String(error); + console.error(`[opencodex] plugin "${registration.name}" upstream rewriter disabled after an error: ${reason}`); + } + } + return { url: target.url, headers: target.headers }; +} + +/** Plain-record variant for callers that hold headers as `Record` (WebSocket dial). */ +export function rewriteUpstreamRecord( + url: string, + headers: Record, + transport: UpstreamTransport, +): { url: string; headers: Record } { + if (registrations.length === 0) return { url, headers }; + const result = rewriteUpstream(url, headers, transport); + const record: Record = {}; + new Headers(result.headers).forEach((value, key) => { record[key] = value; }); + return { url: result.url, headers: record }; +} + +export function isLoopbackUrl(raw: string): boolean { + let host: string; + try { + host = new URL(raw).hostname.toLowerCase().replace(/^\[|\]$/g, ""); + } catch { + return false; + } + return host === "localhost" || host.endsWith(".localhost") || host === "::1" || /^127\.\d{1,3}\.\d{1,3}\.\d{1,3}$/.test(host); +} + +/** + * WebSocket dial variant, run for every exchange before a pooled socket is chosen so the pool + * identity can include the rewritten destination. The caller chose `proxy` for the original + * destination; a proxy elsewhere on the network cannot reach this machine's loopback, so a + * rewrite onto a loopback sidecar dials directly. Any other rewrite keeps the caller's proxy. + */ +export function rewriteWebSocketDial( + url: string, + headers: Record, + proxy: string | undefined, +): { url: string; headers: Record; proxy: string | undefined } { + if (registrations.length === 0) return { url, headers, proxy }; + const target = rewriteUpstreamRecord(url, headers, "websocket"); + const redirectedToLoopback = target.url !== url && isLoopbackUrl(target.url); + return { ...target, proxy: redirectedToLoopback ? undefined : proxy }; +} + +export function resetUpstreamRewritersForTests(): void { + registrations.length = 0; +} diff --git a/src/server/responses/codex-ws-pool.ts b/src/server/responses/codex-ws-pool.ts index 5d406bee4f8..73c99d5a89b 100644 --- a/src/server/responses/codex-ws-pool.ts +++ b/src/server/responses/codex-ws-pool.ts @@ -1,13 +1,13 @@ import { createHmac, randomBytes } from "node:crypto"; import { registerOptionalShutdownHook } from "../../lib/optional-shutdown-hooks"; -import { CODEX_RESPONSES_HTTP_URL } from "./codex-ws-request"; +import { CODEX_RESPONSES_HTTP_URL, CODEX_WS_FRAME_HEADERS } from "./codex-ws-request"; import { CODEX_WS_ID_MAX_BYTES } from "./codex-ws-correlation"; import { CodexWsSession } from "./codex-ws-session"; export const CODEX_WS_POOL_MAX_SESSIONS = 32; export const CODEX_WS_POOL_IDLE_MS = 30_000; export const CODEX_WS_POOL_MAX_AGE_MS = 5 * 60_000; -const MUTABLE_HEADERS = new Set(["x-codex-turn-state", "x-codex-turn-metadata"]); +const MUTABLE_HEADERS = new Set(CODEX_WS_FRAME_HEADERS); let processKey: Buffer | undefined; let poolSequence = 0; @@ -25,7 +25,7 @@ function digest(input: unknown): string { } /** Identity comes from the selected outgoing request, never a model label or caller hint. */ -export function codexWsReuseIdentity(url: string, headers: Record, frameText: string, proxy?: string): CodexWsReuseIdentity | null { +export function codexWsReuseIdentity(url: string, headers: Record, frameText: string, proxy?: string, dialUrl?: string): CodexWsReuseIdentity | null { if (url !== CODEX_RESPONSES_HTTP_URL) return null; let body: unknown; try { body = JSON.parse(frameText); } catch { return null; } @@ -52,7 +52,8 @@ export function codexWsReuseIdentity(url: string, headers: Record, headers: Headers): boo body.client_metadata = { ...(metadata as Record | undefined), [CODEX_RESPONSES_LITE_METADATA_KEY]: lite }; } - for (const name of ["x-codex-turn-state", "x-codex-turn-metadata"]) { + for (const name of CODEX_WS_FRAME_HEADERS) { const value = headers.get(name); const current = body.client_metadata as Record | undefined; if (value !== null && !Object.hasOwn(current ?? {}, name)) { diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index 7bae07896f1..83f16a83b18 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -17,6 +17,7 @@ import { import { withUpstreamHttpVersion } from "../../lib/upstream-http-version"; import type { CodexWsQuotaObserver } from "./codex-ws-metadata"; import { configuredOutboundFetch } from "../../lib/proxy-env"; +import { isLoopbackUrl, rewriteUpstream } from "../../plugins/upstream-hooks"; import { describeProviderEgressForLog, markEgressTransparentExecutor, @@ -40,6 +41,7 @@ const pacingWebsocketDowngradeWarned = new Set(); * `dispatchOverride` performs, and an unknown symbol on a `RequestInit` is inert at the wire. */ const EGRESS_DECIDED = Symbol.for("opencodex.provider-egress.decided"); +const UPSTREAM_REWRITTEN = Symbol.for("opencodex.plugins.upstream-rewritten"); /** * Announce once, per provider, that an explicit egress route moved this provider off the @@ -150,11 +152,28 @@ export type ProviderFetch = typeof globalThis.fetch & PaceAwareFetch; */ export function sendWithConnectionPolicy( physicalFetch: typeof globalThis.fetch, - input: Parameters[0], + rawInput: Parameters[0], init?: RequestInit, egress?: ProviderEgressBinding, ): Promise { - const headers = new Headers(init?.headers ?? (input instanceof Request ? input.headers : undefined)); + let input = rawInput; + let headers = new Headers(init?.headers ?? (input instanceof Request ? input.headers : undefined)); + // Plugin rewrites (src/plugins/upstream-hooks.ts) run here, after the caller chose between + // the Codex WebSocket and HTTP, and before the connection and egress decisions below so + // those follow the rewritten destination. Nested passes rewrite once, like the egress mark. + // A rewrite onto this machine's loopback dials directly: a proxy chosen for the provider + // (per-provider or HTTP_PROXY) cannot reach a local sidecar. The WebSocket dial does the same. + const rewriteDone = (init as Record | undefined)?.[UPSTREAM_REWRITTEN] === true; + let redirectedToLoopback = false; + if (!rewriteDone) { + const original = input instanceof Request ? input.url : String(input); + const target = rewriteUpstream(original, headers, "http"); + headers = target.headers as Headers; + if (target.url !== original) { + redirectedToLoopback = isLoopbackUrl(target.url); + input = input instanceof Request ? new Request(target.url, input) : target.url; + } + } const fresh = wantsFreshConnection(input); if (fresh) { headers.set("Connection", "close"); @@ -168,15 +187,18 @@ export function sendWithConnectionPolicy( // the reselected provider and the rebuilt destination, so it decides and marks the init; the // inner pass honours that mark rather than recomputing from a stale closure. const alreadyDecided = (init as Record | undefined)?.[EGRESS_DECIDED] === true; - const decide = egress !== undefined && !alreadyDecided; - const egressInit = decide ? providerEgressSendInit(egress, physicalFetch, input) : {}; + const decide = egress !== undefined && !alreadyDecided && !redirectedToLoopback; + const egressInit = redirectedToLoopback + ? { proxy: false as const } + : decide ? providerEgressSendInit(egress, physicalFetch, input) : {}; return physicalFetch(input, { ...init, headers, redirect: "manual", ...(fresh ? { keepalive: false } : {}), ...egressInit, - ...(decide ? { [EGRESS_DECIDED]: true } : {}), + ...(decide || redirectedToLoopback ? { [EGRESS_DECIDED]: true } : {}), + ...{ [UPSTREAM_REWRITTEN]: true }, }); } diff --git a/src/server/responses/ws-upstream.ts b/src/server/responses/ws-upstream.ts index 7e7af0791e4..4363a39d24d 100644 --- a/src/server/responses/ws-upstream.ts +++ b/src/server/responses/ws-upstream.ts @@ -18,11 +18,12 @@ import type { NativeResponseControl } from "./native-response-control"; import { compareBunVersions } from "../../lib/bun-stream-caps"; import { resolveProxyRoute, socks5ProxyFromEnv } from "../../lib/proxy-env"; import type { CodexWsQuotaObserver } from "./codex-ws-metadata"; -import { CODEX_RESPONSES_HTTP_URL, CODEX_RESPONSES_WS_URL, prepareCodexHttpInit, prepareCodexWsRequest } from "./codex-ws-request"; +import { CODEX_RESPONSES_HTTP_URL, CODEX_RESPONSES_WS_URL, CODEX_WS_FRAME_HEADERS, prepareCodexHttpInit, prepareCodexWsRequest } from "./codex-ws-request"; import { codexWsExchange } from "./codex-ws-exchange"; import { CodexWsSession } from "./codex-ws-session"; import { codexWsPool, codexWsReuseIdentity } from "./codex-ws-pool"; import { codexWsCreateFrameExceedsLimit } from "./codex-ws-wire"; +import { isLoopbackUrl, rewriteWebSocketDial } from "../../plugins/upstream-hooks"; export { CODEX_WS_LIVENESS_PING_INTERVAL_MS, CODEX_WS_RESPONSE_PRELUDE_TIMEOUT_MS, MAX_CODEX_WS_FRAME_BYTES, MAX_CODEX_WS_QUEUE_BYTES, MAX_CODEX_WS_CREATE_FRAME_BYTES, CODEX_WS_CREATE_FRAME_LIMIT_BYTES, codexWsCreateFrameExceedsLimit, isCodexWsQuotaObservedResponse, isCodexWsUpstreamResponse } from "./codex-ws-wire"; @@ -81,6 +82,40 @@ export function bunSupportsBoundedCodexWsRelay( return comparison !== null && comparison >= 0; } +/** + * Apply plugin rewrites to one Codex WebSocket dial and settle its proxy. `proxy` was resolved + * for the canonical `wsUrl`. A loopback rewrite dials directly; any other rewrite gets its own + * route (scheme and `NO_PROXY` may differ). Null means the rewritten destination needs the SSE + * fallback, exactly as an unusable route for the canonical URL does. + */ +export function planCodexWsDial( + wsUrl: string, + headers: Record, + proxy: string | undefined, + env: Parameters[1] = process.env, +): { url: string; headers: Record; proxy: string | undefined } | null { + const rewritten = rewriteWebSocketDial(wsUrl, headers, proxy); + // Per-turn headers ride in each frame's client_metadata, which was prepared before the rewrite + // and is authoritative; a pooled socket's upgrade copy is intentionally ignored. A rewriter + // therefore cannot change them here, and the upgrade keeps the values the frame carries. + const dialHeaders = { ...rewritten.headers }; + for (const name of CODEX_WS_FRAME_HEADERS) { + if (Object.hasOwn(headers, name)) dialHeaders[name] = headers[name]!; + else delete dialHeaders[name]; + } + const dial = { ...rewritten, headers: dialHeaders }; + if (dial.url === wsUrl || isLoopbackUrl(dial.url)) return dial; + let destination: URL; + try { + destination = new URL(dial.url); + } catch { + return null; + } + const route = resolveProxyRoute(destination, env); + if (route.kind === "fallback") return null; + return { ...dial, proxy: route.kind === "proxy" ? route.proxy : undefined }; +} + export function shouldUseCodexWsUpstream( url: string, init?: RequestInit, @@ -176,9 +211,14 @@ export function codexWsUpstreamFetch( try { // Steering keeps a private physical connection across successor responses; it // must never enter the idle-socket pool or move to a different credential. - const identity = control ? null : codexWsReuseIdentity(url, headers, frameText, proxy); - session = (identity ? codexWsPool.acquire(identity, wsUrl, headers, proxy) : null) - ?? new CodexWsSession(wsUrl, headers, false, undefined, proxy); + // Plugin rewrite runs per exchange, before the pool lookup. The dialled destination, its + // headers and its proxy are all part of the reuse identity, so a pooled socket is never + // reused for a different destination or with stale plugin headers. + const dial = planCodexWsDial(wsUrl, headers, proxy); + if (!dial) return sseFallback(url, init); + const identity = control ? null : codexWsReuseIdentity(url, dial.headers, frameText, dial.proxy, dial.url); + session = (identity ? codexWsPool.acquire(identity, dial.url, dial.headers, dial.proxy) : null) + ?? new CodexWsSession(dial.url, dial.headers, false, undefined, dial.proxy); if (!session.busy && !session.reserve()) { session.dispose(); return sseFallback(url, init); diff --git a/structure/INDEX.md b/structure/INDEX.md index 4f653bf3fb0..d1a258a6810 100644 --- a/structure/INDEX.md +++ b/structure/INDEX.md @@ -89,6 +89,7 @@ Background service, docs, release, and design discipline. | --- | --- | | [`desktop-shell.md`](desktop-shell.md) | Tauri desktop shell, proxy attachment and sidecar lifecycle, tray controls, bootstrap navigation, and desktop companion presence. | | [`ops/service-and-sidecars.md`](ops/service-and-sidecars.md) | Service install/repair, platform launchers, tray, and sidecar processes. | +| [`ops/plugins.md`](ops/plugins.md) | Plugin loading from OPENCODEX_HOME/plugins and the upstream rewrite slot plugins attach to. | | [`ops/docs-and-release.md`](ops/docs-and-release.md) | Docs site, workflow map, branch policy, release flow, and cross-platform CI. | | [`design-methodology.md`](design-methodology.md) | Stage ordering for new GUI, CLI, and user-facing surfaces. | | [`ops/cross-platform-ci.md`](ops/cross-platform-ci.md) | Test lanes, platform coverage, aggregate gating, and release CI proof. | @@ -134,6 +135,7 @@ A source area can be described by more than one doc, because these docs are orga | `src/lib/` | [`overview.md`](overview.md)
[`runtime.md`](runtime.md)
[`transports/byte-accounting.md`](transports/byte-accounting.md)
[`transports/responses-wire-shapes.md`](transports/responses-wire-shapes.md)
[`transports/responses-failover.md`](transports/responses-failover.md)
[`transports/responses-spend.md`](transports/responses-spend.md)
[`transports/inventory.md`](transports/inventory.md)
[`gui-and-management-api.md`](gui-and-management-api.md)
[`dashboard-and-usage.md`](dashboard-and-usage.md)
[`clients/integrations.md`](clients/integrations.md)
[`ops/docs-and-release.md`](ops/docs-and-release.md) | | `src/link/` | [`remote-link.md`](remote-link.md) | | `src/oauth/` | [`runtime.md`](runtime.md)
[`transports/inventory.md`](transports/inventory.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/xai-grok.md`](providers/xai-grok.md) | +| `src/plugins/` | [`ops/plugins.md`](ops/plugins.md) | | `src/protocols/` | [`data-planes/protocol-paths.md`](data-planes/protocol-paths.md) | | `src/providers/` | [`runtime.md`](runtime.md)
[`subagents.md`](subagents.md)
[`transports/inventory.md`](transports/inventory.md)
[`providers-and-adapters.md`](providers-and-adapters.md)
[`providers/xai-grok.md`](providers/xai-grok.md) | | `src/quota/` | [`dashboard-and-usage.md`](dashboard-and-usage.md) | diff --git a/structure/manifest.json b/structure/manifest.json index 734e7d71aa8..5201bb9bf71 100644 --- a/structure/manifest.json +++ b/structure/manifest.json @@ -456,6 +456,15 @@ "src/update/" ] }, + { + "path": "ops/plugins.md", + "tier": 6, + "title": "Local Plugins", + "scope": "Plugin loading from OPENCODEX_HOME/plugins and the upstream rewrite slot plugins attach to.", + "documents": [ + "src/plugins/" + ] + }, { "path": "ops/docs-and-release.md", "tier": 6, diff --git a/structure/ops/plugins.md b/structure/ops/plugins.md new file mode 100644 index 00000000000..6912a501fc4 --- /dev/null +++ b/structure/ops/plugins.md @@ -0,0 +1,68 @@ +# Local Plugins + +Local plugins let an operator put code in front of provider sends without editing the core. +They are local extensions of one install, not a distribution channel: nothing fetches, updates +or signs them. + +## Loading + +- `ocx start` calls `loadAndReportOcxPlugins()` from `src/plugins/loader.ts` after the config is + loaded and before `startServer`, so every hook is registered before the listener binds. + `startServer` itself stays synchronous; plugin loading is awaited in the CLI, never inside it. +- The loader imports `*.ts`, `*.js` and `*.mjs` from `$OPENCODEX_HOME/plugins/`, sorted by name. + Names starting with `.` or `_` and `*.d.ts` are ignored. A missing directory loads nothing. +- `OCX_PLUGINS=0` disables loading for that process. +- A plugin runs with the operator's credentials, so the loader refuses a plugin file or plugin + directory that is a symbolic link (checked with `lstat`), is not a regular file/directory, is + owned by another user, or is writable by group or others. This is the same trust boundary as + `config.json`. Every ancestor of the resolved plugin directory up to `/` must be owned by the user + or root and not group/other-writable unless sticky (`pluginAncestorsTrustError`), so no other user + can swap a checked path before it is imported; files are imported through the resolved directory. + Owner and mode checks are POSIX-only; on Windows only the file type is checked. +- A missing plugin directory means no plugins. Any other read failure (`EACCES`, `ENOTDIR`) is + reported as a skipped `plugins directory` entry. +- A plugin module default-exports `{ name?, setup(context) }`. An asynchronous `setup` has five + seconds; plugins share the proxy thread, so a setup that blocks synchronously cannot be + interrupted. A plugin that throws, times out or has the wrong shape is reported and skipped: + its context is closed, every hook it registered is removed, and a setup that resumes after the + deadline cannot register again. A timed-out setup keeps running; resources it already opened + are not closed. The other plugins and the proxy start normally. +- Plugins cannot import ocx modules: in a compiled binary they live inside `$bunfs`. Everything a + plugin may use arrives through `OcxPluginContext` (`name`, `configDir`, `pluginDir`, `log`, + `registerUpstreamRewriter`, `onShutdown`). `onShutdown` registers through + `src/lib/optional-shutdown-hooks.ts` under a per-file, per-registration key, so plugins sharing a + display name, and several teardowns from one plugin, all run. + +## Upstream rewrite slot + +`src/plugins/upstream-hooks.ts` is the only core-owned seam plugins attach to. It imports nothing, +so the request path depends on it without depending on the loader. + +- It runs synchronously after the transport was chosen: HTTP in `sendWithConnectionPolicy` + (`src/server/responses/fetch-helpers.ts`), including `Request` inputs, and the Codex WebSocket in + `codexWsUpstreamFetch` (`src/server/responses/ws-upstream.ts`) once per exchange, before the pool + lookup. `planCodexWsDial` applies the rewrite and settles the proxy; the dialled destination, rewritten + headers and proxy are part of the reuse identity (`codexWsReuseIdentity` in + `src/server/responses/codex-ws-pool.ts`), so a socket is never reused for another destination or + with stale plugin headers. The per-turn headers in `CODEX_WS_FRAME_HEADERS` + (`src/server/responses/codex-ws-request.ts`) ride in each frame's `client_metadata`, prepared + before the rewrite and authoritative, so `planCodexWsDial` restores their original values and + they stay outside the reuse identity. Rewriting any earlier would + hide the ChatGPT origin from the WebSocket selection and push Codex turns onto HTTP. The target + carries the URL, mutable headers and the transport (`http` or `websocket`). +- `sendWithConnectionPolicy` can run twice for one send (an override handing back to the supplied + executor). The outer pass rewrites and marks the init; the inner pass does not rewrite again. +- A rewrite onto loopback dials directly on both transports: HTTP sends carry `proxy: false` and + mark egress as decided; `rewriteWebSocketDial` drops the proxy the caller chose for the original + destination. Other rewrites resolve their route against the rewritten URL on both transports (a + WebSocket route that needs the SSE fallback falls back, as it would for the canonical URL). The pre-dispatch + egress refusal in `providerFetch` still validates the provider's configured route against the + original URL, so a misconfigured provider fails the same way with or without a plugin. +- With no rewriter registered, the send is returned untouched and nothing is allocated. +- A rewriter that throws has its own edits to that send undone and is disabled for the rest of the + process. Rollback is per rewriter: edits from rewriters that ran before it are kept, and the send + continues with them. `onShutdown` keys are unique per registration (`plugin:#`), so a + plugin may register several teardowns. +- Rewrites happen after the request is built, routed and paced, so they do not change routing, + account selection, retry budgets or logging identity. A rewriter that moves a send + to another host owns that host's behaviour; the core does not re-validate it. diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index d7480980b9a..bbad35f297c 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -7,6 +7,8 @@ "openai-chat-serialized-tool-call-scaling.test.ts": "adapters/openai", "openai-chat-tool-call-id-remint.test.ts": "adapters/openai", "coding-agent-json-lines-scaling.test.ts": "providers", + "plugin-loader.test.ts": "lib", + "plugin-upstream-hooks.test.ts": "lib", "usage-snapshot-digest-reuse.test.ts": "usage", "release-desktop-scripts.test.ts": "ci-workflows", "installed-gate-drivers.test.ts": "ci-workflows", diff --git a/tests/lib/plugin-loader.test.ts b/tests/lib/plugin-loader.test.ts new file mode 100644 index 00000000000..5a754a19e3e --- /dev/null +++ b/tests/lib/plugin-loader.test.ts @@ -0,0 +1,235 @@ +import { afterEach, beforeAll, beforeEach, expect, test } from "bun:test"; +import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import { resetOptionalShutdownHooksForTests, runOptionalShutdownHooks } from "../../src/lib/optional-shutdown-hooks"; +import { loadOcxPlugins, pluginFileTrustError } from "../../src/plugins/loader"; +import { + hasUpstreamRewriters, + resetUpstreamRewritersForTests, + rewriteUpstream, +} from "../../src/plugins/upstream-hooks"; + +let dir: string; + +// The loader refuses a plugin directory with a group- or other-writable, non-sticky ancestor. +// The test runner nests per-process temp roots and creates them with the caller's umask, which is +// group-writable on user-private-group systems (umask 002). Those directories belong to this +// test run, so drop group/other write on every ancestor this user owns. The root-owned sticky +// `/tmp` above them is accepted as is, which also exercises the sticky exception. +beforeAll(() => { + if (process.platform === "win32") return; + for (let current = realpathSync(tmpdir()); ; current = dirname(current)) { + try { + const stats = statSync(current); + if (stats.uid === process.getuid?.() && (stats.mode & 0o022) !== 0 && (stats.mode & 0o1000) === 0) { + chmodSync(current, stats.mode & 0o7755); + } + } catch { /* leave directories this run cannot inspect alone */ } + if (dirname(current) === current) break; + } +}); + +beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "ocx-plugins-")); + delete process.env["OCX_PLUGINS"]; +}); + +afterEach(() => { + resetUpstreamRewritersForTests(); + delete process.env["OCX_PLUGINS"]; + rmSync(dir, { recursive: true, force: true }); +}); + +function writePlugin(file: string, source: string, mode = 0o600): string { + const path = join(dir, file); + writeFileSync(path, source); + chmodSync(path, mode); + return path; +} + +const REDIRECT_PLUGIN = ` +export default { + name: "redirect", + setup(ctx) { + ctx.registerUpstreamRewriter(target => { target.url = "http://127.0.0.1:8787" + new URL(target.url).pathname; }); + }, +}; +`; + +test("a missing plugin directory loads nothing", async () => { + expect(await loadOcxPlugins(join(dir, "absent"))).toEqual([]); + expect(hasUpstreamRewriters()).toBe(false); +}); + +test("a valid plugin registers its upstream rewriter", async () => { + writePlugin("redirect.ts", REDIRECT_PLUGIN); + const results = await loadOcxPlugins(dir); + expect(results.map(result => [result.name, result.loaded])).toEqual([["redirect", true]]); + expect(rewriteUpstream("https://api.example.com/v1/responses", undefined, "http").url) + .toBe("http://127.0.0.1:8787/v1/responses"); +}); + +test("OCX_PLUGINS=0 skips loading", async () => { + writePlugin("redirect.ts", REDIRECT_PLUGIN); + process.env["OCX_PLUGINS"] = "0"; + expect(await loadOcxPlugins(dir)).toEqual([]); + expect(hasUpstreamRewriters()).toBe(false); +}); + +test.skipIf(process.platform === "win32")("a group- or world-writable plugin is refused", async () => { + const path = writePlugin("redirect.ts", REDIRECT_PLUGIN, 0o664); + expect(pluginFileTrustError(path)).toContain("writable by group or others"); + const [result] = await loadOcxPlugins(dir); + expect(result?.loaded).toBe(false); + expect(hasUpstreamRewriters()).toBe(false); +}); + +test.skipIf(process.platform === "win32")("a symbolic link is refused even when it points to a trusted file", async () => { + const outside = mkdtempSync(join(tmpdir(), "ocx-plugin-target-")); + try { + const target = join(outside, "real.ts"); + writeFileSync(target, REDIRECT_PLUGIN); + chmodSync(target, 0o600); + symlinkSync(target, join(dir, "linked.ts")); + const [result] = await loadOcxPlugins(dir); + expect(result?.loaded).toBe(false); + expect(result?.error).toBe("refused: is a symbolic link"); + expect(hasUpstreamRewriters()).toBe(false); + } finally { + rmSync(outside, { recursive: true, force: true }); + } +}); + +test.skipIf(process.platform === "win32")("a plugin directory writable by group or others is refused", async () => { + writePlugin("redirect.ts", REDIRECT_PLUGIN); + chmodSync(dir, 0o775); + const results = await loadOcxPlugins(dir); + expect(results).toEqual([{ + file: dir, + name: "plugins directory", + loaded: false, + error: "refused: writable by group or others (chmod go-w)", + }]); + expect(hasUpstreamRewriters()).toBe(false); +}); + +test.skipIf(process.platform === "win32")("a plugin directory under a group-writable, non-sticky parent is refused", async () => { + const parent = join(dir, "shared"); + const nested = join(parent, "plugins"); + mkdirSync(nested, { recursive: true, mode: 0o700 }); + chmodSync(parent, 0o775); + writeFileSync(join(nested, "redirect.ts"), REDIRECT_PLUGIN); + chmodSync(join(nested, "redirect.ts"), 0o600); + const results = await loadOcxPlugins(nested); + expect(results).toHaveLength(1); + expect(results[0]?.loaded).toBe(false); + expect(results[0]?.error).toContain("is writable by group or others"); + expect(hasUpstreamRewriters()).toBe(false); +}); + +test("a wrong export shape or a throwing setup is skipped and leaves no hooks behind", async () => { + writePlugin("a-shape.ts", "export default { name: 'shape' };"); + writePlugin("b-throws.ts", ` +export default { + setup(ctx) { + ctx.registerUpstreamRewriter(target => { target.url = "http://leaked/"; }); + throw new Error("setup failed"); + }, +}; +`); + writePlugin("c-ok.ts", REDIRECT_PLUGIN); + const results = await loadOcxPlugins(dir); + expect(results.map(result => [result.name, result.loaded])).toEqual([ + ["a-shape", false], + ["b-throws", false], + ["redirect", true], + ]); + expect(results[1]?.error).toBe("setup failed"); + expect(rewriteUpstream("https://api.example.com/v1/x", undefined, "http").url).toBe("http://127.0.0.1:8787/v1/x"); +}); + +test("a plugin path that cannot be read is reported, not treated as empty", async () => { + const notADirectory = writePlugin("file-not-dir", "x"); + const results = await loadOcxPlugins(notADirectory); + expect(results).toHaveLength(1); + expect(results[0]?.loaded).toBe(false); + expect(results[0]?.name).toBe("plugins directory"); + expect(results[0]?.error).toContain("ENOTDIR"); +}); + +test("two plugins with the same name keep separate shutdown teardowns", async () => { + const ran: string[] = []; + (globalThis as Record)["__ocxTeardownLog"] = ran; + const source = (tag: string) => ` +export default { + name: "same", + setup(ctx) { ctx.onShutdown(() => { globalThis.__ocxTeardownLog.push("${tag}"); }); }, +}; +`; + writePlugin("a.ts", source("a")); + writePlugin("b.ts", source("b")); + resetOptionalShutdownHooksForTests(); + try { + const results = await loadOcxPlugins(dir); + expect(results.map(result => result.loaded)).toEqual([true, true]); + runOptionalShutdownHooks(); + expect(ran.sort()).toEqual(["a", "b"]); + } finally { + resetOptionalShutdownHooksForTests(); + delete (globalThis as Record)["__ocxTeardownLog"]; + } +}); + +test("one plugin can register several shutdown teardowns", async () => { + const ran: string[] = []; + (globalThis as Record)["__ocxTeardownLog"] = ran; + writePlugin("multi.ts", ` +export default { + setup(ctx) { + ctx.onShutdown(() => { globalThis.__ocxTeardownLog.push("first"); }); + ctx.onShutdown(() => { globalThis.__ocxTeardownLog.push("second"); }); + }, +}; +`); + resetOptionalShutdownHooksForTests(); + try { + expect((await loadOcxPlugins(dir))[0]?.loaded).toBe(true); + runOptionalShutdownHooks(); + expect(ran.sort()).toEqual(["first", "second"]); + } finally { + resetOptionalShutdownHooksForTests(); + delete (globalThis as Record)["__ocxTeardownLog"]; + } +}); + +test("a setup that resumes after its deadline cannot leave registrations behind", async () => { + writePlugin("slow.ts", ` +export default { + name: "slow", + async setup(ctx) { + await new Promise(resolve => setTimeout(resolve, 60)); + ctx.registerUpstreamRewriter(target => { target.url = "http://leaked/"; }); + }, +}; +`); + const originalError = console.error; + console.error = () => {}; + try { + const results = await loadOcxPlugins(dir, { setupTimeoutMs: 20 }); + expect(results[0]?.loaded).toBe(false); + expect(results[0]?.error).toContain("did not finish within 20ms"); + await new Promise(resolve => setTimeout(resolve, 120)); + } finally { + console.error = originalError; + } + expect(hasUpstreamRewriters()).toBe(false); +}); + +test("hidden, underscore-prefixed and declaration files are ignored", async () => { + writePlugin(".hidden.ts", REDIRECT_PLUGIN); + writePlugin("_draft.ts", REDIRECT_PLUGIN); + writePlugin("types.d.ts", "export {};"); + writePlugin("notes.md", "# not a plugin"); + expect(await loadOcxPlugins(dir)).toEqual([]); +}); diff --git a/tests/lib/plugin-upstream-hooks.test.ts b/tests/lib/plugin-upstream-hooks.test.ts new file mode 100644 index 00000000000..4d72b5b292a --- /dev/null +++ b/tests/lib/plugin-upstream-hooks.test.ts @@ -0,0 +1,204 @@ +import { afterEach, expect, test } from "bun:test"; +import { + hasUpstreamRewriters, + registerUpstreamRewriter, + resetUpstreamRewritersForTests, + rewriteUpstream, + rewriteUpstreamRecord, + rewriteWebSocketDial, +} from "../../src/plugins/upstream-hooks"; +import { sendWithConnectionPolicy } from "../../src/server/responses/fetch-helpers"; +import { planCodexWsDial } from "../../src/server/responses/ws-upstream"; +import { codexWsReuseIdentity } from "../../src/server/responses/codex-ws-pool"; +import { CODEX_RESPONSES_HTTP_URL } from "../../src/server/responses/codex-ws-request"; + +afterEach(() => resetUpstreamRewritersForTests()); + +test("with no plugin registered the send is returned untouched and unallocated", () => { + const headers = new Headers({ authorization: "Bearer x" }); + const result = rewriteUpstream("https://api.example.com/v1/responses", headers, "http"); + expect(hasUpstreamRewriters()).toBe(false); + expect(result.url).toBe("https://api.example.com/v1/responses"); + expect(result.headers).toBe(headers); +}); + +test("a rewriter can redirect the URL and add headers while keeping credentials", () => { + registerUpstreamRewriter("sidecar", target => { + const original = new URL(target.url); + target.url = `http://127.0.0.1:8787${original.pathname}`; + target.headers.set("x-sidecar-upstream", original.origin); + }); + const result = rewriteUpstream("https://api.example.com/v1/responses", { authorization: "Bearer x" }, "http"); + const headers = new Headers(result.headers); + expect(result.url).toBe("http://127.0.0.1:8787/v1/responses"); + expect(headers.get("x-sidecar-upstream")).toBe("https://api.example.com"); + expect(headers.get("authorization")).toBe("Bearer x"); +}); + +test("rewriters see the transport and run in registration order", () => { + const seen: string[] = []; + registerUpstreamRewriter("first", target => { seen.push(`first:${target.transport}`); target.url += "?a"; }); + registerUpstreamRewriter("second", target => { seen.push(`second:${target.transport}`); target.url += "&b"; }); + const result = rewriteUpstreamRecord("wss://chatgpt.com/backend-api/codex/responses", { "x-k": "v" }, "websocket"); + expect(seen).toEqual(["first:websocket", "second:websocket"]); + expect(result.url).toBe("wss://chatgpt.com/backend-api/codex/responses?a&b"); + expect(result.headers["x-k"]).toBe("v"); +}); + +test("a throwing rewriter is disabled and never breaks the send", () => { + let calls = 0; + registerUpstreamRewriter("broken", () => { calls += 1; throw new Error("boom"); }); + const originalError = console.error; + console.error = () => {}; + try { + for (let i = 0; i < 3; i += 1) { + expect(rewriteUpstream("https://api.example.com/v1/messages", undefined, "http").url) + .toBe("https://api.example.com/v1/messages"); + } + } finally { + console.error = originalError; + } + expect(calls).toBe(1); +}); + +test("a rewriter that edits the target and then throws leaves the send unmodified", () => { + registerUpstreamRewriter("half", target => { + target.url = "http://127.0.0.1:9/partial"; + target.headers.set("x-partial", "1"); + target.headers.delete("authorization"); + throw new Error("boom"); + }); + let seenByNext: { url: string; partial: string | null; auth: string | null } | undefined; + registerUpstreamRewriter("next", target => { + seenByNext = { url: target.url, partial: target.headers.get("x-partial"), auth: target.headers.get("authorization") }; + }); + const originalError = console.error; + console.error = () => {}; + let result: ReturnType; + try { + result = rewriteUpstream("https://api.example.com/v1/responses", { authorization: "Bearer x" }, "http"); + } finally { + console.error = originalError; + } + const headers = new Headers(result.headers); + expect(seenByNext).toEqual({ url: "https://api.example.com/v1/responses", partial: null, auth: "Bearer x" }); + expect(result.url).toBe("https://api.example.com/v1/responses"); + expect(headers.get("x-partial")).toBeNull(); + expect(headers.get("authorization")).toBe("Bearer x"); +}); + +test("a WebSocket dial redirected to loopback drops the caller's proxy; other dials keep it", () => { + const proxy = "http://corp-proxy.example:3128"; + expect(rewriteWebSocketDial("wss://chatgpt.com/backend-api/codex/responses", {}, proxy).proxy).toBe(proxy); + + const off = registerUpstreamRewriter("loopback", target => { target.url = "ws://127.0.0.1:8787/backend-api/codex/responses"; }); + const local = rewriteWebSocketDial("wss://chatgpt.com/backend-api/codex/responses", { a: "1" }, proxy); + expect(local).toEqual({ url: "ws://127.0.0.1:8787/backend-api/codex/responses", headers: { a: "1" }, proxy: undefined }); + off(); + + registerUpstreamRewriter("remote", target => { target.url = "wss://relay.example.com/backend-api/codex/responses"; }); + expect(rewriteWebSocketDial("wss://chatgpt.com/backend-api/codex/responses", {}, proxy).proxy).toBe(proxy); +}); + +test("the physical HTTP send rewrites once, even through a nested override pass", async () => { + let calls = 0; + registerUpstreamRewriter("count", target => { + calls += 1; + target.url = target.url.replace("https://api.example.com", "http://127.0.0.1:8787"); + target.headers.set("x-hop", String(calls)); + }); + const seen: Array<{ url: string; hop: string | null }> = []; + const physical = (async (input: Parameters[0], init?: RequestInit) => { + seen.push({ url: String(input), hop: new Headers(init?.headers).get("x-hop") }); + return new Response("ok"); + }) as typeof fetch; + // An override that hands the send back to the supplied executor passes through twice. + const inner = ((input: Parameters[0], init?: RequestInit) => + sendWithConnectionPolicy(physical, input, init)) as typeof fetch; + await sendWithConnectionPolicy(inner, "https://api.example.com/v1/responses", { method: "POST" }); + expect(calls).toBe(1); + expect(seen).toEqual([{ url: "http://127.0.0.1:8787/v1/responses", hop: "1" }]); +}); + +test("an HTTP send redirected to loopback dials directly, bypassing any proxy", async () => { + const seen: Array<{ url: string; proxy: unknown }> = []; + const physical = (async (input: Parameters[0], init?: RequestInit) => { + seen.push({ url: input instanceof Request ? input.url : String(input), proxy: (init as { proxy?: unknown }).proxy }); + return new Response("ok"); + }) as typeof fetch; + registerUpstreamRewriter("loopback", target => { target.url = target.url.replace("https://api.example.com", "http://127.0.0.1:8787"); }); + await sendWithConnectionPolicy(physical, "https://api.example.com/v1/responses", { method: "POST" }); + resetUpstreamRewritersForTests(); + registerUpstreamRewriter("remote", target => { target.url = target.url.replace("https://api.example.com", "https://relay.example.net"); }); + await sendWithConnectionPolicy(physical, "https://api.example.com/v1/responses", { method: "POST" }); + expect(seen).toEqual([ + { url: "http://127.0.0.1:8787/v1/responses", proxy: false }, + { url: "https://relay.example.net/v1/responses", proxy: undefined }, + ]); +}); + +test("a Request input is rewritten too", async () => { + let seenUrl = ""; + const physical = (async (input: Parameters[0]) => { + seenUrl = input instanceof Request ? input.url : String(input); + return new Response("ok"); + }) as typeof fetch; + registerUpstreamRewriter("loopback", target => { target.url = "http://127.0.0.1:8787/v1/messages"; }); + await sendWithConnectionPolicy(physical, new Request("https://api.example.com/v1/messages", { method: "POST", body: "{}" })); + expect(seenUrl).toBe("http://127.0.0.1:8787/v1/messages"); +}); + +test("the Codex WebSocket reuse identity changes with the dialled destination", () => { + const headers = { authorization: "Bearer t", "chatgpt-account-id": "acct", "thread-id": "th" }; + const frame = JSON.stringify({ model: "gpt-x", client_metadata: { thread_id: "th", turn_id: "tu" } }); + const direct = codexWsReuseIdentity(CODEX_RESPONSES_HTTP_URL, headers, frame, undefined, "wss://chatgpt.com/backend-api/codex/responses"); + const local = codexWsReuseIdentity(CODEX_RESPONSES_HTTP_URL, headers, frame, undefined, "ws://127.0.0.1:8787/backend-api/codex/responses"); + expect(direct).not.toBeNull(); + expect(local).not.toBeNull(); + expect(local?.key).not.toBe(direct?.key); +}); + +test("a Codex WebSocket dial resolves its proxy for the rewritten destination", () => { + const ws = "wss://chatgpt.com/backend-api/codex/responses"; + const envProxy = { HTTPS_PROXY: "http://corp:3128", HTTP_PROXY: "http://plain:8080" }; + expect(planCodexWsDial(ws, {}, "http://corp:3128", envProxy)?.proxy).toBe("http://corp:3128"); + + const off = registerUpstreamRewriter("remote-wss", target => { target.url = "wss://relay.example.net/backend-api/codex/responses"; }); + expect(planCodexWsDial(ws, {}, "http://corp:3128", envProxy)?.proxy).toBe("http://corp:3128"); + expect(planCodexWsDial(ws, {}, "http://corp:3128", { ...envProxy, NO_PROXY: "relay.example.net" })?.proxy).toBeUndefined(); + off(); + + const offPlain = registerUpstreamRewriter("remote-ws", target => { target.url = "ws://relay.example.net/backend-api/codex/responses"; }); + expect(planCodexWsDial(ws, {}, "http://corp:3128", envProxy)?.proxy).toBe("http://plain:8080"); + offPlain(); + + registerUpstreamRewriter("loopback", target => { target.url = "ws://127.0.0.1:8787/backend-api/codex/responses"; }); + expect(planCodexWsDial(ws, {}, "http://corp:3128", envProxy)).toEqual({ + url: "ws://127.0.0.1:8787/backend-api/codex/responses", headers: {}, proxy: undefined, + }); +}); + +test("a Codex WebSocket rewriter cannot change the per-turn headers carried in the frame", () => { + registerUpstreamRewriter("turn-headers", target => { + target.headers.set("x-codex-turn-state", "rewritten"); + target.headers.set("x-codex-turn-metadata", "added"); + target.headers.set("x-sidecar", "1"); + }); + const dial = planCodexWsDial( + "wss://chatgpt.com/backend-api/codex/responses", + { "x-codex-turn-state": "original", authorization: "Bearer t" }, + undefined, + {}, + ); + expect(dial?.headers["x-codex-turn-state"]).toBe("original"); + expect(Object.hasOwn(dial?.headers ?? {}, "x-codex-turn-metadata")).toBe(false); + expect(dial?.headers["x-sidecar"]).toBe("1"); + expect(dial?.headers.authorization).toBe("Bearer t"); +}); + +test("unregistering removes the rewriter", () => { + const off = registerUpstreamRewriter("temp", target => { target.url = "http://changed/"; }); + off(); + expect(hasUpstreamRewriters()).toBe(false); + expect(rewriteUpstream("https://a.example/x", undefined, "http").url).toBe("https://a.example/x"); +}); diff --git a/tests/responses/responses-fetch-helpers-boundary.test.ts b/tests/responses/responses-fetch-helpers-boundary.test.ts index e2f63efeacd..455c3d0d671 100644 --- a/tests/responses/responses-fetch-helpers-boundary.test.ts +++ b/tests/responses/responses-fetch-helpers-boundary.test.ts @@ -49,6 +49,8 @@ describe("Responses fetch-helper import boundary", () => { "../../lib/proxy-env", "../../lib/redact", "../../lib/upstream-http-version", + // Import-free plugin rewrite slot (src/plugins/upstream-hooks.ts). + "../../plugins/upstream-hooks", "../../providers/request-pacing", "./ws-upstream", ]); From 698b1e0d55348b188b0f8a62316ddcb7bc64b2e0 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 02:49:44 +0900 Subject: [PATCH 05/19] fix(plugins): disable Windows auto-load and bound failure logs Refuse plugin loading on Windows until ACL trust is available. Replace automatic loader and rewriter exception output with fixed categories, with regressions for Windows execution and secret-bearing errors. --- .../src/content/docs/guides/local-plugins.md | 5 +- src/plugins/loader.ts | 55 ++++++++++++------- src/plugins/upstream-hooks.ts | 5 +- structure/ops/plugins.md | 8 ++- tests/lib/plugin-loader.test.ts | 34 ++++++++++-- tests/lib/plugin-upstream-hooks.test.ts | 15 +++++ 6 files changed, 87 insertions(+), 35 deletions(-) diff --git a/docs-site/src/content/docs/guides/local-plugins.md b/docs-site/src/content/docs/guides/local-plugins.md index a7864b7d4c1..0f735c93e73 100644 --- a/docs-site/src/content/docs/guides/local-plugins.md +++ b/docs-site/src/content/docs/guides/local-plugins.md @@ -30,12 +30,11 @@ Put plugin files in `plugins/` inside the opencodex home (`~/.opencodex/plugins/ not writable by group or others, unless it is sticky like `/tmp`. Fix permissions with `chmod go-w ~/.opencodex/plugins ~/.opencodex/plugins/*`; on systems whose default umask is `002`, check the parent directories too. -- On Windows these owner and permission checks are not performed; only regular files are loaded. - Keep the `plugins/` directory writable by your account only. +- On Windows automatic plugin loading is disabled until an ACL trust check is available. Restart the proxy after adding, changing or removing a plugin (`ocx service restart`, or stop and start `ocx start`). Each loaded plugin prints a `Plugin loaded: ` line at startup; a skipped -plugin prints the reason. +plugin prints a bounded reason category. Raw plugin exception text is never printed automatically. To start once without plugins, set `OCX_PLUGINS=0`. diff --git a/src/plugins/loader.ts b/src/plugins/loader.ts index aaa4e27a4cc..da632437111 100644 --- a/src/plugins/loader.ts +++ b/src/plugins/loader.ts @@ -8,6 +8,7 @@ * A plugin is a module whose default export is `{ name?, setup(ctx) }`. It runs in the proxy * process with the operator's credentials, so the loader accepts only files owned by the * current user that no other user can write — the same trust boundary as `config.json`. + * Windows auto-loading stays disabled until this trust check is backed by an ACL check. * Plugins cannot import ocx internals (a compiled binary keeps them inside `$bunfs`); they * receive everything they may use through `OcxPluginContext`. * @@ -49,7 +50,9 @@ export interface PluginLoadResult { file: string; name: string; loaded: boolean; - error?: string; + error?: "windows_auto_load_disabled" | "directory_read_failed" | "directory_resolution_failed" + | "directory_untrusted" | "ancestor_untrusted" | "file_untrusted" + | "import_failed" | "invalid_export" | "setup_failed" | "setup_timeout"; } const PLUGIN_EXTENSIONS = [".ts", ".js", ".mjs"]; @@ -79,14 +82,14 @@ function listPluginFiles(dir: string): string[] { * Null when `path` is safe to trust, otherwise the reason it is refused. `lstat` is used so a * symbolic link is judged as a link — and refused — rather than as the file it points to: a * link to a file you own would otherwise pass the owner and mode checks. Windows has no - * POSIX owner or mode bits, so there only the file type is checked. + * POSIX owner or mode bits; the loader refuses execution on that platform. */ function trustError(path: string, kind: "file" | "directory"): string | null { let stats: ReturnType; try { stats = lstatSync(path); - } catch (error) { - return error instanceof Error ? error.message : String(error); + } catch { + return "filesystem inspection failed"; } if (stats.isSymbolicLink()) return "is a symbolic link"; if (kind === "file" ? !stats.isFile() : !stats.isDirectory()) return `not a regular ${kind}`; @@ -125,8 +128,8 @@ export function pluginAncestorsTrustError(realDir: string): string | null { let stats: ReturnType; try { stats = lstatSync(current); - } catch (error) { - return `${current}: ${error instanceof Error ? error.message : String(error)}`; + } catch { + return "ancestor inspection failed"; } if (uid !== undefined && stats.uid !== uid && stats.uid !== 0) return `${current} is owned by another user`; if ((stats.mode & 0o022) !== 0 && (stats.mode & 0o1000) === 0) { @@ -142,10 +145,10 @@ function isPlugin(value: unknown): value is OcxPlugin { return typeof value === "object" && value !== null && typeof (value as OcxPlugin).setup === "function"; } -async function withTimeout(work: Promise, ms: number, label: string): Promise { +async function withTimeout(work: Promise, ms: number): Promise { let timer: ReturnType | undefined; const timeout = new Promise((_, reject) => { - timer = setTimeout(() => reject(new Error(`${label} did not finish within ${ms}ms`)), ms); + timer = setTimeout(() => reject(new PluginSetupTimeoutError()), ms); }); try { return await Promise.race([work, timeout]); @@ -154,6 +157,8 @@ async function withTimeout(work: Promise, ms: number, label: string): Prom } } +class PluginSetupTimeoutError extends Error {} + export interface LoadOcxPluginsOptions { /** Deadline for a setup that yields; see the module comment. */ setupTimeoutMs?: number; @@ -164,30 +169,33 @@ export async function loadOcxPlugins( options: LoadOcxPluginsOptions = {}, ): Promise { if (process.env["OCX_PLUGINS"] === "0") return []; + if (process.platform === "win32") { + return [{ file: dir, name: "plugins directory", loaded: false, error: "windows_auto_load_disabled" }]; + } let files: string[]; try { files = listPluginFiles(dir); - } catch (error) { - return [{ file: dir, name: "plugins directory", loaded: false, error: error instanceof Error ? error.message : String(error) }]; + } catch { + return [{ file: dir, name: "plugins directory", loaded: false, error: "directory_read_failed" }]; } if (files.length === 0) return []; const dirRefused = pluginDirectoryTrustError(dir); - if (dirRefused) return [{ file: dir, name: "plugins directory", loaded: false, error: `refused: ${dirRefused}` }]; + if (dirRefused) return [{ file: dir, name: "plugins directory", loaded: false, error: "directory_untrusted" }]; // Check and import through the resolved path, so both refer to the same components. let realDir: string; try { realDir = realpathSync(dir); - } catch (error) { - return [{ file: dir, name: "plugins directory", loaded: false, error: error instanceof Error ? error.message : String(error) }]; + } catch { + return [{ file: dir, name: "plugins directory", loaded: false, error: "directory_resolution_failed" }]; } const ancestorRefused = pluginAncestorsTrustError(realDir); - if (ancestorRefused) return [{ file: dir, name: "plugins directory", loaded: false, error: `refused: ${ancestorRefused}` }]; + if (ancestorRefused) return [{ file: dir, name: "plugins directory", loaded: false, error: "ancestor_untrusted" }]; const results: PluginLoadResult[] = []; for (const file of files.map(listed => join(realDir, basename(listed)))) { const fallbackName = basename(file).replace(/\.(ts|js|mjs)$/, ""); const refused = pluginFileTrustError(file); if (refused) { - results.push({ file, name: fallbackName, loaded: false, error: `refused: ${refused}` }); + results.push({ file, name: fallbackName, loaded: false, error: "file_untrusted" }); continue; } const unregister: Array<() => void> = []; @@ -196,14 +204,16 @@ export async function loadOcxPlugins( let shutdownCount = 0; const whileActive = (name: string, register: () => () => void): void => { if (!active) { - console.error(`[plugin:${name}] registration after a failed setup was ignored`); + console.error("[opencodex] plugin registration ignored: setup_failed"); return; } unregister.push(register()); }; + let phase: "import" | "export" | "setup" = "import"; try { const module = await import(pathToFileURL(file).href) as { default?: unknown; plugin?: unknown }; const plugin = module.default ?? module.plugin; + phase = "export"; if (!isPlugin(plugin)) throw new Error("default export must be { name?, setup(context) }"); const name = typeof plugin.name === "string" && plugin.name.trim() ? plugin.name.trim() : fallbackName; const context: OcxPluginContext = { @@ -217,7 +227,8 @@ export async function loadOcxPlugins( onShutdown: teardown => whileActive(name, () => registerOptionalShutdownHook(`plugin:${file}#${++shutdownCount}`, teardown)), }; const timeoutMs = options.setupTimeoutMs ?? SETUP_TIMEOUT_MS; - await withTimeout(Promise.resolve(plugin.setup(context)), timeoutMs, `plugin "${name}" setup`); + phase = "setup"; + await withTimeout(Promise.resolve(plugin.setup(context)), timeoutMs); results.push({ file, name, loaded: true }); } catch (error) { // A half-initialised plugin must not leave hooks behind, now or later. @@ -227,7 +238,9 @@ export async function loadOcxPlugins( file, name: fallbackName, loaded: false, - error: error instanceof Error ? error.message : String(error), + error: error instanceof PluginSetupTimeoutError ? "setup_timeout" + : phase === "import" ? "import_failed" + : phase === "export" ? "invalid_export" : "setup_failed", }); } } @@ -239,9 +252,9 @@ export async function loadAndReportOcxPlugins(): Promise { try { for (const result of await loadOcxPlugins()) { if (result.loaded) console.log(`🔌 Plugin loaded: ${result.name}`); - else console.error(`⚠️ Plugin ${result.name} skipped: ${result.error}`); + else console.error(`⚠️ Plugin skipped: ${result.error}`); } - } catch (error) { - console.error(`⚠️ Plugin loading failed: ${error instanceof Error ? error.message : String(error)}`); + } catch { + console.error("⚠️ Plugin loading failed: internal_error"); } } diff --git a/src/plugins/upstream-hooks.ts b/src/plugins/upstream-hooks.ts index 77436d1b015..13c64f8896d 100644 --- a/src/plugins/upstream-hooks.ts +++ b/src/plugins/upstream-hooks.ts @@ -64,12 +64,11 @@ export function rewriteUpstream( const headersBefore = new Headers(target.headers); try { registration.rewrite(target); - } catch (error) { + } catch { target.url = urlBefore; target.headers = headersBefore; registration.disabled = true; - const reason = error instanceof Error ? error.message : String(error); - console.error(`[opencodex] plugin "${registration.name}" upstream rewriter disabled after an error: ${reason}`); + console.error("[opencodex] plugin upstream rewriter disabled: plugin_exception"); } } return { url: target.url, headers: target.headers }; diff --git a/structure/ops/plugins.md b/structure/ops/plugins.md index 6912a501fc4..7629ed38cf5 100644 --- a/structure/ops/plugins.md +++ b/structure/ops/plugins.md @@ -18,7 +18,8 @@ or signs them. `config.json`. Every ancestor of the resolved plugin directory up to `/` must be owned by the user or root and not group/other-writable unless sticky (`pluginAncestorsTrustError`), so no other user can swap a checked path before it is imported; files are imported through the resolved directory. - Owner and mode checks are POSIX-only; on Windows only the file type is checked. + Owner and mode checks are POSIX-only; Windows auto-loading is disabled until an ACL trust + check can enforce the same boundary. - A missing plugin directory means no plugins. Any other read failure (`EACCES`, `ENOTDIR`) is reported as a skipped `plugins directory` entry. - A plugin module default-exports `{ name?, setup(context) }`. An asynchronous `setup` has five @@ -27,6 +28,8 @@ or signs them. its context is closed, every hook it registered is removed, and a setup that resumes after the deadline cannot register again. A timed-out setup keeps running; resources it already opened are not closed. The other plugins and the proxy start normally. +- Automatic failure logs use bounded categories such as `setup_failed`, `setup_timeout` and + `file_untrusted`. They omit raw exception text, which can contain credentials or outbound headers. - Plugins cannot import ocx modules: in a compiled binary they live inside `$bunfs`. Everything a plugin may use arrives through `OcxPluginContext` (`name`, `configDir`, `pluginDir`, `log`, `registerUpstreamRewriter`, `onShutdown`). `onShutdown` registers through @@ -60,7 +63,8 @@ so the request path depends on it without depending on the loader. original URL, so a misconfigured provider fails the same way with or without a plugin. - With no rewriter registered, the send is returned untouched and nothing is allocated. - A rewriter that throws has its own edits to that send undone and is disabled for the rest of the - process. Rollback is per rewriter: edits from rewriters that ran before it are kept, and the send + process. Its automatic log reports `plugin_exception` without the exception or plugin name. + Rollback is per rewriter: edits from rewriters that ran before it are kept, and the send continues with them. `onShutdown` keys are unique per registration (`plugin:#`), so a plugin may register several teardowns. - Rewrites happen after the request is built, routed and paced, so they do not change routing, diff --git a/tests/lib/plugin-loader.test.ts b/tests/lib/plugin-loader.test.ts index 5a754a19e3e..7d2e02b4d0b 100644 --- a/tests/lib/plugin-loader.test.ts +++ b/tests/lib/plugin-loader.test.ts @@ -77,6 +77,28 @@ test("OCX_PLUGINS=0 skips loading", async () => { expect(hasUpstreamRewriters()).toBe(false); }); +test("Windows does not auto-load plugins without an ACL trust check", async () => { + writePlugin("redirect.ts", REDIRECT_PLUGIN); + const platform = process.platform; + try { + Object.defineProperty(process, "platform", { value: "win32", configurable: true }); + const results = await loadOcxPlugins(dir); + expect(results).toEqual([{ file: dir, name: "plugins directory", loaded: false, error: "windows_auto_load_disabled" }]); + expect(hasUpstreamRewriters()).toBe(false); + } finally { + Object.defineProperty(process, "platform", { value: platform, configurable: true }); + } +}); + +test("plugin setup exceptions expose only a bounded category", async () => { + const marker = "private plugin error marker"; + writePlugin("throws.ts", `export default { setup() { throw new Error("${marker}"); } };`); + const results = await loadOcxPlugins(dir); + expect(results[0]?.loaded).toBe(false); + expect(results[0]?.error).toBe("setup_failed"); + expect(JSON.stringify(results)).not.toContain(marker); +}); + test.skipIf(process.platform === "win32")("a group- or world-writable plugin is refused", async () => { const path = writePlugin("redirect.ts", REDIRECT_PLUGIN, 0o664); expect(pluginFileTrustError(path)).toContain("writable by group or others"); @@ -94,7 +116,7 @@ test.skipIf(process.platform === "win32")("a symbolic link is refused even when symlinkSync(target, join(dir, "linked.ts")); const [result] = await loadOcxPlugins(dir); expect(result?.loaded).toBe(false); - expect(result?.error).toBe("refused: is a symbolic link"); + expect(result?.error).toBe("file_untrusted"); expect(hasUpstreamRewriters()).toBe(false); } finally { rmSync(outside, { recursive: true, force: true }); @@ -109,7 +131,7 @@ test.skipIf(process.platform === "win32")("a plugin directory writable by group file: dir, name: "plugins directory", loaded: false, - error: "refused: writable by group or others (chmod go-w)", + error: "directory_untrusted", }]); expect(hasUpstreamRewriters()).toBe(false); }); @@ -124,7 +146,7 @@ test.skipIf(process.platform === "win32")("a plugin directory under a group-writ const results = await loadOcxPlugins(nested); expect(results).toHaveLength(1); expect(results[0]?.loaded).toBe(false); - expect(results[0]?.error).toContain("is writable by group or others"); + expect(results[0]?.error).toBe("ancestor_untrusted"); expect(hasUpstreamRewriters()).toBe(false); }); @@ -145,7 +167,7 @@ export default { ["b-throws", false], ["redirect", true], ]); - expect(results[1]?.error).toBe("setup failed"); + expect(results[1]?.error).toBe("setup_failed"); expect(rewriteUpstream("https://api.example.com/v1/x", undefined, "http").url).toBe("http://127.0.0.1:8787/v1/x"); }); @@ -155,7 +177,7 @@ test("a plugin path that cannot be read is reported, not treated as empty", asyn expect(results).toHaveLength(1); expect(results[0]?.loaded).toBe(false); expect(results[0]?.name).toBe("plugins directory"); - expect(results[0]?.error).toContain("ENOTDIR"); + expect(results[0]?.error).toBe("directory_read_failed"); }); test("two plugins with the same name keep separate shutdown teardowns", async () => { @@ -218,7 +240,7 @@ export default { try { const results = await loadOcxPlugins(dir, { setupTimeoutMs: 20 }); expect(results[0]?.loaded).toBe(false); - expect(results[0]?.error).toContain("did not finish within 20ms"); + expect(results[0]?.error).toBe("setup_timeout"); await new Promise(resolve => setTimeout(resolve, 120)); } finally { console.error = originalError; diff --git a/tests/lib/plugin-upstream-hooks.test.ts b/tests/lib/plugin-upstream-hooks.test.ts index 4d72b5b292a..124aa1f08c1 100644 --- a/tests/lib/plugin-upstream-hooks.test.ts +++ b/tests/lib/plugin-upstream-hooks.test.ts @@ -61,6 +61,21 @@ test("a throwing rewriter is disabled and never breaks the send", () => { expect(calls).toBe(1); }); +test("a throwing rewriter logs only a bounded category", () => { + const marker = "private upstream error marker"; + registerUpstreamRewriter(marker, () => { throw new Error(marker); }); + const originalError = console.error; + const lines: string[] = []; + console.error = (...args) => { lines.push(args.map(String).join(" ")); }; + try { + rewriteUpstream("https://api.example.com/v1/messages", { authorization: `Bearer ${marker}` }, "http"); + } finally { + console.error = originalError; + } + expect(lines).toEqual(["[opencodex] plugin upstream rewriter disabled: plugin_exception"]); + expect(lines.join(" ")).not.toContain(marker); +}); + test("a rewriter that edits the target and then throws leaves the send unmodified", () => { registerUpstreamRewriter("half", target => { target.url = "http://127.0.0.1:9/partial"; From 2889ef44d2086ce8e3011bced2914728a21acf2e Mon Sep 17 00:00:00 2001 From: mdwsk88 <924038395@qq.com> Date: Sun, 27 Sep 2026 02:50:21 +0900 Subject: [PATCH 06/19] feat(codebuddy): discover the account roster through the CLI help surface (#5147) Carried from #5147 as one squashed commit. Co-authored-by: mdwsk88 <924038395@qq.com> --- .../src/content/docs/guides/providers.md | 3 +- scripts/test-layout/layout.json | 1 + src/adapters/codebuddy/live-models.ts | 191 ++++++++++++ src/codex/catalog/model-hints.ts | 11 + src/codex/catalog/provider-models.ts | 50 ++++ src/providers/codebuddy-models.ts | 8 +- src/providers/registry/entries-extended.ts | 8 +- tests/fixtures/test-layout-expected.json | 1 + tests/providers/codebuddy-live-models.test.ts | 277 ++++++++++++++++++ 9 files changed, 544 insertions(+), 6 deletions(-) create mode 100644 src/adapters/codebuddy/live-models.ts create mode 100644 tests/providers/codebuddy-live-models.test.ts diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 54372cce2f9..6b931db131b 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -889,10 +889,11 @@ OpenCodex provides official adapter support for Tencent Cloud's CodeBuddy Code C - Global: [CodeBuddy Global API Keys](https://www.codebuddy.ai/profile/keys) - CN: [CodeBuddy CN API Keys](https://copilot.tencent.com/profile/keys) - **Region Isolation:** `codebuddy` and `codebuddy-cn` use separate canonical endpoints (`https://www.codebuddy.ai` and `https://www.codebuddy.cn`) and isolated child environments (`CODEBUDDY_INTERNET_ENVIRONMENT=public` vs `internal`). Credentials are strictly region-scoped and never exchanged across environments. Overriding the canonical base URL fails closed. +- **Model Discovery:** the proxy requests the CodeBuddy product configuration (`GET {baseUrl}/v3/config`) with the configured key as the `X-API-Key` header, and the roster in that answer is the authoritative roster of discovered models: it is the key's own account configuration, so it is proven to belong to the key — a different or wrong key answers the anonymous envelope with no roster instead of another account's models. The authenticated roster is the same list the CLI prints for `--model` (the "Currently supported" line of a signed-in CLI), can differ from the static manifest bundled with the CLI, and the vendor default selectors (`default` for CN, `default-model` for Global) never appear in it but remain callable: the catalog retains them during live discovery and on every fallback path. On start/sync the proxy binds the cached roster to an irreversible fingerprint of the configured key, so a key switch never observes a roster cached for the previous key, and degrades to the stale provider/key-fingerprint-scoped cache, then to the static seed in `src/providers/codebuddy-models.ts`, when the key does not authenticate or the request fails. - **Tool Ownership and the Tool Bridge:** The CLI is always spawned with `--tools ""` and `--strict-mcp-config`, so it has no built-in or user-configured tools of its own. When a request carries a Codex tool catalog, the provider arms a capture-only MCP bridge: the validated catalog and MCP config are written to a private temp dir, the CLI is launched with `--mcp-config` and an exact `--allowedTools` list, and the `system/init` frame must report exactly that bridge server as connected or the turn fails closed. The bridge advertises the Codex tools and captures proposed calls but never executes anything: a completed tool-call batch is returned as `function_call` items (names mapped back to the request's wire names, at most 16 calls per assistant message), the process tree is terminated at `message_stop`, and the external Codex client alone performs approval, sandboxing, and execution. Tool results come back as the next request's input, and the conversation continues. Requests without tools keep the plain text-and-reasoning shape. If the CLI writes an unquoted DSML `calls` control line followed by a `functions.*` invoke control line into text or reasoning, OpenCodex refuses the turn instead of forwarding the scaffold or interpreting it as an executable call. DSML discussed or quoted in prose, inline code, fenced code, or source examples remains ordinary answer text. -- **Entitlements and Billing:** The provider uses the same vendor-documented CodeBuddy account/CLI authentication surface. Availability and billing of free, promotional, trial, or subscription credits remain determined by the user's CodeBuddy account entitlement. - **Tool Choice Enforcement:** When a request specifies `tool_choice: "required"` or selects a specific named tool, the bridge expects a tool call from the model. If the CLI completes the turn with plain text instead of capturing a tool call, OpenCodex fails closed with a 502 `tool_call_required` error rather than returning an invalid text completion. - **Governance Status:** Whether routing this vendor automation surface behind a proxy for a third-party agent satisfies CodeBuddy's acceptable-use terms is an open question flagged for maintainer security review (see the governance note in the provider registry entry). Treat this provider as pending that review, and keep the tool bridge's ownership boundary in mind: the nested CLI advertises tools but never executes them, and approval, sandboxing, and execution remain with the external Codex client. +- **Entitlements and Billing:** The provider uses the same vendor-documented CodeBuddy account/CLI authentication surface. Availability and billing of free, promotional, trial, or subscription credits remain determined by the user's CodeBuddy account entitlement. ### Official Qoder CLI (Global & CN) diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 234f1d261d0..8c2b820250b 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -579,6 +579,7 @@ "codebuddy-adapter.test.ts": "providers", "codebuddy-live-acceptance.test.ts": "providers", "codebuddy-mcp-server.test.ts": "providers", + "codebuddy-live-models.test.ts": "providers", "codebuddy-protocol.test.ts": "providers", "codebuddy-tool-bridge-turn.test.ts": "providers", "codebuddy-tool-bridge.test.ts": "providers", diff --git a/src/adapters/codebuddy/live-models.ts b/src/adapters/codebuddy/live-models.ts new file mode 100644 index 00000000000..e646f3694eb --- /dev/null +++ b/src/adapters/codebuddy/live-models.ts @@ -0,0 +1,191 @@ +import { isValidModelDiscoveryModelId } from "../../providers/model-discovery-limits"; +import type { CodeBuddyProfile } from "./profiles"; + +const MAX_CONFIG_BYTES = 512 * 1024; +const MAX_MODELS = 128; +const MAX_ERROR_BODY_BYTES = 4 * 1024; + +export type CodeBuddyModelsResult = + | { ok: true; models: string[] } + | { ok: false; error: "http" | "timeout" | "invalid_output" | "empty" | "too_large"; detail?: string }; + +export interface CodeBuddyConfigFetchDeps { + /** Test seam for the outbound request; defaults to global fetch. */ + fetch?: typeof fetch; + timeoutMs?: number; +} + +type CodeBuddyModelsFetcher = (profile: CodeBuddyProfile, apiKey: string) => CodeBuddyModelsResult | Promise; +let codeBuddyModelsFetcherForTests: CodeBuddyModelsFetcher | null = null; + +export function setFetchCodeBuddyModelsForTests(next: CodeBuddyModelsFetcher | null): void { + codeBuddyModelsFetcherForTests = next; +} + +function isPlainObject(value: unknown): value is Record { + return typeof value === "object" && value !== null && !Array.isArray(value); +} + +// The product gateway authenticates the X-API-Key and then requires a User-Agent it can parse a +// client version from: a bare default fetch/axios UA answers 400 {"code":12403,"msg":"check ua, +// get coding copilot version error"} (measured 260923 on www.codebuddy.cn and +// copilot.tencent.com). The CLI's own UA shape is `CLI/ CodeBuddy/`; the +// version VALUE is not validated (CLI/0.0.1 measures fine), so a fixed recent shape is stable +// until the vendor tightens it — and a rejection then degrades through the same failure path as +// any other discovery failure. +const CLI_USER_AGENT = "CLI/2.126.0 CodeBuddy/2.126.0"; + +/** + * Read at most `cap` bytes of an untrusted upstream body, then stop reading. A body that + * would cross the cap is cancelled at the reader the moment the crossing chunk arrives, so a + * malformed or compromised upstream cannot make discovery buffer an unbounded response (the + * same contract as the vision sidecar's bounded error-body read). + */ +async function readBoundedBodyText(res: Response, cap: number): Promise< + | { ok: true; text: string } + | { ok: false; reason: "exceeded" | "read"; detail?: string } +> { + if (!res.body) return { ok: true, text: "" }; + const reader = res.body.getReader(); + const decoder = new TextDecoder(); + let out = ""; + let seen = 0; + try { + for (;;) { + const { done, value } = await reader.read(); + if (done) break; + if (seen + value.byteLength > cap) { + try { void reader.cancel("CodeBuddy config body byte limit reached").catch(() => undefined); } + catch { /* best-effort body teardown */ } + return { ok: false, reason: "exceeded" }; + } + seen += value.byteLength; + out += decoder.decode(value, { stream: true }); + } + out += decoder.decode(); + return { ok: true, text: out }; + } catch (error) { + return { ok: false, reason: "read", detail: String((error as Error)?.message ?? error).slice(0, 200) }; + } +} + +/** A declared Content-Length above the cap is refused before a single byte is read. */ +function declaredLengthExceeds(response: Response, cap: number): boolean { + const declared = Number(response.headers.get("content-length")); + return Number.isSafeInteger(declared) && declared > cap; +} + +/** + * Parse the key-scoped roster out of the product configuration envelope. + * + * `GET {canonicalBaseUrl}/v3/config` with the configured key answers the KEY's own account + * configuration: `data.agents[].models` is exactly the roster the CLI's `--help` prints for a + * signed-in account of that key (measured 260923: 17 ids, byte-identical), and `data.models` + * carries per-model metadata for the wider account catalog. An absent or invalid key answers + * the anonymous envelope instead — no `agents` array and an empty `models` list — so the roster + * is proven to belong to the key by construction: it only exists when the key authenticated. + * `custom:*` selectors are per-user CLI configuration pointing at operator-defined upstreams, + * not shared catalog rows, and are excluded. + */ +export function parseCodeBuddyConfigRoster(body: unknown): CodeBuddyModelsResult { + if (!isPlainObject(body)) return { ok: false, error: "invalid_output", detail: "CodeBuddy config response is not an object" }; + const data = body.data; + if (!isPlainObject(data)) return { ok: false, error: "invalid_output", detail: "CodeBuddy config envelope is missing its data object" }; + const agents = data.agents; + if (!Array.isArray(agents)) { + // The authenticated envelope always carries an agents array; the anonymous one (absent or + // invalid key) does not. Both fail closed here, but the distinction names the cause. + return { ok: false, error: "empty", detail: "CodeBuddy answered the anonymous config: the key did not authenticate" }; + } + // The catalog mirrors what the CLI itself accepts for --model: the default agent's models. + // agents has carried exactly one entry named "cli" so far; prefer it by name and fall back to + // the first agent that declares a models array, so a future second agent cannot silently + // widen the roster beyond what the chat path can actually run. + const agent = agents.find(entry => isPlainObject(entry) && entry.name === "cli" && Array.isArray(entry.models)) + ?? agents.find(entry => isPlainObject(entry) && Array.isArray(entry.models)); + const declared = isPlainObject(agent) && Array.isArray(agent.models) ? agent.models : []; + const models: string[] = []; + const seen = new Set(); + for (const raw of declared) { + const id = typeof raw === "string" ? raw : isPlainObject(raw) && typeof raw.id === "string" ? raw.id : undefined; + if (!id || id.startsWith("custom:") || seen.has(id) || !isValidModelDiscoveryModelId(id)) continue; + seen.add(id); + models.push(id); + if (models.length >= MAX_MODELS) break; + } + return models.length > 0 ? { ok: true, models } : { ok: false, error: "empty", detail: "CodeBuddy config roster is empty" }; +} + +/** + * Discover the roster that belongs to this exact key from the product configuration endpoint. + * + * The previous design parsed `codebuddy --help`, whose roster reflects the CLI's signed-in + * account under the caller's home — a key of a different account (or a wrong key) still + * observed the signed-in account's roster, so caching it under the key's fingerprint could + * advertise another account's models for that key (review on #5147). The configuration request + * authenticates with the key itself, so the roster it returns is the key's own: the CLI binary, + * its login state, and the caller's home are all irrelevant to the answer. Measured 260923 + * against www.codebuddy.cn (CN): a valid key answers `data.agents[0].models` with the same 17 + * ids the CLI prints when signed in to that account; an invalid or absent key answers the + * anonymous envelope with no agents and no models. + */ +export async function fetchCodeBuddyModels( + profile: CodeBuddyProfile, + apiKey: string, + deps: CodeBuddyConfigFetchDeps = {}, +): Promise { + if (codeBuddyModelsFetcherForTests) return codeBuddyModelsFetcherForTests(profile, apiKey); + const url = `${profile.canonicalBaseUrl}/v3/config`; + const headers: Record = { + "Accept": "application/json", + "User-Agent": CLI_USER_AGENT, + "X-API-Key": apiKey, + "X-Requested-With": "XMLHttpRequest", + }; + let response: Response; + try { + response = await (deps.fetch ?? fetch)(url, { + headers, + signal: AbortSignal.timeout(deps.timeoutMs ?? 8_000), + }); + } catch (error) { + const name = (error as { name?: string } | null)?.name ?? ""; + if (name === "TimeoutError" || name === "AbortError") { + return { ok: false, error: "timeout", detail: "CodeBuddy model discovery timed out" }; + } + return { ok: false, error: "http", detail: `CodeBuddy config request failed: ${String((error as Error)?.message ?? error).slice(0, 200)}` }; + } + if (response.status !== 200) { + let detail = `HTTP ${response.status}`; + // The error envelope is untrusted upstream output too; read it bounded and skip the + // message entirely when it does not fit a small error-body cap. + const errorBody = await readBoundedBodyText(response, MAX_ERROR_BODY_BYTES); + if (errorBody.ok) { + try { + const envelope = JSON.parse(errorBody.text) as unknown; + if (isPlainObject(envelope) && typeof envelope.msg === "string") { + detail = `HTTP ${response.status} (${String(envelope.msg).slice(0, 120)})`; + } + } catch { + // The status line alone is enough when the body is not a JSON envelope. + } + } + return { ok: false, error: "http", detail }; + } + if (declaredLengthExceeds(response, MAX_CONFIG_BYTES)) { + try { void response.body?.cancel("CodeBuddy config body byte limit reached").catch(() => undefined); } + catch { /* best-effort body teardown */ } + return { ok: false, error: "too_large" }; + } + const body = await readBoundedBodyText(response, MAX_CONFIG_BYTES); + if (!body.ok) { + return body.reason === "exceeded" + ? { ok: false, error: "too_large" } + : { ok: false, error: "http", detail: `CodeBuddy config body read failed${body.detail ? ": " + body.detail : ""}` }; + } + try { + return parseCodeBuddyConfigRoster(JSON.parse(body.text) as unknown); + } catch { + return { ok: false, error: "invalid_output", detail: "CodeBuddy config response is not valid JSON" }; + } +} diff --git a/src/codex/catalog/model-hints.ts b/src/codex/catalog/model-hints.ts index 8fd19f0054f..4e345578c9d 100644 --- a/src/codex/catalog/model-hints.ts +++ b/src/codex/catalog/model-hints.ts @@ -426,6 +426,17 @@ export function suppressedSyntheticMaxCatalogSlugs( export const QUIET_AUTHORITATIVE_CATALOG_PROVIDERS = new Set(["kimi", "xai"]); export const CALLABLE_CONFIGURED_COMPATIBILITY_MODELS: Readonly>> = { + // CodeBuddy's vendor defaults are real callable selectors — both the bundled manifests + // (`product.json` / `product.internal.json`) and `--model` accept them — but the key-scoped + // configuration roster omits them. Without this entry, a successful live roster + // would drop the configured default ("default" for CN, "default-model" for Global) from + // the catalog even though the client can still call it (maintainer review, #5147). + codebuddy: new Set([ + "default-model", + ]), + "codebuddy-cn": new Set([ + "default", + ]), kimi: new Set([ "k3[1m]", "kimi-k2.7-code", diff --git a/src/codex/catalog/provider-models.ts b/src/codex/catalog/provider-models.ts index 23a9696850c..617a3737263 100644 --- a/src/codex/catalog/provider-models.ts +++ b/src/codex/catalog/provider-models.ts @@ -55,6 +55,9 @@ import { fetchCursorUsableModels } from "../../adapters/cursor/live-models"; import { cursorLiveRosterScope, recordLiveCursorClaudeModels, recordLiveCursorMaxModeModels } from "../../adapters/cursor/catalog"; import { fetchQoderModels } from "../../adapters/qoder/live-models"; import { resolveQoderProfile } from "../../adapters/qoder/profiles"; +import { CODEBUDDY_PROFILES, type CodeBuddyProfile } from "../../adapters/codebuddy/profiles"; +import { fetchCodeBuddyModels } from "../../adapters/codebuddy/live-models"; +import { resolveProfileByBaseUrl } from "../../adapters/coding-agent/profile"; import { fetchDevinUsableModels } from "../../adapters/devin/live-models"; import { resolveDevinApiBaseUrl } from "../../oauth/devin/api-base"; import { isCanonicalOpenAiForwardProvider, OPENAI_API_PROVIDER_ID, OPENAI_CODEX_PROVIDER_ID } from "../../providers/openai-tiers"; @@ -245,6 +248,53 @@ export async function fetchProviderModelsWithAuth( ? [...models, vertexDefaultSeed] : models ); + if (prov.adapter === "codebuddy") { + if (!apiKey) return observed(configured, "degraded"); + const resolvedProfile = resolveProfileByBaseUrl(CODEBUDDY_PROFILES, prov.baseUrl); + if (!resolvedProfile) return observed(configured, "degraded"); + const profile = resolvedProfile as CodeBuddyProfile; + // Cache reads/writes are provider/key-fingerprint-scoped: an irreversible fingerprint of + // the configured key means a key switch never reuses the roster cached for the previous + // key. The roster comes from the product configuration endpoint authenticated with that + // same key, so the fingerprint scope and the roster's authority are the same identity: the + // roster is the key's own account answer, never the CLI login's. + const authorityIdentity = createHash("sha256").update(apiKey).digest("hex"); + const fresh = getFreshCached(name, ttlMs, Date.now(), authorityIdentity); + if (fresh) { + return observed(withConfiguredRetention( + applyConfigHintsToCachedModels(name, prov, fresh, contextCap, metadataModelIdCaseFold, captured.effectiveAlias), + ), "authoritative"); + } + const scopedStale = getStaleCached(name, authorityIdentity); + if (isModelsFetchCoolingDown(name, undefined, undefined, authorityIdentity) && scopedStale) { + return observed(withConfiguredRetention( + applyConfigHintsToCachedModels(name, prov, scopedStale, contextCap, metadataModelIdCaseFold, captured.effectiveAlias), + ), "degraded"); + } + const live = await fetchCodeBuddyModels(profile, apiKey); + if (live.ok) { + const discovered = live.models.map(id => ({ + id, + provider: name, + ...catalogHintsFromProviderConfig(name, prov, id, contextCap, metadataModelIdCaseFold, captured.effectiveAlias), + })); + const forCache = withConfiguredRetention(discovered, { retainComboTargets: false }); + if (!setCached(name, forCache, Date.now(), cacheGeneration, authorityIdentity)) { + return observed(withConfiguredRetention(configured), "degraded"); + } + markProviderDiscoveryOk(name, live.models.length); + return observed(withConfiguredRetention(forCache, { warnDrops: true }), "authoritative"); + } + if (isCurrentCacheGeneration()) { + markModelsFetchFailure(name, undefined, authorityIdentity); + markProviderDiscoveryFailed(name, { reason: "provider" }); + console.warn(`[opencodex] CodeBuddy model discovery for "${name}" failed [${live.error}]${live.detail ? ": " + live.detail : ""}; using stale/static catalog degradation.`); + } + const stale = getStaleCached(name, authorityIdentity); + return observed(withConfiguredRetention( + stale ? applyConfigHintsToCachedModels(name, prov, stale, contextCap, metadataModelIdCaseFold, captured.effectiveAlias) : configured, + ), "degraded"); + } if (prov.adapter === "qoder") { if (!apiKey) return observed(configured, "degraded"); const profile = resolveQoderProfile(prov.baseUrl); diff --git a/src/providers/codebuddy-models.ts b/src/providers/codebuddy-models.ts index 0cdcaa1a8c0..59e1509267d 100644 --- a/src/providers/codebuddy-models.ts +++ b/src/providers/codebuddy-models.ts @@ -7,8 +7,12 @@ * Global and CN are deliberately NOT the same roster (§八). Context windows, output caps, vision * and reasoning ladders are filled ONLY where the official manifest states them; a model with no * published figure is omitted rather than guessed (§二十八/§二十九). CodeBuddy exposes no documented - * third-party live `/v1/models` endpoint, so these providers seed a static catalog - * (`liveModels: false`) exactly like the Kiro and Command Code entries. + * third-party live `/v1/models` endpoint, so live discovery instead reads the key-authenticated + * product configuration roster (src/adapters/codebuddy/live-models.ts) and this static catalog is + * only the degraded seed for keys that fail to authenticate or requests that fail. The + * server-side roster can list models the manifest does not know — an account's entitlement can + * be newer than the bundled manifest — so a mismatch between this file and a live roster is + * expected, not a catalog bug. */ /** Global (`public`) session models accepted by `codebuddy --model`. */ diff --git a/src/providers/registry/entries-extended.ts b/src/providers/registry/entries-extended.ts index 11f7dc51cf2..96defecfd92 100644 --- a/src/providers/registry/entries-extended.ts +++ b/src/providers/registry/entries-extended.ts @@ -1402,7 +1402,7 @@ export const PROVIDER_REGISTRY_EXTENDED: readonly ProviderRegistryEntry[] = [ dashboardUrl: "https://www.codebuddy.ai/profile/keys", defaultModel: "default-model", models: CODEBUDDY_GLOBAL_MODELS, - liveModels: false, + liveModels: true, modelContextWindows: CODEBUDDY_GLOBAL_MODEL_CONTEXT_WINDOWS, modelMaxOutputTokens: CODEBUDDY_GLOBAL_MODEL_MAX_OUTPUT_TOKENS, defaultMaxOutputTokens: 32_000, @@ -1415,7 +1415,9 @@ export const PROVIDER_REGISTRY_EXTENDED: readonly ProviderRegistryEntry[] = [ // Official CodeBuddy Code CLI provider, CHINA / `internal` environment. Identical adapter and // binary as `codebuddy`; the region is fixed by the profile's CODEBUDDY_INTERNET_ENVIRONMENT // and this canonical baseUrl. CN key: https://copilot.tencent.com/profile/keys. The CN model - // roster differs from Global (see codebuddy-models.ts) and is seeded separately (§八). + // roster differs from Global and is discovered live from the key-authenticated product + // configuration roster; the seeded list in codebuddy-models.ts is only the degraded + // fallback (§八). id: "codebuddy-cn", label: "CodeBuddy (CN)", adapter: "codebuddy", @@ -1426,7 +1428,7 @@ export const PROVIDER_REGISTRY_EXTENDED: readonly ProviderRegistryEntry[] = [ dashboardUrl: "https://copilot.tencent.com/profile/keys", defaultModel: "default", models: CODEBUDDY_CN_MODELS, - liveModels: false, + liveModels: true, modelContextWindows: CODEBUDDY_CN_MODEL_CONTEXT_WINDOWS, modelMaxOutputTokens: CODEBUDDY_CN_MODEL_MAX_OUTPUT_TOKENS, defaultMaxOutputTokens: 32_000, diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index bbad35f297c..4f72e1ea683 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -405,6 +405,7 @@ "codebuddy-adapter.test.ts": "providers", "codebuddy-live-acceptance.test.ts": "providers", "codebuddy-mcp-server.test.ts": "providers", + "codebuddy-live-models.test.ts": "providers", "codebuddy-protocol.test.ts": "providers", "codebuddy-tool-bridge-turn.test.ts": "providers", "codebuddy-tool-bridge.test.ts": "providers", diff --git a/tests/providers/codebuddy-live-models.test.ts b/tests/providers/codebuddy-live-models.test.ts new file mode 100644 index 00000000000..49fee3f02ae --- /dev/null +++ b/tests/providers/codebuddy-live-models.test.ts @@ -0,0 +1,277 @@ +import { afterEach, describe, expect, spyOn, test } from "bun:test"; +import { createHash } from "node:crypto"; +import { + fetchCodeBuddyModels, + parseCodeBuddyConfigRoster, + setFetchCodeBuddyModelsForTests, +} from "../../src/adapters/codebuddy/live-models"; +import { CODEBUDDY_CN_PROFILE, CODEBUDDY_GLOBAL_PROFILE } from "../../src/adapters/codebuddy/profiles"; +import { gatherRoutedModels, resetCatalogRuntimeStateForTests } from "../../src/codex/catalog"; +import { clearModelCache, setCached } from "../../src/codex/model-cache"; +import type { OcxConfig } from "../../src/types"; + +// Envelope captured 260923 from GET https://www.codebuddy.cn/v3/config with a valid CN key: +// data.agents[0].models is the same 17-id roster the CLI prints for --model on a signed-in +// account of that key, and data.models carries the wider per-account metadata catalog. +function authenticatedEnvelope(models: string[] = ["hy4-preview-f", "hy3", "hy3-x", "deepseek-v4.1-flash", "glm-5.3", "glm-5.3-flash", "glm-5.3-flashx", "glm-5.2", "glm-5.1", "glm-5v-turbo", "minimax-m3-pay", "minimax-m2.7", "kimi-k3-2", "kimi-k2.8-preview", "kimi-k2.7", "kimi-k2.6", "deepseek-v4-pro"]): unknown { + return { code: 0, msg: "ok", requestId: "req-test", data: { agents: [{ name: "cli", models, tools: [] }], enterpriseId: "ent", models: models.map(id => ({ id, name: id })), productFeatures: {} } }; +} + +// Envelope measured 260923 for an absent or invalid key: the anonymous config answers no +// agents array at all and an empty models list, so no roster exists to misattribute. +const ANONYMOUS_ENVELOPE: unknown = { code: 0, msg: "ok", requestId: "req-test", data: { agent: {}, models: [], mcp: {}, codebase: {}, features: {} } }; + +describe("CodeBuddy configuration-roster parser", () => { + test("parses the authenticated key's roster in order", () => { + const result = parseCodeBuddyConfigRoster(authenticatedEnvelope()); + expect(result.ok).toBe(true); + if (result.ok) { + expect(result.models).toHaveLength(17); + expect(result.models[0]).toBe("hy4-preview-f"); + expect(result.models).toContain("kimi-k3-2"); + expect(result.models).toContain("deepseek-v4.1-flash"); + } + }); + + test("filters custom selectors, blanks, and duplicates", () => { + const result = parseCodeBuddyConfigRoster(authenticatedEnvelope(["kimi-k3-2", "custom:mine", "kimi-k3-2", ""])); + expect(result.ok).toBe(true); + if (result.ok) expect(result.models).toEqual(["kimi-k3-2"]); + }); + + test("prefers the cli agent when several agents are declared", () => { + const body = { data: { agents: [ + { name: "other", models: ["other-model"] }, + { name: "cli", models: ["cli-model"] }, + ] } }; + const result = parseCodeBuddyConfigRoster(body); + expect(result.ok).toBe(true); + if (result.ok) expect(result.models).toEqual(["cli-model"]); + }); + + test("the anonymous envelope an invalid key receives fails closed as empty", () => { + const result = parseCodeBuddyConfigRoster(ANONYMOUS_ENVELOPE); + expect(result).toMatchObject({ ok: false, error: "empty" }); + if (!result.ok) expect(result.detail).toContain("anonymous"); + }); + + test("a missing data object fails closed", () => { + expect(parseCodeBuddyConfigRoster({ code: 0, msg: "ok" })).toMatchObject({ ok: false, error: "invalid_output" }); + expect(parseCodeBuddyConfigRoster(null)).toMatchObject({ ok: false, error: "invalid_output" }); + }); + + test("an authenticated envelope with an empty agent roster fails closed", () => { + expect(parseCodeBuddyConfigRoster(authenticatedEnvelope([]))).toMatchObject({ ok: false, error: "empty" }); + }); +}); + +describe("CodeBuddy live model fetch", () => { + function recordingFetch(status: number, body: unknown) { + const seen: { url: string; headers: Record }[] = []; + const fetchLike = (async (url: RequestInfo | URL, init?: RequestInit) => { + const headers: Record = {}; + for (const [key, value] of Object.entries(init?.headers ?? {})) headers[key.toLowerCase()] = String(value); + seen.push({ url: String(url), headers }); + return new Response(status === 200 ? JSON.stringify(body) : JSON.stringify(body), { status }); + }) as typeof fetch; + return { fetchLike, seen }; + } + + test("requests the region's configuration endpoint with the key and returns the roster", async () => { + const { fetchLike, seen } = recordingFetch(200, authenticatedEnvelope()); + const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); + expect(result.ok).toBe(true); + if (result.ok) expect(result.models).toContain("kimi-k3-2"); + expect(seen).toHaveLength(1); + expect(seen[0]!.url).toBe("https://www.codebuddy.cn/v3/config"); + // The roster's authority is the key on the request: the header must carry it, and the + // gateway requires a CLI-shaped User-Agent before it authenticates the key at all. + expect(seen[0]!.headers["x-api-key"]).toBe("cb-cn-key"); + expect(seen[0]!.headers["user-agent"]).toMatch(/^CLI\/\d+\.\d+\.\d+ CodeBuddy\/\d+\.\d+\.\d+$/); + }); + + test("the global profile addresses the global configuration endpoint", async () => { + const { fetchLike, seen } = recordingFetch(200, authenticatedEnvelope(["glm-5.3"])); + const result = await fetchCodeBuddyModels(CODEBUDDY_GLOBAL_PROFILE, "cb-global-key", { fetch: fetchLike }); + expect(result.ok).toBe(true); + expect(seen[0]!.url).toBe("https://www.codebuddy.ai/v3/config"); + }); + + test("a non-200 answer is a clear error carrying the gateway's message", async () => { + const { fetchLike } = recordingFetch(400, { code: 12403, msg: "check ua, get coding copilot version error" }); + const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); + expect(result).toMatchObject({ ok: false, error: "http" }); + if (!result.ok) expect(result.detail).toContain("check ua"); + }); + + test("a timed-out request is a timeout, never a crash", async () => { + const fetchLike = (async () => { + throw Object.assign(new Error("timed out"), { name: "TimeoutError" }); + }) as typeof fetch; + const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); + expect(result).toMatchObject({ ok: false, error: "timeout" }); + }); + + test("a body that is not JSON fails closed as invalid output", async () => { + const fetchLike = (async () => new Response("gateway error page", { status: 200 })) as typeof fetch; + const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); + expect(result).toMatchObject({ ok: false, error: "invalid_output" }); + }); + + test("a chunked body that crosses the byte limit fails as too_large and cancels the stream", async () => { + // 256 KiB chunks: the third crossing chunk must cancel the reader, so a compromised + // upstream cannot keep discovery reading (or buffering) past the advertised cap. + const chunk = new Uint8Array(256 * 1024).fill(0x61); + let cancelled = false; + let pulls = 0; + const stream = new ReadableStream({ + pull(controller) { + pulls += 1; + controller.enqueue(chunk); + }, + cancel() { cancelled = true; }, + }); + const fetchLike = (async () => new Response(stream, { status: 200, headers: { "content-type": "application/json" } })) as typeof fetch; + const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); + expect(result).toMatchObject({ ok: false, error: "too_large" }); + expect(cancelled).toBe(true); + // Two chunks fit under the cap; the third is the crossing one. The stream machinery + // may prefetch one chunk ahead, so the bound is "a handful", never stream-sized. + expect(pulls).toBeLessThanOrEqual(4); + }); + + test("a declared Content-Length above the cap is refused without reading the body", async () => { + const chunk = new Uint8Array(16).fill(0x61); + let pulls = 0; + let cancelled = false; + const stream = new ReadableStream({ + pull(controller) { + pulls += 1; + controller.enqueue(chunk); + }, + cancel() { cancelled = true; }, + }); + const fetchLike = (async () => new Response(stream, { + status: 200, + headers: { "content-type": "application/json", "content-length": String(600 * 1024) }, + })) as typeof fetch; + const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); + expect(result).toMatchObject({ ok: false, error: "too_large" }); + // The declared length is refused before the body is read; the wrapper's teardown may + // still cost one prefetch chunk, never the declared 600 KiB. + expect(pulls).toBeLessThanOrEqual(1); + expect(cancelled).toBe(true); + }); +}); + +describe("CodeBuddy catalog cache isolation", () => { + afterEach(() => { + setFetchCodeBuddyModelsForTests(null); + clearModelCache(); + resetCatalogRuntimeStateForTests(); + }); + + function codeBuddyConfig(apiKey: string): OcxConfig { + return { + providers: { + "codebuddy-cn": { + adapter: "codebuddy", + baseUrl: "https://www.codebuddy.cn", + authMode: "key", + apiKey, + liveModels: true, + defaultModel: "default", + // Mirrors the registry seed: the static list ships the vendor default even though + // the key-scoped configuration roster does not list it. + models: ["default"], + }, + }, + } as unknown as OcxConfig; + } + + test("a fetch-failure cooldown for one key does not suppress another key's discovery", async () => { + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + let keyBFetches = 0; + setFetchCodeBuddyModelsForTests((_profile, apiKey) => { + if (apiKey === "cb-key-a") return { ok: false, error: "http", detail: "denied" }; + keyBFetches += 1; + return { ok: true, models: ["roster-b-model"] }; + }); + + // Seed a stale (TTL-expired) roster for key B so the cooldown branch is reachable. + const identityB = createHash("sha256").update("cb-key-b").digest("hex"); + setCached("codebuddy-cn", [{ id: "roster-b-old", provider: "codebuddy-cn" }], Date.now() - 3_600_000, undefined, identityB); + + // Key A fails discovery: the cooldown must be recorded against A's fingerprint only. + const withA = await gatherRoutedModels(codeBuddyConfig("cb-key-a")); + expect(withA.filter(m => m.provider === "codebuddy-cn").map(m => m.id)).not.toContain("roster-b-model"); + + // Key B still has its own stale roster, but A's cooldown is not B's: discovery must run. + const withB = await gatherRoutedModels(codeBuddyConfig("cb-key-b")); + expect(keyBFetches).toBe(1); + expect(withB.filter(m => m.provider === "codebuddy-cn").map(m => m.id)).toContain("roster-b-model"); + } finally { + warn.mockRestore(); + } + }); + + test("a second key never receives the first key's fresh or stale roster", async () => { + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + setFetchCodeBuddyModelsForTests((_profile, apiKey) => ( + apiKey === "cb-key-a" + ? { ok: true, models: ["roster-a-model"] } + : { ok: false, error: "http", detail: "denied" } + )); + + const first = await gatherRoutedModels(codeBuddyConfig("cb-key-a")); + const firstIds = first.filter(model => model.provider === "codebuddy-cn").map(model => model.id); + expect(firstIds).toContain("roster-a-model"); + // The vendor default is callable even though the live roster omits it. + expect(firstIds).toContain("default"); + + // Key B's fetch fails: neither the fresh nor the stale cache entry recorded for key A + // may leak into key B's catalog. + const second = await gatherRoutedModels(codeBuddyConfig("cb-key-b")); + const secondIds = second.filter(model => model.provider === "codebuddy-cn").map(model => model.id); + expect(secondIds).not.toContain("roster-a-model"); + expect(secondIds).toContain("default"); + } finally { + warn.mockRestore(); + } + }); +}); + +// The roster authority is the key on the request, so the cached roster can only ever be the +// key's own answer. The remaining cross-key guard is the cooldown/fingerprint isolation above. +test("an invalid key answers the anonymous envelope and never caches a roster", async () => { + setFetchCodeBuddyModelsForTests(() => ({ ok: false, error: "empty", detail: "anonymous" })); + const warn = spyOn(console, "warn").mockImplementation(() => {}); + try { + clearModelCache(); + resetCatalogRuntimeStateForTests(); + const config = { + providers: { + "codebuddy-cn": { + adapter: "codebuddy", + baseUrl: "https://www.codebuddy.cn", + authMode: "key", + apiKey: "cb-wrong-key", + liveModels: true, + defaultModel: "default", + models: ["default"], + }, + }, + } as unknown as OcxConfig; + const models = await gatherRoutedModels(config); + const ids = models.filter(m => m.provider === "codebuddy-cn").map(m => m.id); + // Degraded to the configured selector only — no roster from any other account. + expect(ids).toEqual(["default"]); + } finally { + warn.mockRestore(); + setFetchCodeBuddyModelsForTests(null); + clearModelCache(); + resetCatalogRuntimeStateForTests(); + } +}); From 82bcdfe882fbd7dc56cb2b2d72b9127772555358 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 02:53:36 +0900 Subject: [PATCH 07/19] fix(codebuddy): keep credentialed discovery failures out of logs Return only a bounded failure category and optional HTTP status from key-scoped roster discovery. Stop reading upstream error messages and avoid logging transport exception text or injected detail; add redaction regressions. --- .../src/content/docs/guides/providers.md | 2 +- src/adapters/codebuddy/live-models.ts | 43 +++++++------------ src/codex/catalog/provider-models.ts | 2 +- structure/catalog.md | 2 +- structure/providers-and-adapters.md | 4 ++ tests/providers/codebuddy-live-models.test.ts | 43 ++++++++++++++++--- 6 files changed, 58 insertions(+), 38 deletions(-) diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 6b931db131b..3f98b844a94 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -889,7 +889,7 @@ OpenCodex provides official adapter support for Tencent Cloud's CodeBuddy Code C - Global: [CodeBuddy Global API Keys](https://www.codebuddy.ai/profile/keys) - CN: [CodeBuddy CN API Keys](https://copilot.tencent.com/profile/keys) - **Region Isolation:** `codebuddy` and `codebuddy-cn` use separate canonical endpoints (`https://www.codebuddy.ai` and `https://www.codebuddy.cn`) and isolated child environments (`CODEBUDDY_INTERNET_ENVIRONMENT=public` vs `internal`). Credentials are strictly region-scoped and never exchanged across environments. Overriding the canonical base URL fails closed. -- **Model Discovery:** the proxy requests the CodeBuddy product configuration (`GET {baseUrl}/v3/config`) with the configured key as the `X-API-Key` header, and the roster in that answer is the authoritative roster of discovered models: it is the key's own account configuration, so it is proven to belong to the key — a different or wrong key answers the anonymous envelope with no roster instead of another account's models. The authenticated roster is the same list the CLI prints for `--model` (the "Currently supported" line of a signed-in CLI), can differ from the static manifest bundled with the CLI, and the vendor default selectors (`default` for CN, `default-model` for Global) never appear in it but remain callable: the catalog retains them during live discovery and on every fallback path. On start/sync the proxy binds the cached roster to an irreversible fingerprint of the configured key, so a key switch never observes a roster cached for the previous key, and degrades to the stale provider/key-fingerprint-scoped cache, then to the static seed in `src/providers/codebuddy-models.ts`, when the key does not authenticate or the request fails. +- **Model Discovery:** the proxy requests the CodeBuddy product configuration (`GET {baseUrl}/v3/config`) with the configured key as the `X-API-Key` header, and the roster in that answer is the authoritative roster of discovered models: it is the key's own account configuration, so it is proven to belong to the key — a different or wrong key answers the anonymous envelope with no roster instead of another account's models. The authenticated roster is the same list the CLI prints for `--model` (the "Currently supported" line of a signed-in CLI), can differ from the static manifest bundled with the CLI, and the vendor default selectors (`default` for CN, `default-model` for Global) never appear in it but remain callable: the catalog retains them during live discovery and on every fallback path. On start/sync the proxy binds the cached roster to an irreversible fingerprint of the configured key, so a key switch never observes a roster cached for the previous key, and degrades to the stale provider/key-fingerprint-scoped cache, then to the static seed in `src/providers/codebuddy-models.ts`, when the key does not authenticate or the request fails. Discovery failure logs contain only a category and HTTP status, without the gateway's message or a raw transport exception. - **Tool Ownership and the Tool Bridge:** The CLI is always spawned with `--tools ""` and `--strict-mcp-config`, so it has no built-in or user-configured tools of its own. When a request carries a Codex tool catalog, the provider arms a capture-only MCP bridge: the validated catalog and MCP config are written to a private temp dir, the CLI is launched with `--mcp-config` and an exact `--allowedTools` list, and the `system/init` frame must report exactly that bridge server as connected or the turn fails closed. The bridge advertises the Codex tools and captures proposed calls but never executes anything: a completed tool-call batch is returned as `function_call` items (names mapped back to the request's wire names, at most 16 calls per assistant message), the process tree is terminated at `message_stop`, and the external Codex client alone performs approval, sandboxing, and execution. Tool results come back as the next request's input, and the conversation continues. Requests without tools keep the plain text-and-reasoning shape. If the CLI writes an unquoted DSML `calls` control line followed by a `functions.*` invoke control line into text or reasoning, OpenCodex refuses the turn instead of forwarding the scaffold or interpreting it as an executable call. DSML discussed or quoted in prose, inline code, fenced code, or source examples remains ordinary answer text. - **Tool Choice Enforcement:** When a request specifies `tool_choice: "required"` or selects a specific named tool, the bridge expects a tool call from the model. If the CLI completes the turn with plain text instead of capturing a tool call, OpenCodex fails closed with a 502 `tool_call_required` error rather than returning an invalid text completion. - **Governance Status:** Whether routing this vendor automation surface behind a proxy for a third-party agent satisfies CodeBuddy's acceptable-use terms is an open question flagged for maintainer security review (see the governance note in the provider registry entry). Treat this provider as pending that review, and keep the tool bridge's ownership boundary in mind: the nested CLI advertises tools but never executes them, and approval, sandboxing, and execution remain with the external Codex client. diff --git a/src/adapters/codebuddy/live-models.ts b/src/adapters/codebuddy/live-models.ts index e646f3694eb..94649ac62b1 100644 --- a/src/adapters/codebuddy/live-models.ts +++ b/src/adapters/codebuddy/live-models.ts @@ -3,11 +3,10 @@ import type { CodeBuddyProfile } from "./profiles"; const MAX_CONFIG_BYTES = 512 * 1024; const MAX_MODELS = 128; -const MAX_ERROR_BODY_BYTES = 4 * 1024; export type CodeBuddyModelsResult = | { ok: true; models: string[] } - | { ok: false; error: "http" | "timeout" | "invalid_output" | "empty" | "too_large"; detail?: string }; + | { ok: false; error: "http" | "timeout" | "invalid_output" | "empty" | "too_large"; status?: number }; export interface CodeBuddyConfigFetchDeps { /** Test seam for the outbound request; defaults to global fetch. */ @@ -43,7 +42,7 @@ const CLI_USER_AGENT = "CLI/2.126.0 CodeBuddy/2.126.0"; */ async function readBoundedBodyText(res: Response, cap: number): Promise< | { ok: true; text: string } - | { ok: false; reason: "exceeded" | "read"; detail?: string } + | { ok: false; reason: "exceeded" | "read" } > { if (!res.body) return { ok: true, text: "" }; const reader = res.body.getReader(); @@ -64,8 +63,8 @@ async function readBoundedBodyText(res: Response, cap: number): Promise< } out += decoder.decode(); return { ok: true, text: out }; - } catch (error) { - return { ok: false, reason: "read", detail: String((error as Error)?.message ?? error).slice(0, 200) }; + } catch { + return { ok: false, reason: "read" }; } } @@ -88,14 +87,14 @@ function declaredLengthExceeds(response: Response, cap: number): boolean { * not shared catalog rows, and are excluded. */ export function parseCodeBuddyConfigRoster(body: unknown): CodeBuddyModelsResult { - if (!isPlainObject(body)) return { ok: false, error: "invalid_output", detail: "CodeBuddy config response is not an object" }; + if (!isPlainObject(body)) return { ok: false, error: "invalid_output" }; const data = body.data; - if (!isPlainObject(data)) return { ok: false, error: "invalid_output", detail: "CodeBuddy config envelope is missing its data object" }; + if (!isPlainObject(data)) return { ok: false, error: "invalid_output" }; const agents = data.agents; if (!Array.isArray(agents)) { // The authenticated envelope always carries an agents array; the anonymous one (absent or // invalid key) does not. Both fail closed here, but the distinction names the cause. - return { ok: false, error: "empty", detail: "CodeBuddy answered the anonymous config: the key did not authenticate" }; + return { ok: false, error: "empty" }; } // The catalog mirrors what the CLI itself accepts for --model: the default agent's models. // agents has carried exactly one entry named "cli" so far; prefer it by name and fall back to @@ -113,7 +112,7 @@ export function parseCodeBuddyConfigRoster(body: unknown): CodeBuddyModelsResult models.push(id); if (models.length >= MAX_MODELS) break; } - return models.length > 0 ? { ok: true, models } : { ok: false, error: "empty", detail: "CodeBuddy config roster is empty" }; + return models.length > 0 ? { ok: true, models } : { ok: false, error: "empty" }; } /** @@ -151,26 +150,14 @@ export async function fetchCodeBuddyModels( } catch (error) { const name = (error as { name?: string } | null)?.name ?? ""; if (name === "TimeoutError" || name === "AbortError") { - return { ok: false, error: "timeout", detail: "CodeBuddy model discovery timed out" }; + return { ok: false, error: "timeout" }; } - return { ok: false, error: "http", detail: `CodeBuddy config request failed: ${String((error as Error)?.message ?? error).slice(0, 200)}` }; + return { ok: false, error: "http" }; } if (response.status !== 200) { - let detail = `HTTP ${response.status}`; - // The error envelope is untrusted upstream output too; read it bounded and skip the - // message entirely when it does not fit a small error-body cap. - const errorBody = await readBoundedBodyText(response, MAX_ERROR_BODY_BYTES); - if (errorBody.ok) { - try { - const envelope = JSON.parse(errorBody.text) as unknown; - if (isPlainObject(envelope) && typeof envelope.msg === "string") { - detail = `HTTP ${response.status} (${String(envelope.msg).slice(0, 120)})`; - } - } catch { - // The status line alone is enough when the body is not a JSON envelope. - } - } - return { ok: false, error: "http", detail }; + try { void response.body?.cancel().catch(() => undefined); } + catch { /* best-effort body teardown */ } + return { ok: false, error: "http", status: response.status }; } if (declaredLengthExceeds(response, MAX_CONFIG_BYTES)) { try { void response.body?.cancel("CodeBuddy config body byte limit reached").catch(() => undefined); } @@ -181,11 +168,11 @@ export async function fetchCodeBuddyModels( if (!body.ok) { return body.reason === "exceeded" ? { ok: false, error: "too_large" } - : { ok: false, error: "http", detail: `CodeBuddy config body read failed${body.detail ? ": " + body.detail : ""}` }; + : { ok: false, error: "http" }; } try { return parseCodeBuddyConfigRoster(JSON.parse(body.text) as unknown); } catch { - return { ok: false, error: "invalid_output", detail: "CodeBuddy config response is not valid JSON" }; + return { ok: false, error: "invalid_output" }; } } diff --git a/src/codex/catalog/provider-models.ts b/src/codex/catalog/provider-models.ts index 617a3737263..f8592053afa 100644 --- a/src/codex/catalog/provider-models.ts +++ b/src/codex/catalog/provider-models.ts @@ -288,7 +288,7 @@ export async function fetchProviderModelsWithAuth( if (isCurrentCacheGeneration()) { markModelsFetchFailure(name, undefined, authorityIdentity); markProviderDiscoveryFailed(name, { reason: "provider" }); - console.warn(`[opencodex] CodeBuddy model discovery for "${name}" failed [${live.error}]${live.detail ? ": " + live.detail : ""}; using stale/static catalog degradation.`); + console.warn(`[opencodex] CodeBuddy model discovery failed [${live.error}]${live.status === undefined ? "" : ` status=${live.status}`}; using stale/static catalog degradation.`); } const stale = getStaleCached(name, authorityIdentity); return observed(withConfiguredRetention( diff --git a/structure/catalog.md b/structure/catalog.md index 3be918b4193..aa8b82c0fbe 100644 --- a/structure/catalog.md +++ b/structure/catalog.md @@ -212,7 +212,7 @@ destination is invalid, registered Devin discovery and routing use the registry' instead of a stale configured override. For Devin, the irreversible roster fingerprint covers both credential and validated destination, so switching either observes neither fresh nor stale data recorded under the previous pair. -Entitlement-specific rosters (Qoder, Devin, Cursor) additionally bind their cache entry to an +Entitlement-specific rosters (Qoder, Devin, Cursor, CodeBuddy) additionally bind their cache entry to an irreversible credential fingerprint: a credential switch observes neither the fresh nor the stale roster recorded under the previous credential, and a failed discovery's cooldown neither supplies the previous credential's stale roster nor suppresses the next credential's first discovery. diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 4a874b48e22..601d4089f63 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -139,6 +139,10 @@ OAuth presets resolve discovery against the same canonical registry transport as before any adapter-specific transport override, so a stale configured `baseUrl` cannot receive an OAuth bearer token. +CodeBuddy discovery in `src/adapters/codebuddy/live-models.ts` reads the roster scoped to the +configured key. Its failure result carries only a category and optional HTTP status; untrusted +gateway messages and transport exceptions do not reach catalog warnings. + Provider request pacing in `src/providers/request-pacing.ts` combines start intervals with optional `maxConcurrentRequests` limits. Provider capacity is shared across models; exact-model limits apply in addition to that capacity. Admission reserves both counters atomically, and eligible diff --git a/tests/providers/codebuddy-live-models.test.ts b/tests/providers/codebuddy-live-models.test.ts index 49fee3f02ae..5e5125f2fac 100644 --- a/tests/providers/codebuddy-live-models.test.ts +++ b/tests/providers/codebuddy-live-models.test.ts @@ -52,7 +52,6 @@ describe("CodeBuddy configuration-roster parser", () => { test("the anonymous envelope an invalid key receives fails closed as empty", () => { const result = parseCodeBuddyConfigRoster(ANONYMOUS_ENVELOPE); expect(result).toMatchObject({ ok: false, error: "empty" }); - if (!result.ok) expect(result.detail).toContain("anonymous"); }); test("a missing data object fails closed", () => { @@ -97,11 +96,10 @@ describe("CodeBuddy live model fetch", () => { expect(seen[0]!.url).toBe("https://www.codebuddy.ai/v3/config"); }); - test("a non-200 answer is a clear error carrying the gateway's message", async () => { + test("a non-200 answer reports only its status", async () => { const { fetchLike } = recordingFetch(400, { code: 12403, msg: "check ua, get coding copilot version error" }); const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); - expect(result).toMatchObject({ ok: false, error: "http" }); - if (!result.ok) expect(result.detail).toContain("check ua"); + expect(result).toEqual({ ok: false, error: "http", status: 400 }); }); test("a timed-out request is a timeout, never a crash", async () => { @@ -112,6 +110,22 @@ describe("CodeBuddy live model fetch", () => { expect(result).toMatchObject({ ok: false, error: "timeout" }); }); + test("credentialed HTTP failures expose status without the upstream message", async () => { + const marker = "private upstream message marker"; + const { fetchLike } = recordingFetch(403, { msg: marker }); + const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); + expect(result).toEqual({ ok: false, error: "http", status: 403 }); + expect(JSON.stringify(result)).not.toContain(marker); + }); + + test("credentialed transport failures expose a category without exception text", async () => { + const marker = "private transport exception marker"; + const fetchLike = (async () => { throw new Error(marker); }) as typeof fetch; + const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); + expect(result).toEqual({ ok: false, error: "http" }); + expect(JSON.stringify(result)).not.toContain(marker); + }); + test("a body that is not JSON fails closed as invalid output", async () => { const fetchLike = (async () => new Response("gateway error page", { status: 200 })) as typeof fetch; const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); @@ -194,7 +208,7 @@ describe("CodeBuddy catalog cache isolation", () => { try { let keyBFetches = 0; setFetchCodeBuddyModelsForTests((_profile, apiKey) => { - if (apiKey === "cb-key-a") return { ok: false, error: "http", detail: "denied" }; + if (apiKey === "cb-key-a") return { ok: false, error: "http" }; keyBFetches += 1; return { ok: true, models: ["roster-b-model"] }; }); @@ -222,7 +236,7 @@ describe("CodeBuddy catalog cache isolation", () => { setFetchCodeBuddyModelsForTests((_profile, apiKey) => ( apiKey === "cb-key-a" ? { ok: true, models: ["roster-a-model"] } - : { ok: false, error: "http", detail: "denied" } + : { ok: false, error: "http" } )); const first = await gatherRoutedModels(codeBuddyConfig("cb-key-a")); @@ -241,12 +255,27 @@ describe("CodeBuddy catalog cache isolation", () => { warn.mockRestore(); } }); + + test("catalog failure logging omits any untrusted detail from a credentialed request", async () => { + const marker = "private catalog failure marker"; + const untrustedFailure = { ok: false as const, error: "http" as const, detail: marker }; + setFetchCodeBuddyModelsForTests(() => untrustedFailure); + const warnings: string[] = []; + const warn = spyOn(console, "warn").mockImplementation((...args) => { warnings.push(args.map(String).join(" ")); }); + try { + await gatherRoutedModels(codeBuddyConfig("cb-key-a")); + expect(warnings.join(" ")).not.toContain(marker); + expect(warnings.join(" ")).toContain("http"); + } finally { + warn.mockRestore(); + } + }); }); // The roster authority is the key on the request, so the cached roster can only ever be the // key's own answer. The remaining cross-key guard is the cooldown/fingerprint isolation above. test("an invalid key answers the anonymous envelope and never caches a roster", async () => { - setFetchCodeBuddyModelsForTests(() => ({ ok: false, error: "empty", detail: "anonymous" })); + setFetchCodeBuddyModelsForTests(() => ({ ok: false, error: "empty" })); const warn = spyOn(console, "warn").mockImplementation(() => {}); try { clearModelCache(); From ce6a981719557a79de858a0db4e6993dfffc8d77 Mon Sep 17 00:00:00 2001 From: chilung Date: Sun, 27 Sep 2026 02:56:57 +0900 Subject: [PATCH 08/19] perf(server): single-pass log query and hoist passthrough drop headers (#4740) Carried from #4740 as one squashed commit. Ported its log-query implementation to the current request-log-filter.ts owner. Co-authored-by: chilung --- src/server/index.ts | 2 +- src/server/management/logs-usage-routes.ts | 7 +-- src/server/relay.ts | 31 ++++++------ src/server/request-log-filter.ts | 55 +++++++++++----------- src/server/request-log.ts | 2 +- tests/usage/request-log.test.ts | 37 +++++++++++++++ 6 files changed, 86 insertions(+), 48 deletions(-) diff --git a/src/server/index.ts b/src/server/index.ts index eadd955ea12..620329a490f 100644 --- a/src/server/index.ts +++ b/src/server/index.ts @@ -117,7 +117,7 @@ import { setUsageLedgerRetention } from "./usage-ledger-retention"; import { admitHttpWorkflowTurn, workflowDecisionRefusalResponse, type WorkflowRefusalLog } from "./workflow-refusal"; export { addFinalRequestLog, - filterRequestLogs, + filterRequestLogs, queryRequestLogs, hydrateRequestLogsFromDisk, httpStatusForTerminalStatus, httpStatusFromTerminalError, diff --git a/src/server/management/logs-usage-routes.ts b/src/server/management/logs-usage-routes.ts index 73ad30cb250..1c67753031d 100644 --- a/src/server/management/logs-usage-routes.ts +++ b/src/server/management/logs-usage-routes.ts @@ -67,7 +67,7 @@ import { } from "../../lib/debug-settings"; import type { OcxClaudeCodeConfig, OcxConfig, OcxCustomModel, OcxProviderConfig } from "../../types"; import { drainAndShutdown } from "../lifecycle"; -import { filterRequestLogs, filteredRequestLogCount, getRequestLogEntries, type RequestLogEntry } from "../request-log"; +import { getRequestLogEntries, queryRequestLogs, type RequestLogEntry } from "../request-log"; import { decodeRequestLogCursor, selectRequestLogPoll } from "../request-log-cursor"; import { estimateComboCost, estimateRequestCost, normalizeCostTokens, tokensPerSecond } from "../../usage/cost"; import { userCostOverlayVersion } from "../../usage/user-cost-overlays"; @@ -120,11 +120,12 @@ export async function handleLogsUsageRoutes(ctx: ManagementContext): Promise requestLogDto(entry)); + const logs = queried.logs.map(entry => requestLogDto(entry)); const poll = selectRequestLogPoll(logs, url.searchParams, cursor); return jsonResponse({ timeZone: Intl.DateTimeFormat().resolvedOptions().timeZone, diff --git a/src/server/relay.ts b/src/server/relay.ts index afcafc66173..c2cc93758f0 100644 --- a/src/server/relay.ts +++ b/src/server/relay.ts @@ -1624,6 +1624,21 @@ export const CODEX_SAFETY_BUFFERING_HEADERS = [ const CODEX_SAFETY_BUFFERING_HEADER_SET: ReadonlySet = new Set(CODEX_SAFETY_BUFFERING_HEADERS); +const PASSTHROUGH_DROP_HEADERS: ReadonlySet = new Set([ + "content-encoding", + "content-length", + "transfer-encoding", + "connection", + "keep-alive", + "proxy-authenticate", + "proxy-authorization", + "set-cookie", + "set-cookie2", + "te", + "trailer", + "upgrade", +]); + export interface CodexSafetyBufferingFilterOptions { /** * Drop Codex safety-buffering hints: the `x-codex-safety-buffering-*` response @@ -1642,24 +1657,10 @@ export function codexSafetyBufferingFilterOptions( export function sanitizePassthroughHeaders(upstream: Headers, options?: CodexSafetyBufferingFilterOptions): Headers { const dropSafetyBuffering = options?.dropCodexSafetyBuffering === true; - const DROP = new Set([ - "content-encoding", - "content-length", - "transfer-encoding", - "connection", - "keep-alive", - "proxy-authenticate", - "proxy-authorization", - "set-cookie", - "set-cookie2", - "te", - "trailer", - "upgrade", - ]); const out = new Headers(); upstream.forEach((value, key) => { const lower = key.toLowerCase(); - if (DROP.has(lower)) return; + if (PASSTHROUGH_DROP_HEADERS.has(lower)) return; if (dropSafetyBuffering && CODEX_SAFETY_BUFFERING_HEADER_SET.has(lower)) return; out.set(key, value); }); diff --git a/src/server/request-log-filter.ts b/src/server/request-log-filter.ts index 9b45191f4d9..162af9ff840 100644 --- a/src/server/request-log-filter.ts +++ b/src/server/request-log-filter.ts @@ -22,49 +22,47 @@ export function matchesProtocolMode(entry: Pick entry.provider === provider - || entry.attempts?.some(attempt => attempt.provider === provider)); - } const conversationId = params.get("conversationId")?.trim() || params.get("conversation")?.trim(); - if (conversationId) { - filtered = filtered.filter(entry => matchesLogConversationId(entry.conversationId, conversationId)); - } // #2704: there was no `model` clause at all, so `?model=x` was ACCEPTED and silently // ignored -- worse than an error, because it yields wrong conclusions from output that // looks correct. Attempts are matched for the same reason `provider` matches them: a // request that failed over should be findable by the model that actually served it. const model = params.get("model")?.trim(); - if (model) { - filtered = filtered.filter(entry => entry.model === model - || entry.attempts?.some(attempt => attempt.model === model)); - } // #4057: "which account served this request" is the first question asked when one provider // holds several accounts, and until now the only way to answer it was to grep usage.jsonl by // hand. Attempts are matched for the same reason `provider` and `model` match them: when a // request failed over between pool accounts, a search for the account that finally served it // has to find that request, not only the account that first refused it. const account = params.get("account")?.trim(); - if (account) { - filtered = filtered.filter(entry => entry.accountLogLabel === account - || entry.attempts?.some(attempt => attempt.accountLogLabel === account)); - } - const protocolMode = params.get("protocolMode")?.trim().toLowerCase(); - if (protocolMode) filtered = filtered.filter(entry => matchesProtocolMode(entry, protocolMode)); const status = params.get("status")?.trim().toLowerCase(); - if (status) { - filtered = /^[1-5]xx$/.test(status) - ? filtered.filter(entry => Math.floor(entry.status / 100) === Number(status[0])) - : filtered.filter(entry => String(entry.status) === status); + + const hasFilters = Boolean(provider || conversationId || model || account || status); + let filtered: RequestLogEntry[]; + if (!hasFilters) { + filtered = logs; + } else { + const statusClass = status && /^[1-5]xx$/.test(status) ? Number(status[0]) : undefined; + filtered = []; + for (const entry of logs) { + if (provider && entry.provider !== provider && !entry.attempts?.some(attempt => attempt.provider === provider)) continue; + if (conversationId && !matchesLogConversationId(entry.conversationId, conversationId)) continue; + if (model && entry.model !== model && !entry.attempts?.some(attempt => attempt.model === model)) continue; + if (account && entry.accountLogLabel !== account && !entry.attempts?.some(attempt => attempt.accountLogLabel === account)) continue; + if (status && (statusClass === undefined ? String(entry.status) !== status : Math.floor(entry.status / 100) !== statusClass)) continue; + filtered.push(entry); + } } const tailRaw = params.get("tail")?.trim(); if (tailRaw) { const tail = Number.parseInt(tailRaw, 10); if (Number.isFinite(tail) && tail > 0) filtered = filtered.slice(-Math.min(tail, MAX_LOG_SIZE)); } + const total = filtered.length; const offsetRaw = params.get("offset")?.trim(); const limitRaw = params.get("limit")?.trim(); if (limitRaw) { @@ -81,12 +79,13 @@ export function filterRequestLogs(logs: RequestLogEntry[], params: URLSearchPara } } } - return filtered; + return { total, logs: filtered }; +} + +export function filterRequestLogs(logs: RequestLogEntry[], params: URLSearchParams): RequestLogEntry[] { + return queryRequestLogs(logs, params).logs; } export function filteredRequestLogCount(logs: RequestLogEntry[], params: URLSearchParams): number { - const withoutPagination = new URLSearchParams(params); - withoutPagination.delete("limit"); - withoutPagination.delete("offset"); - return filterRequestLogs(logs, withoutPagination).length; + return queryRequestLogs(logs, params).total; } diff --git a/src/server/request-log.ts b/src/server/request-log.ts index e565f546b06..cfbb181bbd5 100644 --- a/src/server/request-log.ts +++ b/src/server/request-log.ts @@ -73,7 +73,7 @@ import { type UsageDebugBodyKind, } from "../usage/debug"; import { MAX_LOG_SIZE } from "./request-log-filter"; -export { filterRequestLogs, filteredRequestLogCount } from "./request-log-filter"; +export { filterRequestLogs, filteredRequestLogCount, queryRequestLogs } from "./request-log-filter"; import { enforceAppOwnedMemoryBudget, type RetainedStoreSnapshot } from "../lib/app-owned-memory"; import { capEstimateAtContextWindow } from "../lib/token-estimate"; import { inferCursorContextWindow } from "../adapters/cursor/discovery"; diff --git a/tests/usage/request-log.test.ts b/tests/usage/request-log.test.ts index bb6208cdcb1..d4e805fc4b0 100644 --- a/tests/usage/request-log.test.ts +++ b/tests/usage/request-log.test.ts @@ -7,6 +7,7 @@ import { responseWithDeferredRequestLog, requestLogErrorCode, requestLogSpeedLabel, + queryRequestLogs, type RequestLogEntry, } from "../../src/server"; import { @@ -938,6 +939,42 @@ describe("request log metadata", () => { expect(filterRequestLogs(logs, new URLSearchParams("limit=3")).map(entry => entry.requestId)).toEqual(["r7", "r8", "r9"]); }); + test("queryRequestLogs returns both total count and paginated logs in a single pass", () => { + const logs = [ + log({ requestId: "r0", provider: "openai", model: "gpt-4", status: 200, accountLogLabel: "acc-1" }), + log({ requestId: "r1", provider: "openai", model: "gpt-4", status: 500, accountLogLabel: "acc-1" }), + log({ requestId: "r2", provider: "xai", model: "grok-1", status: 200, accountLogLabel: "acc-2" }), + log({ requestId: "r3", provider: "openai", model: "gpt-4", status: 200, accountLogLabel: "acc-1" }), + log({ requestId: "r4", provider: "openai", model: "gpt-4", status: 204, accountLogLabel: "acc-1" }), + log({ requestId: "r5", provider: "openai", model: "gpt-4", status: 200, accountLogLabel: "acc-1" }), + ]; + + // Filter by provider, status 2xx, with limit and offset + const query = new URLSearchParams("provider=openai&status=2xx&limit=2&offset=1"); + const result = queryRequestLogs(logs, query); + + // Matches r0 (200), r3 (200), r4 (204), r5 (200) -> total = 4 + // Paginating from end: length = 4, offset = 1 -> end = 3, capped = 2 -> begin = 1 -> slice(1, 3) -> [r3, r4] + expect(result.total).toBe(4); + expect(result.logs.map(e => e.requestId)).toEqual(["r3", "r4"]); + + // Tail with limit and offset + const tailQuery = new URLSearchParams("provider=openai&status=2xx&tail=3&limit=2"); + const tailResult = queryRequestLogs(logs, tailQuery); + // Matches 4 items, tail=3 takes last 3: [r3, r4, r5] -> total = 3 + // limit=2 from end takes last 2: [r4, r5] + expect(tailResult.total).toBe(3); + expect(tailResult.logs.map(e => e.requestId)).toEqual(["r4", "r5"]); + + // Fast path: no filter + const noFilter = queryRequestLogs(logs, new URLSearchParams()); + expect(noFilter.total).toBe(6); + expect(noFilter.logs.length).toBe(6); + + // Empty array + expect(queryRequestLogs([], new URLSearchParams("provider=openai"))).toEqual({ total: 0, logs: [] }); + }); + test("deferred JSON logging preserves response service tier before final log", async () => { const entries: RequestLogEntry[] = []; const logCtx = { From a9feb9b32f59881d136538c9fbd5efba5ef3b512 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 02:58:51 +0900 Subject: [PATCH 09/19] fix(logs): preserve protocol mode in single-pass queries Include protocolMode in the one-pass predicate before tail/count/pagination. A combined-filter regression fails on the contributor port without this correction. --- src/server/request-log-filter.ts | 4 +++- structure/dashboard-and-usage.md | 2 ++ .../usage/request-log-protocol-trace.test.ts | 22 +++++++++++++++++++ 3 files changed, 27 insertions(+), 1 deletion(-) diff --git a/src/server/request-log-filter.ts b/src/server/request-log-filter.ts index 162af9ff840..60008eb7223 100644 --- a/src/server/request-log-filter.ts +++ b/src/server/request-log-filter.ts @@ -39,9 +39,10 @@ export function queryRequestLogs( // request failed over between pool accounts, a search for the account that finally served it // has to find that request, not only the account that first refused it. const account = params.get("account")?.trim(); + const protocolMode = params.get("protocolMode")?.trim().toLowerCase(); const status = params.get("status")?.trim().toLowerCase(); - const hasFilters = Boolean(provider || conversationId || model || account || status); + const hasFilters = Boolean(provider || conversationId || model || account || protocolMode || status); let filtered: RequestLogEntry[]; if (!hasFilters) { filtered = logs; @@ -53,6 +54,7 @@ export function queryRequestLogs( if (conversationId && !matchesLogConversationId(entry.conversationId, conversationId)) continue; if (model && entry.model !== model && !entry.attempts?.some(attempt => attempt.model === model)) continue; if (account && entry.accountLogLabel !== account && !entry.attempts?.some(attempt => attempt.accountLogLabel === account)) continue; + if (protocolMode && !matchesProtocolMode(entry, protocolMode)) continue; if (status && (statusClass === undefined ? String(entry.status) !== status : Math.floor(entry.status / 100) !== statusClass)) continue; filtered.push(entry); } diff --git a/structure/dashboard-and-usage.md b/structure/dashboard-and-usage.md index 93fb16fdca2..857b2d35f1f 100644 --- a/structure/dashboard-and-usage.md +++ b/structure/dashboard-and-usage.md @@ -565,6 +565,8 @@ labels collapse for reporting; configured provider names ending in `-main` remai Rows also carry the observed protocol path (`protocolTrace`), persisted in `usage.jsonl` and re-validated on read; the Logs list shows it as a text badge, the detail dialog as a section, and `src/server/request-log-filter.ts` owns the `/api/logs` query filters including `protocolMode`. +Its single-pass query applies provider, conversation, model, account, protocol mode and status +before `tail`, then reports the pre-pagination count alongside the offset/limit page. [Protocol Paths](data-planes/protocol-paths.md) owns its derivation. Request-history selectors longer than 130 characters persist as a prefix plus a digest of the complete diff --git a/tests/usage/request-log-protocol-trace.test.ts b/tests/usage/request-log-protocol-trace.test.ts index e9fb0592d1b..77426b49f5b 100644 --- a/tests/usage/request-log-protocol-trace.test.ts +++ b/tests/usage/request-log-protocol-trace.test.ts @@ -10,6 +10,7 @@ import { beginRequestAttempt, filterRequestLogs, filteredRequestLogCount, + queryRequestLogs, requestLogEntryFromPersistedUsage, type RequestLogContext, type RequestLogEntry, @@ -125,4 +126,25 @@ describe("protocolMode filter", () => { expect(ids("protocolMode=verified")).toEqual([]); expect(ids("")).toEqual(["bridge", "native", "blocked", "old"]); }); + + test("combined filters retain protocol mode through tail, count and pagination", () => { + const native = { ...bridgeTrace, mode: "native" as const, requestPath: ["chat", "chat"] as ProtocolTraceV1["requestPath"], responsePath: ["chat", "chat"] as ProtocolTraceV1["responsePath"] }; + const matching = (requestId: string, status = 200) => ({ + ...row(requestId, native), accountLogLabel: "account-a", status, + }); + const entries = [ + matching("first"), + matching("middle", 204), + matching("error", 500), + { ...matching("other-provider"), provider: "q" }, + { ...matching("other-mode"), protocolTrace: bridgeTrace }, + matching("last"), + ]; + const params = new URLSearchParams("provider=p&model=m&account=account-a&status=2xx&protocolMode=native&tail=2&limit=1&offset=1"); + const result = queryRequestLogs(entries, params); + expect(result.total).toBe(2); + expect(result.logs.map(entry => entry.requestId)).toEqual(["middle"]); + expect(filterRequestLogs(entries, params).map(entry => entry.requestId)).toEqual(["middle"]); + expect(filteredRequestLogCount(entries, params)).toBe(2); + }); }); From 9bbf17ade118eab1e0bdd19c3559060b3f79b233 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 03:36:09 +0900 Subject: [PATCH 10/19] fix(plugins): refuse ACL-bearing plugin paths on macOS Inspect the plugin file, directory, and resolved ancestors for ACL entries before importing code. Probe Linux ACLs when getfacl is available and document the fallback. Mac regressions cover a 0600 file, plugin directory, and ancestor ACL. --- .../src/content/docs/guides/local-plugins.md | 4 ++- src/plugins/loader.ts | 29 +++++++++++++++- structure/ops/plugins.md | 6 ++-- tests/lib/plugin-loader.test.ts | 33 +++++++++++++++++++ 4 files changed, 68 insertions(+), 4 deletions(-) diff --git a/docs-site/src/content/docs/guides/local-plugins.md b/docs-site/src/content/docs/guides/local-plugins.md index 0f735c93e73..5f30fa6d81e 100644 --- a/docs-site/src/content/docs/guides/local-plugins.md +++ b/docs-site/src/content/docs/guides/local-plugins.md @@ -29,7 +29,9 @@ Put plugin files in `plugins/` inside the opencodex home (`~/.opencodex/plugins/ symbolic links. Every directory above `plugins/`, up to `/`, must also be owned by you or root and not writable by group or others, unless it is sticky like `/tmp`. Fix permissions with `chmod go-w ~/.opencodex/plugins ~/.opencodex/plugins/*`; on systems whose default umask is - `002`, check the parent directories too. + `002`, check the parent directories too. On macOS, any ACL on the file or a path directory also + blocks loading, even if its mode is `0600`; inspect with `ls -le`. On Linux, extended ACLs are + checked when `getfacl` is installed. Without it, only owner and mode bits are verified. - On Windows automatic plugin loading is disabled until an ACL trust check is available. Restart the proxy after adding, changing or removing a plugin (`ocx service restart`, or stop and diff --git a/src/plugins/loader.ts b/src/plugins/loader.ts index da632437111..568c3fb68fb 100644 --- a/src/plugins/loader.ts +++ b/src/plugins/loader.ts @@ -21,6 +21,7 @@ import { lstatSync, readdirSync, realpathSync } from "node:fs"; import { basename, dirname, join } from "node:path"; import { pathToFileURL } from "node:url"; +import { spawnSync } from "node:child_process"; import { getConfigDir } from "../config/paths"; import { registerOptionalShutdownHook } from "../lib/optional-shutdown-hooks"; import { registerUpstreamRewriter, type UpstreamRewriter } from "./upstream-hooks"; @@ -57,6 +58,30 @@ export interface PluginLoadResult { const PLUGIN_EXTENSIONS = [".ts", ".js", ".mjs"]; const SETUP_TIMEOUT_MS = 5_000; +const ACL_PROBE_TIMEOUT_MS = 2_000; + +/** Refuse extended ACLs: mode bits alone cannot prove who can rewrite a plugin path. */ +function aclTrustError(path: string): string | null { + if (process.platform !== "darwin" && process.platform !== "linux") return null; + const mac = process.platform === "darwin"; + const result = spawnSync(mac ? "ls" : "getfacl", mac + ? ["-lebd", "--", path] + : ["-cp", "--", path], { + encoding: "utf8", + timeout: ACL_PROBE_TIMEOUT_MS, + maxBuffer: 64 * 1024, + env: { ...process.env, LC_ALL: "C" }, + }); + // getfacl is optional on Linux. Without it, the POSIX owner/mode checks below remain; + // the documented Linux ACL limitation must be visible to operators. + if (!mac && (result.error as NodeJS.ErrnoException | undefined)?.code === "ENOENT") return null; + if (result.error || result.status !== 0) return "access control list inspection failed"; + const lines = result.stdout.split("\n").slice(mac ? 1 : 0); + return lines.some(line => mac + ? /^\s*\d+:\s/.test(line) + : /^(?:user:[^:]+:|group:[^:]+:|mask::|default:)/.test(line)) + ? "has an access control list" : null; +} export function pluginDirectory(): string { return join(getConfigDir(), "plugins"); @@ -97,7 +122,7 @@ function trustError(path: string, kind: "file" | "directory"): string | null { const uid = process.getuid?.(); if (uid !== undefined && stats.uid !== uid) return "owned by another user"; if ((stats.mode & 0o022) !== 0) return "writable by group or others (chmod go-w)"; - return null; + return aclTrustError(path); } /** Null when the file is safe to execute, otherwise the reason it is refused. */ @@ -135,6 +160,8 @@ export function pluginAncestorsTrustError(realDir: string): string | null { if ((stats.mode & 0o022) !== 0 && (stats.mode & 0o1000) === 0) { return `${current} is writable by group or others`; } + const aclError = aclTrustError(current); + if (aclError) return aclError; const parent = dirname(current); if (parent === current) return null; current = parent; diff --git a/structure/ops/plugins.md b/structure/ops/plugins.md index 7629ed38cf5..ca04dd99ca5 100644 --- a/structure/ops/plugins.md +++ b/structure/ops/plugins.md @@ -18,8 +18,10 @@ or signs them. `config.json`. Every ancestor of the resolved plugin directory up to `/` must be owned by the user or root and not group/other-writable unless sticky (`pluginAncestorsTrustError`), so no other user can swap a checked path before it is imported; files are imported through the resolved directory. - Owner and mode checks are POSIX-only; Windows auto-loading is disabled until an ACL trust - check can enforce the same boundary. + On macOS, `ls -lebd` must show no ACL entries on the file, plugin directory, or any ancestor; + an inspection error refuses loading. Linux uses `getfacl` when installed and refuses extended + ACL entries or probe failures; without that utility, only owner/mode checks apply. Windows + auto-loading is disabled until an ACL trust check can enforce the same boundary. - A missing plugin directory means no plugins. Any other read failure (`EACCES`, `ENOTDIR`) is reported as a skipped `plugins directory` entry. - A plugin module default-exports `{ name?, setup(context) }`. An asynchronous `setup` has five diff --git a/tests/lib/plugin-loader.test.ts b/tests/lib/plugin-loader.test.ts index 7d2e02b4d0b..8bf1089644f 100644 --- a/tests/lib/plugin-loader.test.ts +++ b/tests/lib/plugin-loader.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeAll, beforeEach, expect, test } from "bun:test"; import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; +import { execFileSync } from "node:child_process"; import { resetOptionalShutdownHooksForTests, runOptionalShutdownHooks } from "../../src/lib/optional-shutdown-hooks"; import { loadOcxPlugins, pluginFileTrustError } from "../../src/plugins/loader"; import { @@ -107,6 +108,38 @@ test.skipIf(process.platform === "win32")("a group- or world-writable plugin is expect(hasUpstreamRewriters()).toBe(false); }); +test.skipIf(process.platform !== "darwin")("a mode-0600 plugin with an everyone-write ACL is refused", async () => { + const path = writePlugin("acl.ts", REDIRECT_PLUGIN, 0o600); + execFileSync("chmod", ["+a", "everyone allow write", path]); + expect(pluginFileTrustError(path)).toBe("has an access control list"); + const [result] = await loadOcxPlugins(dir); + expect(result?.error).toBe("file_untrusted"); + expect(hasUpstreamRewriters()).toBe(false); +}); + +test.skipIf(process.platform !== "darwin")("an ACL on an ancestor directory blocks plugin loading", async () => { + const parent = mkdtempSync(join(tmpdir(), "ocx-plugin-acl-parent-")); + const nested = join(parent, "plugins"); + try { + mkdirSync(nested); + writeFileSync(join(nested, "redirect.ts"), REDIRECT_PLUGIN); + execFileSync("chmod", ["+a", "everyone allow write", parent]); + const [result] = await loadOcxPlugins(nested); + expect(result?.error).toBe("ancestor_untrusted"); + expect(hasUpstreamRewriters()).toBe(false); + } finally { + rmSync(parent, { recursive: true, force: true }); + } +}); + +test.skipIf(process.platform !== "darwin")("an ACL on the plugin directory blocks plugin loading", async () => { + writePlugin("redirect.ts", REDIRECT_PLUGIN); + execFileSync("chmod", ["+a", "everyone allow write", dir]); + const [result] = await loadOcxPlugins(dir); + expect(result?.error).toBe("directory_untrusted"); + expect(hasUpstreamRewriters()).toBe(false); +}); + test.skipIf(process.platform === "win32")("a symbolic link is refused even when it points to a trusted file", async () => { const outside = mkdtempSync(join(tmpdir(), "ocx-plugin-target-")); try { From 124b8564c8e41210796ef1b892fbef89ad5c3569 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 03:36:34 +0900 Subject: [PATCH 11/19] fix(codebuddy): refuse credentialed config redirects Use manual redirect handling for the key-scoped roster fetch. A two-origin Bun regression first observed the synthetic X-API-Key at the redirect target and now confirms no target request occurs. --- .../src/content/docs/guides/providers.md | 1 + src/adapters/codebuddy/live-models.ts | 2 ++ structure/providers-and-adapters.md | 4 +++- tests/providers/codebuddy-live-models.test.ts | 24 +++++++++++++++++++ 4 files changed, 30 insertions(+), 1 deletion(-) diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 3f98b844a94..e8b9509a6be 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -890,6 +890,7 @@ OpenCodex provides official adapter support for Tencent Cloud's CodeBuddy Code C - CN: [CodeBuddy CN API Keys](https://copilot.tencent.com/profile/keys) - **Region Isolation:** `codebuddy` and `codebuddy-cn` use separate canonical endpoints (`https://www.codebuddy.ai` and `https://www.codebuddy.cn`) and isolated child environments (`CODEBUDDY_INTERNET_ENVIRONMENT=public` vs `internal`). Credentials are strictly region-scoped and never exchanged across environments. Overriding the canonical base URL fails closed. - **Model Discovery:** the proxy requests the CodeBuddy product configuration (`GET {baseUrl}/v3/config`) with the configured key as the `X-API-Key` header, and the roster in that answer is the authoritative roster of discovered models: it is the key's own account configuration, so it is proven to belong to the key — a different or wrong key answers the anonymous envelope with no roster instead of another account's models. The authenticated roster is the same list the CLI prints for `--model` (the "Currently supported" line of a signed-in CLI), can differ from the static manifest bundled with the CLI, and the vendor default selectors (`default` for CN, `default-model` for Global) never appear in it but remain callable: the catalog retains them during live discovery and on every fallback path. On start/sync the proxy binds the cached roster to an irreversible fingerprint of the configured key, so a key switch never observes a roster cached for the previous key, and degrades to the stale provider/key-fingerprint-scoped cache, then to the static seed in `src/providers/codebuddy-models.ts`, when the key does not authenticate or the request fails. Discovery failure logs contain only a category and HTTP status, without the gateway's message or a raw transport exception. + The credentialed discovery request does not follow redirects; a 3xx response degrades the roster without forwarding the key to another origin. - **Tool Ownership and the Tool Bridge:** The CLI is always spawned with `--tools ""` and `--strict-mcp-config`, so it has no built-in or user-configured tools of its own. When a request carries a Codex tool catalog, the provider arms a capture-only MCP bridge: the validated catalog and MCP config are written to a private temp dir, the CLI is launched with `--mcp-config` and an exact `--allowedTools` list, and the `system/init` frame must report exactly that bridge server as connected or the turn fails closed. The bridge advertises the Codex tools and captures proposed calls but never executes anything: a completed tool-call batch is returned as `function_call` items (names mapped back to the request's wire names, at most 16 calls per assistant message), the process tree is terminated at `message_stop`, and the external Codex client alone performs approval, sandboxing, and execution. Tool results come back as the next request's input, and the conversation continues. Requests without tools keep the plain text-and-reasoning shape. If the CLI writes an unquoted DSML `calls` control line followed by a `functions.*` invoke control line into text or reasoning, OpenCodex refuses the turn instead of forwarding the scaffold or interpreting it as an executable call. DSML discussed or quoted in prose, inline code, fenced code, or source examples remains ordinary answer text. - **Tool Choice Enforcement:** When a request specifies `tool_choice: "required"` or selects a specific named tool, the bridge expects a tool call from the model. If the CLI completes the turn with plain text instead of capturing a tool call, OpenCodex fails closed with a 502 `tool_call_required` error rather than returning an invalid text completion. - **Governance Status:** Whether routing this vendor automation surface behind a proxy for a third-party agent satisfies CodeBuddy's acceptable-use terms is an open question flagged for maintainer security review (see the governance note in the provider registry entry). Treat this provider as pending that review, and keep the tool bridge's ownership boundary in mind: the nested CLI advertises tools but never executes them, and approval, sandboxing, and execution remain with the external Codex client. diff --git a/src/adapters/codebuddy/live-models.ts b/src/adapters/codebuddy/live-models.ts index 94649ac62b1..c2fc6b40a5d 100644 --- a/src/adapters/codebuddy/live-models.ts +++ b/src/adapters/codebuddy/live-models.ts @@ -145,6 +145,8 @@ export async function fetchCodeBuddyModels( try { response = await (deps.fetch ?? fetch)(url, { headers, + // A redirect target must never receive this key, even when Bun preserves custom headers. + redirect: "manual", signal: AbortSignal.timeout(deps.timeoutMs ?? 8_000), }); } catch (error) { diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 601d4089f63..df201fb6a4c 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -141,7 +141,9 @@ OAuth bearer token. CodeBuddy discovery in `src/adapters/codebuddy/live-models.ts` reads the roster scoped to the configured key. Its failure result carries only a category and optional HTTP status; untrusted -gateway messages and transport exceptions do not reach catalog warnings. +gateway messages and transport exceptions do not reach catalog warnings. The credentialed +config fetch uses manual redirect handling; any 3xx is a failed discovery and cannot forward +`X-API-Key` to a second origin. Provider request pacing in `src/providers/request-pacing.ts` combines start intervals with optional `maxConcurrentRequests` limits. Provider capacity is shared across models; exact-model limits diff --git a/tests/providers/codebuddy-live-models.test.ts b/tests/providers/codebuddy-live-models.test.ts index 5e5125f2fac..303f1ea0fa0 100644 --- a/tests/providers/codebuddy-live-models.test.ts +++ b/tests/providers/codebuddy-live-models.test.ts @@ -126,6 +126,30 @@ describe("CodeBuddy live model fetch", () => { expect(JSON.stringify(result)).not.toContain(marker); }); + test("a cross-origin 302 never forwards the configured key", async () => { + let secondHits = 0; + let secondKey: string | null = null; + const second = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + secondHits += 1; + secondKey = req.headers.get("x-api-key"); + return Response.json(authenticatedEnvelope()); + } }); + const first = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch() { + return new Response(null, { status: 302, + headers: { location: `http://127.0.0.1:${second.port}/v3/config` } }); + } }); + try { + const profile = { ...CODEBUDDY_CN_PROFILE, canonicalBaseUrl: `http://127.0.0.1:${first.port}` }; + const result = await fetchCodeBuddyModels(profile, "synthetic-codebuddy-key"); + expect(secondKey).toBeNull(); + expect(secondHits).toBe(0); + expect(result).toEqual({ ok: false, error: "http", status: 302 }); + } finally { + await first.stop(true); + await second.stop(true); + } + }); + test("a body that is not JSON fails closed as invalid output", async () => { const fetchLike = (async () => new Response("gateway error page", { status: 200 })) as typeof fetch; const result = await fetchCodeBuddyModels(CODEBUDDY_CN_PROFILE, "cb-cn-key", { fetch: fetchLike }); From a99a88acd20cd6079a8910b3bf2e8a69c5fa907b Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 03:39:38 +0900 Subject: [PATCH 12/19] fix(pacing): count Cursor concurrency by active turn Keep the pre-acquired runTurn lease until the turn owner finishes instead of transferring it to RunSSE body completion. This permits overlapping RunSSE and BidiAppend within one admitted turn while another turn waits; document the exception across configuration locales and pin it with a red/green regression. --- .../fr/reference/configuration/providers.md | 2 +- .../ja/reference/configuration/providers.md | 2 +- .../ko/reference/configuration/providers.md | 2 +- .../docs/reference/configuration/providers.md | 2 +- .../ru/reference/configuration/providers.md | 2 +- .../reference/configuration/providers.md | 2 +- .../reference/configuration/providers.md | 2 +- src/server/responses/fetch-helpers.ts | 7 +++-- structure/providers-and-adapters.md | 14 +++++---- tests/usage/request-pacing.test.ts | 29 +++++++++++++++++++ 10 files changed, 49 insertions(+), 15 deletions(-) diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index 0d95bbc1bc0..d93d432cb63 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -86,7 +86,7 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | --- | --- | --- | | `adapter` | `string` | L'un des `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (ou alias `azure`). | | `baseUrl` | `string` | URL de base de l'API en amont. La plupart des points de terminaison fixes intégrés ignorent une valeur incompatible ; les préréglages de clés protégés contre les collisions préservent une ancienne destination personnalisée portant le même nom. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. Avec une limite de concurrence, un tour Responses WebSocket canonique utilise HTTP/SSE afin de libérer la capacité à la fin, en cas d’erreur ou d’annulation du corps de réponse. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. Avec une limite de concurrence, un tour Responses WebSocket canonique utilise HTTP/SSE afin de libérer la capacité à la fin, en cas d’erreur ou d’annulation du corps de réponse. Pour les adaptateurs `runTurn`, dont Cursor, la limite compte les tours actifs plutôt que les envois physiques : RunSSE et BidiAppend peuvent se chevaucher dans un même tour, tandis qu’un autre tour attend. Les envois suivants respectent toujours les intervalles de démarrage. | | `responsesPath?` | `string` | Chemin de ressource relatif pour les requêtes d'authentification par clé `openai-responses`. Il doit commencer par `/` et ne contenir aucun schéma, requête ou fragment. | | `chatCompletionsPath?` | `string` | Chemin de ressource relatif pour les requêtes `openai-chat`, miroir de `responsesPath` et soumis aux mêmes règles de forme. Nécessaire lorsqu'un même service en amont sert Chat Completions et Responses sous des préfixes différents : un override wire par modèle change l'adaptateur sans toucher `baseUrl`, donc sans ce réglage une requête Chat activée serait envoyée vers la base Responses. L'exemple fourni est Z.AI. | | `upstreamWebsocket?` | `boolean` | Active le transport Responses WebSocket en amont pour les requêtes `openai-responses` (désactivé par défaut). N'est honoré que pour l'amont first-party `https://api.openai.com/v1` ; les points de terminaison des fournisseurs personnalisés utilisent toujours HTTP/SSE borné, car Bun ne peut pas appliquer de limite de taille aux messages WebSocket entrants avant d'avoir alloué le message complet. Pour le fournisseur canonique ChatGPT `openai`, l'omettre conserve le WebSocket en amont sur les tours éligibles, `false` envoie les tours en streaming via HTTP/SSE, et `true` est refusé ; avec `false`, le pilotage et l'injection natifs en cours de tour sont indisponibles. Ce champ est indépendant du réglage `websockets` côté client et ne change ni le point de terminaison ni les identifiants. Une base HTTP reste en SSE ; les chemins qui ne sont pas Responses et les requêtes `openai-chat` restent en HTTP. | diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 5a83763c5eb..e3f7b972a3d 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -80,7 +80,7 @@ account を削除しても mapping は保持され、同じ id を再追加す | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai` (または別名 `azure`) のいずれか。 | | `baseUrl` | `string` |アップストリーム API のベース URL。ほとんどの組み込み固定エンドポイントは不一致を無視します。衝突安全キー プリセットは、古い同じ名前のカスタム宛先を保持します。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP と明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。同時実行数の上限がある場合、標準の Responses WebSocket ターンは応答本文の完了・エラー・キャンセル時に枠を解放できるよう HTTP/SSE を使います。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP と明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。同時実行数の上限がある場合、標準の Responses WebSocket ターンは応答本文の完了・エラー・キャンセル時に枠を解放できるよう HTTP/SSE を使います。 Cursor を含む `runTurn` アダプターでは、上限は物理送信数ではなく進行中のターン数を数えます。同じターンの RunSSE と BidiAppend は重なり得ますが、別のターンは待機します。後続送信にも開始間隔は適用されます。 | | `responsesPath?` | `string` |キー認証 `openai-responses` リクエストの相対リソース パス。 `/` で始まり、スキーム、クエリ、またはフラグメントが含まれていない必要があります。 | | `chatCompletionsPath?` | `string` | `openai-chat` リクエストの相対リソース パス。 `responsesPath` の対となる設定で、同じ形式ルールが適用されます。1つのアップストリームが Chat Completions と Responses を異なるプレフィックスで提供する場合に必要です。モデルごとの wire override はアダプターのみを変更し `baseUrl` は変更しないため、この設定がないと有効化された Chat リクエストが Responses ベースへ送信されます。同梱例は Z.AI です。 | | `upstreamWebsocket?` | `boolean` | `openai-responses` リクエストで使用するアップストリーム Responses WebSocket トランスポート（既定値は無効）。ファーストパーティの `https://api.openai.com/v1` アップストリームでのみ有効です。カスタムプロバイダーのエンドポイントは常に制限付き HTTP/SSE を使用します。Bun はメッセージ全体を確保する前に受信 WebSocket メッセージのサイズ上限を適用できないためです。正規の ChatGPT `openai` プロバイダーでは、省略すると対象となるターンでアップストリーム WebSocket を使用し、`false` はストリーミングのターンを HTTP/SSE で送信し、`true` は拒否されます。`false` の間はネイティブのターン途中のステアリングとインジェクションを利用できません。このフィールドはクライアント側の `websockets` 設定とは独立しており、エンドポイントと認証情報のどちらも変更しません。HTTP のベース URL は SSE のままとなり、Responses 以外のパスと `openai-chat` リクエストは HTTP を使用します。 | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index 9dc6bbff29f..f96ebbfd3d2 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -80,7 +80,7 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | --- | --- | --- | | `adapter` | `string` | `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` 중 하나이며, `azure`는 별칭입니다. | | `baseUrl` | `string` | 상위 API 기본 URL입니다. 대부분의 내장 고정 엔드포인트는 불일치를 무시합니다. 충돌 안전 키 프리셋은 같은 이름의 이전 사용자 지정 목적지를 보존합니다. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP 및 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. 동시성 제한이 있으면 표준 Responses WebSocket 턴은 응답 본문의 완료·오류·취소 때 점유 슬롯을 해제할 수 있도록 HTTP/SSE를 사용합니다. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP 및 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. 동시성 제한이 있으면 표준 Responses WebSocket 턴은 응답 본문의 완료·오류·취소 때 점유 슬롯을 해제할 수 있도록 HTTP/SSE를 사용합니다. Cursor를 포함한 `runTurn` 어댑터에서는 물리적 전송 수가 아니라 진행 중인 턴 수를 제한합니다. 같은 턴의 RunSSE와 BidiAppend는 겹칠 수 있지만 다른 턴은 대기합니다. 후속 전송에도 시작 간격은 적용됩니다. | | `responsesPath?` | `string` | 키 인증 `openai-responses` 요청의 상대 리소스 경로입니다. 반드시 `/`로 시작해야 하며 스킴, query, fragment를 포함하면 안 됩니다. | | `chatCompletionsPath?` | `string` | `openai-chat` 요청의 상대 리소스 경로로, `responsesPath`와 동일한 형식 규칙이 적용되는 대응 항목입니다. 하나의 업스트림이 Chat Completions와 Responses를 서로 다른 접두사로 제공할 때 필요합니다. 모델별 wire override는 어댑터만 바꾸고 `baseUrl`은 그대로 두므로, 이 설정이 없으면 옵트인된 Chat 요청이 Responses base로 전송됩니다. Z.AI가 제공되는 예시입니다. | | `upstreamWebsocket?` | `boolean` | `openai-responses` 요청에 대한 업스트림 Responses WebSocket 전송을 선택적으로 활성화합니다(기본값 `false`). 퍼스트파티 `https://api.openai.com/v1` 업스트림에서만 적용되며, 사용자 지정 공급자 엔드포인트는 항상 제한된 HTTP/SSE를 사용합니다. Bun은 전체 메시지를 할당하기 전에는 수신 WebSocket 메시지 크기 제한을 적용할 수 없기 때문입니다. 정식 ChatGPT `openai` 공급자에서는 생략하면 대상 턴에서 업스트림 WebSocket을 사용하고, `false`는 스트리밍 턴을 HTTP/SSE로 전송하며, `true`는 거부됩니다. `false`이면 네이티브 턴 중 스티어링과 주입을 사용할 수 없습니다. 이 필드는 클라이언트 측 `websockets` 설정과 독립적이며 엔드포인트와 자격 증명을 변경하지 않습니다. HTTP 기본 URL은 SSE를 유지하고, Responses가 아닌 경로와 `openai-chat` 요청은 HTTP를 사용합니다. | diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 57144f924e4..5ad66176829 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -198,7 +198,7 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `baseUrl` | `string` | Upstream API base URL. Most built-in fixed endpoints ignore a mismatch; collision-safe key presets preserve an older same-named custom destination. | | `proxy?` | `string \| null` | Per-provider egress route. Omit it to inherit the global proxy decision; use `"direct"` or `null` to force direct egress; or provide an absolute `http://`, `https://`, `socks5://`, or `socks5h://` proxy URL. An empty string is rejected. | | `noProxy?` | `string \| string[]` | Destinations this provider reaches directly, using `NO_PROXY` host-pattern syntax. A match bypasses both this provider's own proxy and an inherited global proxy. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. A concurrency-capped canonical Responses WebSocket turn uses HTTP/SSE so its lease can be released when the response body completes, errors, or is cancelled. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. A concurrency-capped canonical Responses WebSocket turn uses HTTP/SSE so its lease can be released when the response body completes, errors, or is cancelled. For `runTurn` adapters, including Cursor, the cap counts active turns rather than physical sends: RunSSE and BidiAppend may overlap within one turn, while another turn waits. Follow-up sends still obey start intervals. | | `upstreamHttpVersion?` | `"auto" \| "http1.1" \| "h1" \| "http2" \| "h2"` | Pin the HTTP version used for upstream requests to this provider. Defaults to `auto`, which lets Bun negotiate. An explicit pin requires an HTTPS target and fails locally when it cannot be honored. Set `http1.1` when a provider's HTTP/2 SSE stream stalls instead of delivering events — the symptom is a long-running streaming request that produces nothing and eventually times out. For Cursor, `http1.1`/`h1` selects its `RunSSE` + `BidiAppend` compatibility transport for inference and also pins live model discovery. Management `POST`/`PATCH` accept `null` to clear it back to `auto`. | | `responsesPath?` | `string` | Relative resource path for key-auth `openai-responses` requests. It must start with `/` and contain no scheme, query, or fragment. | | `chatCompletionsPath?` | `string` | Relative resource path for `openai-chat` requests, the mirror of `responsesPath` and subject to the same shape rules. Needed when one upstream serves Chat Completions and Responses under different prefixes: a per-model wire override changes the adapter and leaves `baseUrl` alone, so without this an opted-in Chat request would be sent to the Responses base. Z.AI is the shipped example. | diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index 62798fc78e6..273d6a5dd13 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -93,7 +93,7 @@ cross-route credential fallback не существует. Строки API GPT- | --- | --- | --- | | `adapter` | `string` | Один из `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (или alias `azure`). | | `baseUrl` | `string` | Базовый URL API upstream'а. Большинство built-in fixed-endpoint'ов игнорируют несовпадение; collision-safe key-preset'ы сохраняют старый custom destination с тем же именем. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP и явные вызовы адаптеров `fetchResponse`/`runTurn`. При ограничении параллелизма канонический запрос Responses WebSocket использует HTTP/SSE, чтобы освободить слот после завершения, ошибки или отмены тела ответа. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP и явные вызовы адаптеров `fetchResponse`/`runTurn`. При ограничении параллелизма канонический запрос Responses WebSocket использует HTTP/SSE, чтобы освободить слот после завершения, ошибки или отмены тела ответа. Для адаптеров `runTurn`, включая Cursor, лимит считает активные ходы, а не физические отправки: RunSSE и BidiAppend могут пересекаться внутри одного хода, пока следующий ход ждёт. Последующие отправки по-прежнему соблюдают интервалы запуска. | | `responsesPath?` | `string` | Relative resource path для key-auth запросов `openai-responses`. Должен начинаться с `/` и не может содержать scheme, query или fragment. | | `chatCompletionsPath?` | `string` | Relative resource path для запросов `openai-chat`, зеркало `responsesPath` с теми же правилами формы. Нужен, когда один upstream обслуживает Chat Completions и Responses под разными префиксами: per-model wire override меняет адаптер и не трогает `baseUrl`, поэтому без него включённый Chat-запрос ушёл бы в Responses base. Поставляемый пример — Z.AI. | | `upstreamWebsocket?` | `boolean` | Необязательный upstream Responses WebSocket для запросов `openai-responses` (по умолчанию `false`). Учитывается только для first-party upstream `https://api.openai.com/v1`; конечные точки пользовательских провайдеров всегда используют ограниченный HTTP/SSE, поскольку Bun не может применить ограничение размера входящего сообщения WebSocket до выделения памяти под всё сообщение. У канонического провайдера ChatGPT `openai` пропуск сохраняет upstream WebSocket для подходящих ходов, `false` отправляет потоковые ходы по HTTP/SSE, а `true` отклоняется; при `false` нативное управление и внедрение в середине хода недоступны. Это поле не зависит от клиентской настройки `websockets` и не меняет ни конечную точку, ни учётные данные. Для HTTP остаётся SSE; пути, не относящиеся к Responses, и запросы `openai-chat` остаются на HTTP. | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index 1cb588d964d..a7464eab8e9 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -80,7 +80,7 @@ selector，而不是分配一个新名称。 | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或别名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API 基础 URL。大多数内置固定端点会忽略不匹配的值；具备冲突安全键的预设会保留一个更早、同名的自定义目标。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP 以及显式适配器 `fetchResponse`/`runTurn` 调用。配置并发上限时，标准 Responses WebSocket 请求改用 HTTP/SSE，以便在响应体完成、出错或取消时释放并发名额。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP 以及显式适配器 `fetchResponse`/`runTurn` 调用。配置并发上限时，标准 Responses WebSocket 请求改用 HTTP/SSE，以便在响应体完成、出错或取消时释放并发名额。 对包括 Cursor 在内的 `runTurn` 适配器，并发上限计算进行中的回合数，而非物理发送数：同一回合内的 RunSSE 和 BidiAppend 可以重叠，其他回合仍须等待。后续发送仍遵守启动间隔。 | | `responsesPath?` | `string` | 用于 key-auth `openai-responses` 请求的相对资源路径。必须以 `/` 开头，且不能包含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | 用于 `openai-chat` 请求的相对资源路径，是 `responsesPath` 的对应项，适用相同的路径规则。当同一上游以不同前缀提供 Chat Completions 和 Responses 时需要此配置：按模型的 wire override 只更换适配器而不改动 `baseUrl`，否则已启用的 Chat 请求会被发送到 Responses base。随附示例为 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 为 `openai-responses` 请求选择性启用上游 Responses WebSocket 传输（默认 `false`）。仅对第一方 `https://api.openai.com/v1` 上游生效；自定义提供者端点始终使用有界 HTTP/SSE，因为 Bun 无法在分配完整消息之前对入站 WebSocket 消息实施大小限制。对于规范 ChatGPT `openai` 提供商，省略该字段会在符合条件的轮次使用上游 WebSocket，`false` 通过 HTTP/SSE 发送流式轮次，`true` 会被拒绝；设为 `false` 时，原生轮次中操控与注入不可用。该字段独立于客户端侧的 `websockets` 设置，且不改变端点或凭据。普通 HTTP 仍使用 SSE；非 Responses 路径和 `openai-chat` 请求仍使用 HTTP。 | diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index 780e40020c1..7e0aa7777cb 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -60,7 +60,7 @@ ocx models provider openrouter on | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或別名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API base URL。多數內建固定端點忽略不符；碰撞安全的金鑰預設保留較舊的同名自訂目的地。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。設定並行上限時，標準 Responses WebSocket 請求改用 HTTP/SSE，以便在回應本文完成、出錯或取消時釋放並行名額。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。設定並行上限時，標準 Responses WebSocket 請求改用 HTTP/SSE，以便在回應本文完成、出錯或取消時釋放並行名額。 對包含 Cursor 的 `runTurn` 適配器，並行上限計算進行中的回合數，而非實體傳送數：同一回合內的 RunSSE 與 BidiAppend 可重疊，其他回合仍須等待。後續傳送仍遵守啟動間隔。 | | `responsesPath?` | `string` | Key-auth `openai-responses` 請求的相對資源路徑。必須以 `/` 開頭且不含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | `openai-chat` 請求的相對資源路徑，為 `responsesPath` 的對應項，適用相同的路徑規則。當同一上游以不同前綴提供 Chat Completions 與 Responses 時需要此設定：按模型的 wire override 只更換適配器而不改動 `baseUrl`，否則已啟用的 Chat 請求會送往 Responses base。隨附範例為 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 為 `openai-responses` 請求選用上游 Responses WebSocket 傳輸（預設 `false`）。僅對第一方 `https://api.openai.com/v1` 上游生效；自訂供應商端點一律使用有界 HTTP/SSE，因為 Bun 無法在配置完整訊息之前對傳入 WebSocket 訊息套用大小限制。對於規範 ChatGPT `openai` 供應商，省略此欄位會在符合條件的回合使用上游 WebSocket，`false` 會以 HTTP/SSE 傳送串流回合，`true` 會被拒絕；設為 `false` 時，原生回合中操控與注入無法使用。此欄位獨立於用戶端 `websockets` 設定，且不會變更端點或認證資料。一般 HTTP 仍使用 SSE；非 Responses 路徑與 `openai-chat` 請求仍使用 HTTP。 | diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index 83f16a83b18..c6cfd2525e5 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -217,7 +217,7 @@ export interface ProviderFetchOptions { /** One pacing slot was acquired immediately before this fetch wrapper was created. */ pacingSlotAcquired?: boolean; pacingSlot?: ProviderRequestSlot; - /** A turn transport may issue interval-paced follow-up sends while its lease is held. */ + /** A runTurn transport holds one concurrency lease across overlapping physical sends. */ turnScopedPacing?: boolean; /** Captured selected-account observer, attached before the native WS send. */ onCodexWsQuota?: CodexWsQuotaObserver; @@ -332,7 +332,10 @@ export function providerFetch( if (!options.pacingSlot && requestPacingMaxConcurrentRequests(provider, options.modelId) > 0) { throw new Error("A pre-acquired concurrency slot must be passed to providerFetch"); } - return options.pacingSlot; + // Cursor RunSSE and BidiAppend can overlap. The runTurn owner releases this slot when + // the whole turn finishes; transferring it to RunSSE's body would admit another turn + // while a BidiAppend from this one is still active. + return options.turnScopedPacing ? undefined : options.pacingSlot; } if (options.providerName) { return waitForProviderRequestSlot(options.providerName, provider, options.modelId, signal, diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index df201fb6a4c..af5f003adf7 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -149,12 +149,14 @@ Provider request pacing in `src/providers/request-pacing.ts` combines start inte `maxConcurrentRequests` limits. Provider capacity is shared across models; exact-model limits apply in addition to that capacity. Admission reserves both counters atomically, and eligible sibling models may bypass a saturated model lane. Releases are idempotent, wake queued requests, -and retain interval deadlines. A lease follows each physical HTTP send through response-body -completion, error, or cancellation; failed dispatch and active abort also return it. Unconsumed -or inactive bodies are cancelled after a bounded deadline so a dropped response cannot hold -capacity indefinitely. A capped canonical Codex WebSocket turn uses HTTP/SSE because the socket -has no response-body lifecycle to return the lease. Capacity waits use the same bounded queue -and retryable queue-overload errors as interval waits. +and retain interval deadlines. Ordinary HTTP leases follow each physical send through body +completion, error, or cancellation; failed dispatch and active abort also return them. Unconsumed +or inactive bodies are cancelled after a bounded deadline. For `runTurn` adapters, including +Cursor, one lease spans the whole turn: RunSSE and BidiAppend may overlap inside it, while other +turns wait at the cap. Follow-up sends still obey start intervals. The turn owner returns its lease +after `runTurn` settles, so RunSSE body completion cannot admit another turn early. A capped +canonical Codex WebSocket turn uses HTTP/SSE because the socket has no response-body lifecycle. +Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. ## TypeSafe JEV decision provider diff --git a/tests/usage/request-pacing.test.ts b/tests/usage/request-pacing.test.ts index 6a7280f2ffa..5183100b743 100644 --- a/tests/usage/request-pacing.test.ts +++ b/tests/usage/request-pacing.test.ts @@ -325,6 +325,35 @@ describe("provider request pacing queue", () => { }); describe("provider request concurrency", () => { + test("Cursor counts overlapping RunSSE and BidiAppend as one held turn", async () => { + let sends = 0; + const fetchImpl = Object.assign(async () => { + sends += 1; + return new Response("ok"); + }, { preconnect() {} }) as typeof globalThis.fetch; + const configured = { ...provider({ enabled: true, maxConcurrentRequests: 1 }), fetch: fetchImpl }; + const turnSlot = await waitForProviderRequestSlot("cursor", configured, "model-a"); + const turnFetch = providerFetch(configured, undefined, { + providerName: "cursor", modelId: "model-a", pacingSlotAcquired: true, + pacingSlot: turnSlot, turnScopedPacing: true, + }); + const otherTurnFetch = providerFetch(configured, undefined, { providerName: "cursor", modelId: "model-a" }); + try { + expect(await (await turnFetch("https://example.test/RunSSE")).text()).toBe("ok"); + expect(providerRequestPacingStatus("cursor", configured).inFlight).toBe(1); + expect(await (await turnFetch("https://example.test/BidiAppend")).text()).toBe("ok"); + expect(sends).toBe(2); + const otherTurn = otherTurnFetch("https://example.test/other", { signal: AbortSignal.timeout(500) }); + await Bun.sleep(0); + expect(providerRequestPacingStatus("cursor", configured).queued).toBe(1); + turnSlot.release(); + expect(await (await otherTurn).text()).toBe("ok"); + expect(sends).toBe(3); + } finally { + turnSlot.release(); + } + }); + test("an active body holds capacity until cancellation", async () => { let sends = 0; const fetchImpl = Object.assign(async () => { From aa7add2c159f5bb611ae8a80d024725eb366f738 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 03:43:03 +0900 Subject: [PATCH 13/19] test(compaction): verify fallback Request strips inbound credentials The fixture now carries inbound Authorization and chatgpt-account-id, and a wrapper observes the actual child Request at fallback dispatch. Removing the two production header deletions makes this test fail; restoring them makes it pass. --- .../responses-compaction-recovery.test.ts | 30 +++++++++++++++++-- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/tests/responses/responses-compaction-recovery.test.ts b/tests/responses/responses-compaction-recovery.test.ts index a60f3af0105..b001c3919c5 100644 --- a/tests/responses/responses-compaction-recovery.test.ts +++ b/tests/responses/responses-compaction-recovery.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; import { getDefaultConfig } from "../../src/config"; import { handleResponses, handleResponsesCompact } from "../../src/server/responses"; +import { runWithCompactionRecovery } from "../../src/server/responses/compaction-recovery"; import { decodeCompactionSummary } from "../../src/responses/compaction"; import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; import { createTranslatorBudget } from "../../src/lib/translator-budget"; @@ -14,7 +15,7 @@ const originalFetch = globalThis.fetch; const sourceError: AdapterEvent = { type: "error", status: 400, code: "invalid_argument", message: "Source rejected compact fixture" }; let sourceEvents: AdapterEvent[]; let fallbackEvents: AdapterEvent[]; -let calls: Array<{ model: string; parsed: OcxParsedRequest }>; +let calls: Array<{ model: string; parsed: OcxParsedRequest; headers: Headers }>; let releaseSpend: (() => void) | undefined; let restoreFactory: (() => void) | undefined; let restoreChatFactory: (() => void) | undefined; @@ -45,7 +46,10 @@ function body(stream = false, compact = true): Record { function request(payload = body(), path = "responses", signal?: AbortSignal): Request { return new Request(`http://localhost/v1/${path}`, { - method: "POST", headers: { "content-type": "application/json", session_id: "recovery-fixture" }, + method: "POST", headers: { + "content-type": "application/json", session_id: "recovery-fixture", + authorization: "Bearer inbound-fixture", "chatgpt-account-id": "inbound-account-fixture", + }, body: JSON.stringify(payload), signal, }); } @@ -73,7 +77,7 @@ beforeEach(() => { return; } incoming.onPhysicalSend?.({ ordinal: 1 }); - calls.push({ model: parsed.modelId, parsed: structuredClone(parsed) }); + calls.push({ model: parsed.modelId, parsed: structuredClone(parsed), headers: new Headers(incoming.headers) }); const isSource = context.providerId === "source"; for (const event of isSource ? sourceEvents : fallbackEvents) emit(event); if (isSource) abortOnSource?.abort(); @@ -124,6 +128,8 @@ describe("routed compaction emergency integration", () => { expect(summary).toContain("ALPHA-729"); expect(summary).toContain("Latest goal: finish the report"); expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); + expect(calls[1]!.headers.has("authorization")).toBe(false); + expect(calls[1]!.headers.has("chatgpt-account-id")).toBe(false); expect(calls[1]!.parsed.options.maxOutputTokens).toBe(512); expect(calls[1]!.parsed.context.tools).toBeUndefined(); expect(completed).toEqual(["source/swe-2"]); @@ -144,6 +150,24 @@ describe("routed compaction emergency integration", () => { expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); }); + test("the actual fallback Request strips inbound auth and account headers", async () => { + const original = request(); + let fallbackHeaders: Headers | undefined; + const response = await runWithCompactionRecovery( + original, settings(), { model: "", provider: "" }, + { translatorBudget: createTranslatorBudget(), sendBudget: createRequestExecutionBudget() }, + async (sent, config, log, options) => { + if (sent !== original) fallbackHeaders = new Headers(sent.headers); + return handleResponses(sent, config, log, { ...options, compactionRecoveryAttempted: true }); + }, + ); + expect((await response.json()).status).toBe("completed"); + expect(calls.map(call => call.model)).toEqual(["swe-2", "rescue"]); + expect(fallbackHeaders).toBeDefined(); + expect(fallbackHeaders!.has("authorization")).toBe(false); + expect(fallbackHeaders!.has("chatgpt-account-id")).toBe(false); + }); + test("a failed fallback returns and logs the original failure", async () => { fallbackEvents = [{ type: "error", status: 503, code: "server_is_overloaded", message: "Emergency overloaded fixture" }]; const log: RequestLogContext = { model: "", provider: "" }; From 05586aa302dcf347cfd23a18f9c427945ed87918 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 04:13:55 +0900 Subject: [PATCH 14/19] test(plugins): exercise Linux ACL trust on Ubuntu CI Exercise named write ACLs on a 0600 plugin and its directory with setfacl, plus a clean-file loading control. The ubuntu-latest image includes acl; a CI guard fails if getfacl/setfacl disappear. A recorded getfacl output regression proves the parser on macOS too. --- src/plugins/loader.ts | 10 ++++-- tests/lib/plugin-loader.test.ts | 63 +++++++++++++++++++++++++++++++-- 2 files changed, 67 insertions(+), 6 deletions(-) diff --git a/src/plugins/loader.ts b/src/plugins/loader.ts index 568c3fb68fb..ce6bc5c84c6 100644 --- a/src/plugins/loader.ts +++ b/src/plugins/loader.ts @@ -121,8 +121,12 @@ function trustError(path: string, kind: "file" | "directory"): string | null { if (process.platform === "win32") return null; const uid = process.getuid?.(); if (uid !== undefined && stats.uid !== uid) return "owned by another user"; + // A POSIX ACL write mask may also set group-write mode bits. Inspect the ACL first so + // the extended grant is explicitly refused even when the mode check would also refuse it. + const aclError = aclTrustError(path); + if (aclError) return aclError; if ((stats.mode & 0o022) !== 0) return "writable by group or others (chmod go-w)"; - return aclTrustError(path); + return null; } /** Null when the file is safe to execute, otherwise the reason it is refused. */ @@ -157,11 +161,11 @@ export function pluginAncestorsTrustError(realDir: string): string | null { return "ancestor inspection failed"; } if (uid !== undefined && stats.uid !== uid && stats.uid !== 0) return `${current} is owned by another user`; + const aclError = aclTrustError(current); + if (aclError) return aclError; if ((stats.mode & 0o022) !== 0 && (stats.mode & 0o1000) === 0) { return `${current} is writable by group or others`; } - const aclError = aclTrustError(current); - if (aclError) return aclError; const parent = dirname(current); if (parent === current) return null; current = parent; diff --git a/tests/lib/plugin-loader.test.ts b/tests/lib/plugin-loader.test.ts index 8bf1089644f..76ea99422cd 100644 --- a/tests/lib/plugin-loader.test.ts +++ b/tests/lib/plugin-loader.test.ts @@ -1,10 +1,10 @@ import { afterEach, beforeAll, beforeEach, expect, test } from "bun:test"; import { chmodSync, mkdirSync, mkdtempSync, realpathSync, rmSync, statSync, symlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; -import { dirname, join } from "node:path"; -import { execFileSync } from "node:child_process"; +import { delimiter, dirname, join } from "node:path"; +import { execFileSync, spawnSync } from "node:child_process"; import { resetOptionalShutdownHooksForTests, runOptionalShutdownHooks } from "../../src/lib/optional-shutdown-hooks"; -import { loadOcxPlugins, pluginFileTrustError } from "../../src/plugins/loader"; +import { loadOcxPlugins, pluginDirectoryTrustError, pluginFileTrustError } from "../../src/plugins/loader"; import { hasUpstreamRewriters, resetUpstreamRewritersForTests, @@ -12,6 +12,8 @@ import { } from "../../src/plugins/upstream-hooks"; let dir: string; +const linuxAclToolsAvailable = process.platform === "linux" + && ["getfacl", "setfacl"].every(command => spawnSync(command, ["--version"], { stdio: "ignore" }).status === 0); // The loader refuses a plugin directory with a group- or other-writable, non-sticky ancestor. // The test runner nests per-process temp roots and creates them with the caller's umask, which is @@ -140,6 +142,61 @@ test.skipIf(process.platform !== "darwin")("an ACL on the plugin directory block expect(hasUpstreamRewriters()).toBe(false); }); +test.skipIf(process.platform === "win32")("recorded Linux getfacl output rejects a named ACL entry", () => { + const file = writePlugin("redirect.ts", REDIRECT_PLUGIN, 0o600); + // POSIX ACL write masks can surface as group-write mode bits after setfacl. + chmodSync(file, 0o660); + const fakeGetfacl = join(dir, "getfacl"); + writeFileSync(fakeGetfacl, '#!/bin/sh\nprintf "user::rw-\\nuser:nobody:rw-\\ngroup::---\\nmask::rw-\\nother::---\\n"\n'); + chmodSync(fakeGetfacl, 0o700); + const previousPath = process.env.PATH; + const platform = process.platform; + try { + process.env.PATH = `${dir}${delimiter}${previousPath ?? ""}`; + Object.defineProperty(process, "platform", { value: "linux", configurable: true }); + expect(pluginFileTrustError(file)).toBe("has an access control list"); + } finally { + Object.defineProperty(process, "platform", { value: platform, configurable: true }); + if (previousPath === undefined) delete process.env.PATH; + else process.env.PATH = previousPath; + } +}); + +test.skipIf(process.platform !== "linux" || process.env.GITHUB_ACTIONS !== "true")( + "ubuntu-latest exposes getfacl and setfacl for the ACL regression", () => { + expect(linuxAclToolsAvailable).toBe(true); + }, +); + +test.skipIf(process.platform !== "linux" || !linuxAclToolsAvailable)("a 0600 Linux plugin with a named write ACL is refused", async () => { + const file = writePlugin("redirect.ts", REDIRECT_PLUGIN, 0o600); + expect(statSync(file).mode & 0o777).toBe(0o600); + execFileSync("setfacl", ["-m", "u:65534:rw", file]); + expect(pluginFileTrustError(file)).toBe("has an access control list"); + const [result] = await loadOcxPlugins(dir); + expect(result?.error).toBe("file_untrusted"); + expect(hasUpstreamRewriters()).toBe(false); +}); + +test.skipIf(process.platform !== "linux" || !linuxAclToolsAvailable)("a Linux plugin directory with a named write ACL is refused", async () => { + writePlugin("redirect.ts", REDIRECT_PLUGIN); + chmodSync(dir, 0o700); + expect(statSync(dir).mode & 0o777).toBe(0o700); + execFileSync("setfacl", ["-m", "u:65534:rwx", dir]); + expect(pluginDirectoryTrustError(dir)).toBe("has an access control list"); + const [result] = await loadOcxPlugins(dir); + expect(result?.error).toBe("directory_untrusted"); + expect(hasUpstreamRewriters()).toBe(false); +}); + +test.skipIf(process.platform !== "linux" || !linuxAclToolsAvailable)("a Linux plugin without an extended ACL still loads", async () => { + const file = writePlugin("redirect.ts", REDIRECT_PLUGIN, 0o600); + expect(pluginFileTrustError(file)).toBeNull(); + const [result] = await loadOcxPlugins(dir); + expect(result?.loaded).toBe(true); + expect(hasUpstreamRewriters()).toBe(true); +}); + test.skipIf(process.platform === "win32")("a symbolic link is refused even when it points to a trusted file", async () => { const outside = mkdtempSync(join(tmpdir(), "ocx-plugin-target-")); try { From 318332e7c446b1655698d05dc9f28939d9938559 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 05:09:14 +0900 Subject: [PATCH 15/19] fix(compaction): return source Kiro lease before emergency admission The compaction child shares the source account-load holder. Release and clear its completed source lease before fallback dispatch so a cap-one Kiro account can admit the child and higher caps cannot leak the overwritten lease. Regression covers caps one and two plus cancellation. --- src/server/responses/compaction-recovery.ts | 7 ++ structure/transports/responses-failover.md | 8 ++- .../kiro/kiro-leased-responses.test.ts | 67 ++++++++++++++++++- 3 files changed, 78 insertions(+), 4 deletions(-) diff --git a/src/server/responses/compaction-recovery.ts b/src/server/responses/compaction-recovery.ts index d04cd08d8c3..5503518b833 100644 --- a/src/server/responses/compaction-recovery.ts +++ b/src/server/responses/compaction-recovery.ts @@ -252,6 +252,13 @@ export async function runWithCompactionRecovery( try { const child = new Request(req.url, { method: "POST", headers, body: JSON.stringify(nextBody), signal: req.signal }); linkRequestSessionLane(req, child); + // The source send is finished. The child reuses this account-load holder and may + // select the same Kiro account, so return its lease before child admission can + // acquire and replace the holder's reference. + if (options.accountLoad?.lease) { + options.accountLoad.lease.release(); + options.accountLoad.lease = null; + } fallback = await dispatch(child, config, logCtx, { ...options, compactionRecoveryAttempted: true, compactionRecoveryPermit: recoveryPermit, compactionRoutingOverride: { sourceModel: originalModel }, diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 1bb102f949c..67e512b608e 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -14,8 +14,12 @@ with adapter dispatch, and only additional retries draw from the remainder. Adap before response projection. Native encrypted compaction, uploaded files, stored continuations, and policy/combo routes are excluded. Emergency output must contain one readable portable compaction item; recent original user messages are retained verbatim, and recovery failure keeps -the original failure. `tests/responses/responses-compaction-recovery-policy.test.ts` and -`tests/responses/responses-compaction-recovery.test.ts` pin these boundaries. +the original failure. A source Kiro account lease is returned before the emergency child is +admitted: the child shares its holder and may select the same account, so replacing that holder +without returning the source lease would block cap-one fallback and leak capacity at higher caps. +`tests/responses/responses-compaction-recovery-policy.test.ts`, +`tests/responses/responses-compaction-recovery.test.ts` and +`tests/providers/kiro/kiro-leased-responses.test.ts` pin these boundaries. Retry, replay, and combo failover on the Responses data plane: upstream reset retry, the ambiguous-resend gate and replay boundary, combo quota fallback and commit boundaries, compaction diff --git a/tests/providers/kiro/kiro-leased-responses.test.ts b/tests/providers/kiro/kiro-leased-responses.test.ts index d218ed01e97..f0612f511d8 100644 --- a/tests/providers/kiro/kiro-leased-responses.test.ts +++ b/tests/providers/kiro/kiro-leased-responses.test.ts @@ -44,10 +44,10 @@ function config(): OcxConfig { models: ["claude-sonnet-4.5"], oauthAccountFailover: { maxConcurrentPerAccount: 1 }, } } } as OcxConfig; } -async function seed(count = 1) { +async function seed(count = 1, accountPrefix = "load-account") { for (let i = 0; i < count; i++) await saveCredential("kiro", { access: `load-access-${i}`, refresh: `load-refresh-${i}`, expires: Date.now() + 3_600_000, - accountId: `load-account-${i}`, source: "oauth", kiro: { profileArn: `arn:aws:codewhisperer:us-east-1:123456789012:profile/${i}`, + accountId: `${accountPrefix}-${i}`, source: "oauth", kiro: { profileArn: `arn:aws:codewhisperer:us-east-1:123456789012:profile/${i}`, apiRegion: "us-east-1" }, }, { addAccount: true }); const ids = getAccountSet("kiro")!.accounts.map(row => row.id); @@ -65,6 +65,69 @@ function answer() { { headers: { "content-type": "application/vnd.amazon.eventstream" } }); } +function compactionConfig(cap: number): OcxConfig { + const cfg = config(); + cfg.providers.kiro!.models = ["claude-sonnet-4.5", "claude-haiku-4.5"]; + cfg.providers.kiro!.oauthAccountFailover = { maxConcurrentPerAccount: cap }; + cfg.compactionRecovery = { enabled: true, model: "kiro/claude-haiku-4.5" }; + return cfg; +} + +function compactionRequest(signal?: AbortSignal): Request { + return new Request("http://localhost/v1/responses", { method: "POST", + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "kiro/claude-sonnet-4.5", store: false, stream: false, + input: [{ type: "message", role: "user", content: "Keep this goal." }, { type: "compaction_trigger" }] }), signal }); +} + +test.each([1, 2])("Kiro compaction fallback returns the source lease before next admission (cap=%s)", async cap => { + const [id] = await seed(1, `compact-cap-${cap}`); + let sends = 0; + globalThis.fetch = (async () => { + sends += 1; + return sends === 1 + ? Response.json({ error: { code: "server_error", message: "source failed" } }, { status: 500 }) + : answer(); + }) as typeof fetch; + const response = await handleResponses(compactionRequest(), compactionConfig(cap), { model: "claude-sonnet-4.5", provider: "kiro" }); + await response.text(); + expect(response.status).toBe(200); + expect(sends).toBe(2); + expect(accountInFlight("kiro", id!)).toBe(0); +}); + +test("cancelled Kiro compaction fallback releases both account leases", async () => { + const [id] = await seed(1, "compact-cancel"); + const controller = new AbortController(); + let fallbackStarted!: () => void; + const started = new Promise(resolve => { fallbackStarted = resolve; }); + let sends = 0; + globalThis.fetch = (async (_input, init) => { + sends += 1; + if (sends === 1) return Response.json({ error: { code: "server_error" } }, { status: 500 }); + fallbackStarted(); + return new Promise((_resolve, reject) => { + const signal = init?.signal; + const abort = () => reject(signal?.reason ?? new DOMException("Aborted", "AbortError")); + if (signal?.aborted) abort(); + else signal?.addEventListener("abort", abort, { once: true }); + }); + }) as typeof fetch; + const pending = handleResponses(compactionRequest(controller.signal), compactionConfig(2), + { model: "claude-sonnet-4.5", provider: "kiro" }, { abortSignal: controller.signal }); + try { + await started; + controller.abort(); + const response = await pending; + await response.text(); + expect(response.status).toBe(499); + expect(sends).toBe(2); + expect(accountInFlight("kiro", id!)).toBe(0); + } finally { + controller.abort(); + } +}); + test("a full selected account waits then returns 503 account_capacity without a store write", async () => { const [id] = await seed(); held.push((await acquireAccountLease("kiro", id!))!); From 3d1d6d1c8cb32f0d53c0b22e5d8871c1ef1e4f22 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 05:11:27 +0900 Subject: [PATCH 16/19] fix(test-layout): keep merged registry under file-size guard The latest dev model-catalog registrations put layout.json at the 2000-line threshold. Compact the unchanged keepAtRoot array to retain every mapping while restoring file-size headroom. --- scripts/test-layout/layout.json | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 1e9b2ddf10a..d6ce6486825 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1,13 +1,7 @@ { "version": 1, "root": "tests", - "keepAtRoot": [ - "preload.ts", - "fake-codex-server.ts", - "tsconfig.doctor-service-memory-contract.json", - "test-layout.test.ts", - "test-layout-tooling.test.ts" - ], + "keepAtRoot": ["preload.ts", "fake-codex-server.ts", "tsconfig.doctor-service-memory-contract.json", "test-layout.test.ts", "test-layout-tooling.test.ts"], "domains": { "providers": { "match": [ From 6ea89bb94bb66a1e9aeea4d54c7ab5ab6af192ec Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 06:10:11 +0900 Subject: [PATCH 17/19] test(routing): assert final auth on response send, not quota prime --- tests/routing/subagent-fallback-handle-responses.test.ts | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/routing/subagent-fallback-handle-responses.test.ts b/tests/routing/subagent-fallback-handle-responses.test.ts index cc7eca13c0a..c7005dba67e 100644 --- a/tests/routing/subagent-fallback-handle-responses.test.ts +++ b/tests/routing/subagent-fallback-handle-responses.test.ts @@ -874,9 +874,13 @@ describe("native fallback account preview", () => { expect(entitlementCalls).toBe(2); // Final auth is authoritative and sees the second snapshot, not the preview snapshot. expect(finalAuth).toMatchObject({ kind: "pool", accountId: "pool-a" }); + if (finalAuth?.kind === "pool") expect(finalAuth.accessToken).toBe("pool-a_token"); expect((logCtx as unknown as Record).subagentModelFallbackTo) .toBe("gpt-daybreak-blue-latest"); - expect(capture.auths[0]).toContain("pool-a_token"); + // A pool quota prime can reach /wham/usage before or after the response send. + // Only the response wire proves that final-route auth used the second snapshot. + expect(capture.urls.flatMap((url, index) => url.endsWith("/codex/responses") + ? [capture.auths[index]] : [])).toEqual(["Bearer pool-a_token"]); }); test("pending preview entitlement errors release admission after preserving the original path", async () => { From 71d3dbc9a72fa396c3f04f06364cb665a41c5287 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 06:10:11 +0900 Subject: [PATCH 18/19] fix(plugins): use native macOS ACL tools and honor Windows disabled loading --- src/plugins/loader.ts | 4 +++- tests/lib/plugin-loader.test.ts | 39 +++++++++++++++++++++++---------- 2 files changed, 30 insertions(+), 13 deletions(-) diff --git a/src/plugins/loader.ts b/src/plugins/loader.ts index ce6bc5c84c6..a51157bc38f 100644 --- a/src/plugins/loader.ts +++ b/src/plugins/loader.ts @@ -64,7 +64,9 @@ const ACL_PROBE_TIMEOUT_MS = 2_000; function aclTrustError(path: string): string | null { if (process.platform !== "darwin" && process.platform !== "linux") return null; const mac = process.platform === "darwin"; - const result = spawnSync(mac ? "ls" : "getfacl", mac + // CI and operator PATHs may put GNU coreutils ahead of the macOS tool. + // GNU ls does not support -e, so pin the OS ACL inspector. + const result = spawnSync(mac ? "/bin/ls" : "getfacl", mac ? ["-lebd", "--", path] : ["-cp", "--", path], { encoding: "utf8", diff --git a/tests/lib/plugin-loader.test.ts b/tests/lib/plugin-loader.test.ts index 76ea99422cd..213a65ad2cb 100644 --- a/tests/lib/plugin-loader.test.ts +++ b/tests/lib/plugin-loader.test.ts @@ -60,12 +60,12 @@ export default { }; `; -test("a missing plugin directory loads nothing", async () => { +test.skipIf(process.platform === "win32")("a missing plugin directory loads nothing", async () => { expect(await loadOcxPlugins(join(dir, "absent"))).toEqual([]); expect(hasUpstreamRewriters()).toBe(false); }); -test("a valid plugin registers its upstream rewriter", async () => { +test.skipIf(process.platform === "win32")("a valid plugin registers its upstream rewriter", async () => { writePlugin("redirect.ts", REDIRECT_PLUGIN); const results = await loadOcxPlugins(dir); expect(results.map(result => [result.name, result.loaded])).toEqual([["redirect", true]]); @@ -93,7 +93,7 @@ test("Windows does not auto-load plugins without an ACL trust check", async () = } }); -test("plugin setup exceptions expose only a bounded category", async () => { +test.skipIf(process.platform === "win32")("plugin setup exceptions expose only a bounded category", async () => { const marker = "private plugin error marker"; writePlugin("throws.ts", `export default { setup() { throw new Error("${marker}"); } };`); const results = await loadOcxPlugins(dir); @@ -110,9 +110,24 @@ test.skipIf(process.platform === "win32")("a group- or world-writable plugin is expect(hasUpstreamRewriters()).toBe(false); }); +test.skipIf(process.platform !== "darwin")("ACL trust uses macOS ls when PATH contains incompatible ls", () => { + const file = writePlugin("redirect.ts", REDIRECT_PLUGIN); + const fakeLs = join(dir, "ls"); + writeFileSync(fakeLs, "#!/bin/sh\nexit 2\n"); + chmodSync(fakeLs, 0o700); + const previousPath = process.env.PATH; + try { + process.env.PATH = `${dir}${delimiter}${previousPath ?? ""}`; + expect(pluginFileTrustError(file)).toBeNull(); + } finally { + if (previousPath === undefined) delete process.env.PATH; + else process.env.PATH = previousPath; + } +}); + test.skipIf(process.platform !== "darwin")("a mode-0600 plugin with an everyone-write ACL is refused", async () => { const path = writePlugin("acl.ts", REDIRECT_PLUGIN, 0o600); - execFileSync("chmod", ["+a", "everyone allow write", path]); + execFileSync("/bin/chmod", ["+a", "everyone allow write", path]); expect(pluginFileTrustError(path)).toBe("has an access control list"); const [result] = await loadOcxPlugins(dir); expect(result?.error).toBe("file_untrusted"); @@ -125,7 +140,7 @@ test.skipIf(process.platform !== "darwin")("an ACL on an ancestor directory bloc try { mkdirSync(nested); writeFileSync(join(nested, "redirect.ts"), REDIRECT_PLUGIN); - execFileSync("chmod", ["+a", "everyone allow write", parent]); + execFileSync("/bin/chmod", ["+a", "everyone allow write", parent]); const [result] = await loadOcxPlugins(nested); expect(result?.error).toBe("ancestor_untrusted"); expect(hasUpstreamRewriters()).toBe(false); @@ -136,7 +151,7 @@ test.skipIf(process.platform !== "darwin")("an ACL on an ancestor directory bloc test.skipIf(process.platform !== "darwin")("an ACL on the plugin directory blocks plugin loading", async () => { writePlugin("redirect.ts", REDIRECT_PLUGIN); - execFileSync("chmod", ["+a", "everyone allow write", dir]); + execFileSync("/bin/chmod", ["+a", "everyone allow write", dir]); const [result] = await loadOcxPlugins(dir); expect(result?.error).toBe("directory_untrusted"); expect(hasUpstreamRewriters()).toBe(false); @@ -240,7 +255,7 @@ test.skipIf(process.platform === "win32")("a plugin directory under a group-writ expect(hasUpstreamRewriters()).toBe(false); }); -test("a wrong export shape or a throwing setup is skipped and leaves no hooks behind", async () => { +test.skipIf(process.platform === "win32")("a wrong export shape or a throwing setup is skipped and leaves no hooks behind", async () => { writePlugin("a-shape.ts", "export default { name: 'shape' };"); writePlugin("b-throws.ts", ` export default { @@ -261,7 +276,7 @@ export default { expect(rewriteUpstream("https://api.example.com/v1/x", undefined, "http").url).toBe("http://127.0.0.1:8787/v1/x"); }); -test("a plugin path that cannot be read is reported, not treated as empty", async () => { +test.skipIf(process.platform === "win32")("a plugin path that cannot be read is reported, not treated as empty", async () => { const notADirectory = writePlugin("file-not-dir", "x"); const results = await loadOcxPlugins(notADirectory); expect(results).toHaveLength(1); @@ -270,7 +285,7 @@ test("a plugin path that cannot be read is reported, not treated as empty", asyn expect(results[0]?.error).toBe("directory_read_failed"); }); -test("two plugins with the same name keep separate shutdown teardowns", async () => { +test.skipIf(process.platform === "win32")("two plugins with the same name keep separate shutdown teardowns", async () => { const ran: string[] = []; (globalThis as Record)["__ocxTeardownLog"] = ran; const source = (tag: string) => ` @@ -293,7 +308,7 @@ export default { } }); -test("one plugin can register several shutdown teardowns", async () => { +test.skipIf(process.platform === "win32")("one plugin can register several shutdown teardowns", async () => { const ran: string[] = []; (globalThis as Record)["__ocxTeardownLog"] = ran; writePlugin("multi.ts", ` @@ -315,7 +330,7 @@ export default { } }); -test("a setup that resumes after its deadline cannot leave registrations behind", async () => { +test.skipIf(process.platform === "win32")("a setup that resumes after its deadline cannot leave registrations behind", async () => { writePlugin("slow.ts", ` export default { name: "slow", @@ -338,7 +353,7 @@ export default { expect(hasUpstreamRewriters()).toBe(false); }); -test("hidden, underscore-prefixed and declaration files are ignored", async () => { +test.skipIf(process.platform === "win32")("hidden, underscore-prefixed and declaration files are ignored", async () => { writePlugin(".hidden.ts", REDIRECT_PLUGIN); writePlugin("_draft.ts", REDIRECT_PLUGIN); writePlugin("types.d.ts", "export {};"); From ee9f18ffdd5405af5c0bec0af3a929366c3366b9 Mon Sep 17 00:00:00 2001 From: JUN Date: Sun, 27 Sep 2026 06:10:11 +0900 Subject: [PATCH 19/19] test(windows): preserve sibling recycle process-exit cleanup --- tests/clients/client-link-runtime.test.ts | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/tests/clients/client-link-runtime.test.ts b/tests/clients/client-link-runtime.test.ts index 196650a3aae..4e0fa6bfd54 100644 --- a/tests/clients/client-link-runtime.test.ts +++ b/tests/clients/client-link-runtime.test.ts @@ -131,7 +131,15 @@ test("an ended link recycles a connected sibling after listener cleanup", async if (child) await child.exited; if (replacementPid !== null) { try { process.kill(replacementPid, "SIGTERM"); } catch { /* already exited */ } - await waitFor(() => existsSync(join(home, "runtime-port.json")) ? null : true, "replacement shutdown"); + await waitFor(() => { + try { return process.kill(replacementPid, 0) ? null : true; } + catch { return true; } + }, "replacement process exit"); + // Windows SIGTERM is TerminateProcess, so no signal/exit handler removes the runtime + // record. The fixture cleanup owns that stale file after proving the process is gone. + if (process.platform !== "win32") { + await waitFor(() => existsSync(join(home, "runtime-port.json")) ? null : true, "replacement runtime cleanup"); + } } if (priorHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = priorHome;