fix: bug-PR merge train batch 7 (multi-account OAuth 429 budget with a per-request ceiling)

devlog/_plan/260926_bug_train_6/010_roadmap.md

@@ -0,0 +1,25 @@
+# Bug-PR merge train after batch 6 — roadmap
+
+Batch 6 landed as #5918 (`76b26a0881`). Open bug-labelled PRs re-queried at that head:
+#5921, #5920, #5916, #5915, #5914, #5911, #5831, #5800, #5782, #5539, #5497, #4222.
+
+## Decisions
+
+| PR | Decision | Reason |
+|---|---|---|
+| #5914, #5882, #5849 | Close as landed | Carried in #5918. Close each with the same thank-you note earlier batches used (author credit, merge SHA, what changed on top). |
+| #5916 (OAuth 429 send budget, #5880) | **Carry in batch 7 with a fix** | Real starvation bug and meaningful tests (4 cases red on `dev`, 86 focused pass). The independent security review failed on one point: the allowance scales with roster size with no fixed ceiling, so one request can fan out to 3 × N sends. Batch 7 adds a hard per-request ceiling that roster size cannot raise, then re-runs the security review. |
+| #5497 (FastWire tier authority for relays) | Evaluate in batch 7 P | Bug-labelled, 357 lines, but it adds a provider config field and fails hygiene. Carry only if the failure is mechanical and the field is opt-in with no default change; otherwise leave it. |
+| #5831 (main-lock recovery from two-window WHAM) | NEEDS_HUMAN | @Ingwannu holds approval for an owner decision on whether an omitted short window counts as proof no window exists. That is a policy call for the owner. |
+| #5911, #5915 (Meta Muse OAuth continuations) | Leave | Two overlapping drafts on the same OAuth surface; #5911 is the maintainer's own active draft with an ADR. Consolidation is theirs to decide. |
+| #5539 (effort wire mapper on unpinned routes) | Evaluate a narrowed carry in batch 7 P | The failure is real: strict upstreams answer `400 Invalid option` for `minimal`/`ultra` sent to a provider with no configured ladder. The native Chat half reverses tests that deliberately preserve the caller's spelling on unpinned routes, so it stays out. The Responses half (`mapRoutedResponsesReasoningEffort` for unconfigured providers) touches no existing assertion; carry it alone if focused tests show no other behavior change. |
+| #5800 (Claude Agent SDK harness) | Leave | Adds a runtime dependency (`package.json`, `bun.lock`) and replaces a provider; needs dependency and maintainer security review beyond a bug train. |
+| #5782 (Windows manual stops + bridge) | Leave | 9.3k lines, two feature lines, conflicting with `dev`. |
+| #4222 (side-chat cache) | Leave | Experimental feature behind a setting, not a bug fix. |
+| #5920, #5921 (desktop quota rows, widget reload) | Leave | Opened by the owner account minutes ago with their own devlog plan; another task owns them. |
+
+## Next cycles
+
+- wp3 — batch 7: `codex/bug-train-7` from `dev`; carry #5916 plus the ceiling fix; decide #5497 and the narrowed #5539 at P. Security review of the
+  final diff by an independent reviewer before merge. Exact-head CI, then `--admin --match-head-commit`.
+- Closing the loop: re-query open bug PRs and confirm every one left open has a row above.

devlog/_plan/260926_bug_train_6/020_batch7.md

@@ -0,0 +1,45 @@
+# Batch 7 — plan
+
+Previous D (wp2): roadmap locked in `010_roadmap.md`; direction: batch 7 carries #5916 with a fixed per-request ceiling
+and decides #5497 and a narrowed #5539 here.
+
+## P decisions on the open rows
+
+- #5497 — leave. It adds a public provider field (`responseTierAuthoritative`) with docs in three locales and new cost
+  and usage-log handling (the canonical ChatGPT exception is kept); that is a product/config review, not a merge-train carry.
+- #5539 (Responses half) — leave. For an unconfigured `openai-responses` provider (API-key OpenAI included) it would fold
+  `minimal` to `low`, which changes requests for OpenAI models that accept `minimal`. The strict-gateway 400 is
+  fixable today by declaring the provider's `reasoningEfforts` ladder.
+
+## Carry
+
+- #5916 (@codingbooo, closes #5880): squash onto `codex/bug-train-7` from `dev` `76b26a0881`, author + `Co-authored-by`.
+
+## Integration fix (security review blocker)
+
+The generic OAuth send allowance is `accounts × TRANSIENT_RETRY_MAX_ATTEMPTS` and the hop limit is `accounts - 1`, both
+from an uncapped roster snapshot. Add `GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST = 6` in
+`src/oauth/generic-account-failover.ts` and apply it in two places:
+
+1. `src/server/responses/request-transport.ts`: `budgetAccounts = Math.min(genericRosterSize, MAX)`;
+   `genericFailoverLimit = Math.max(GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST, budgetAccounts - 1)`;
+   `expandInferenceOAuthSendBudget(options.sendBudget, budgetAccounts)`.
+2. `src/server/inference/context.ts` `expandInferenceOAuthSendBudget`: clamp `accounts` to the same constant, so any
+   future caller cannot raise the ceiling either.
+
+Result: the default ingress OAuth ceiling is 6 × 3 = 18 physical sends per request however many accounts are enrolled,
+versus 4 before #5916. Explicitly supplied caller budgets are left unchanged.
+The PR's 4- and 5-account cases keep their expectations.
+
+Regression: in `tests/server/server-google-antigravity-oauth-429-budget.test.ts` (or a sibling if the file-size cap binds),
+eight accounts with transient 429 everywhere: exactly 18 sends across exactly the first 6 accounts, then terminal 429.
+Must fail without the clamp (24 sends / 8 accounts).
+
+Docs/structure: update the #5916 text in `structure/transports/responses-failover.md`, `responses-spend.md` and
+`docs-site/.../providers.md` to state the ceiling.
+
+## Check
+
+tsc, structure:check, privacy:scan, the #5916 test files plus `tests/oauth/generic-oauth-failover.test.ts`,
+`tests/oauth/oauth-account-attribution.test.ts`, layout and file-size guards. Independent security re-review of the final
+diff. Exact-head hosted CI, then `--admin --match-head-commit`; close #5916 with the batch note.

docs-site/src/content/docs/reference/configuration/providers.md

@@ -865,10 +865,17 @@ The Codex pool and the Anthropic pool are excluded and keep their own rotation;
 changes neither. A provider with a single stored account is a strict no-op, and no cooldown is
 recorded for it.
 
-On a 429 the failed account is cooled using `Retry-After` when present (capped at 15 minutes)
-or a default backoff, and the request is replayed on the next eligible account, up to three
-rotations per request. An account flagged for reauthentication is never selected. Cooldowns are
-process-local, so a restart forgets them.
+Before dispatch, generic OAuth snapshots the eligible roster. On a 429 the failed account is cooled
+using `Retry-After` when present (capped at 15 minutes) or a default backoff, and the request is
+replayed on the next account selected from the live roster. The stable rotation ceiling is
+`max(3, min(eligibleCount, 6) - 1)` per request; live selection still filters cooldowns, and an account
+flagged for reauthentication is never selected. Cooldowns are process-local, so a restart forgets
+them.
+
+When at least two accounts are eligible, the ingress-owned default send allowance covers up to three
+sends per eligible account, counting at most six accounts, so one request
+makes no more than 18 sends however many accounts are enrolled. A single eligible account keeps the existing base allowance of three and
+total allowance of four. Explicit caller ceilings and combo scopes keep their existing limits.
 
 Rotation carries the alternate account's **full** credential snapshot, not just its bearer, so a
 provider that pairs routing metadata with its token — Antigravity's Cloud Code Assist project id,

scripts/test-layout/layout.json

@@ -1634,6 +1634,7 @@
     "server-combo-reasoning-replay-eligibility.test.ts": "server",
     "server-combo-zero-output-failover.test.ts": "server",
     "server-google-antigravity-oauth-401-replay.test.ts": "server",
+    "server-google-antigravity-oauth-429-budget.test.ts": "server",
     "server-gui-bundle-freshness.test.ts": "server",
     "server-images-bodyless-content-length.test.ts": "server",
     "server-images.test.ts": "server",

src/adapters/google-http.ts

@@ -14,10 +14,10 @@ import {
   retryBackoffDelayMs,
   sleepWithAbort,
   SendBudgetExhaustedError,
+  TRANSIENT_RETRY_MAX_ATTEMPTS,
   isConnectionResetError,
 } from "../lib/upstream-retry";
 
-const GOOGLE_RETRY_ATTEMPTS = 3;
 const GOOGLE_RETRY_BASE_MS = 250;
 const GOOGLE_RETRY_MAX_MS = 2_000;
 
@@ -60,7 +60,7 @@ export async function fetchGoogleWithRetry(
   let retryDelayMs = 0;
   let sendClass: SendClass = "transient";
   let recovery: AttemptRecoveryKind | undefined;
-  for (let attempt = 0; attempt < GOOGLE_RETRY_ATTEMPTS; attempt++) {
+  for (let attempt = 0; attempt < TRANSIENT_RETRY_MAX_ATTEMPTS; attempt++) {
     if (ctx.abortSignal?.aborted) throw abortError(ctx.abortSignal);
     try {
       const res = await send({ url: activeRequest.url, sendClass, recovery,
@@ -105,7 +105,7 @@ export async function fetchGoogleWithRetry(
           continue;
         }
       }
-      if (!retryableGoogleStatus(res.status) || attempt === GOOGLE_RETRY_ATTEMPTS - 1) {
+      if (!retryableGoogleStatus(res.status) || attempt === TRANSIENT_RETRY_MAX_ATTEMPTS - 1) {
         return ctx.returnRawErrors ? res : normalizeFinalGoogleError(label, res, ctx.abortSignal);
       }
       // A 429 may be a transient rate limit (retry) or hard quota exhaustion (do NOT retry —
@@ -141,7 +141,7 @@ export async function fetchGoogleWithRetry(
         throw err;
       }
       lastError = err;
-      if (attempt === GOOGLE_RETRY_ATTEMPTS - 1) throw err;
+      if (attempt === TRANSIENT_RETRY_MAX_ATTEMPTS - 1) throw err;
       sendClass = "transient";
       recovery = isConnectionResetError(err) ? "connection-reset" : undefined;
       retryDelayMs = retryBackoffDelayMs(attempt, {

src/oauth/generic-account-failover.ts

@@ -40,6 +40,14 @@ import type { OcxConfig, OcxProviderConfig } from "../types";
 /** Cap same-request rotations so a short Retry-After cannot spin. Mirrors the Anthropic bound. */
 export const GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST = 3;
 
+/**
+ * Most accounts one request may fund and rotate through, whatever the roster size. Each funded
+ * account gets the normal transient ladder, so this fixes the default ingress ceiling at
+ * 6 × TRANSIENT_RETRY_MAX_ATTEMPTS physical sends: enrolling more accounts cannot turn one request
+ * into a pool-wide 429 storm.
+ */
+export const GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST = 6;
+
 const DEFAULT_COOLDOWN_MS = 60_000;
 
 /**

src/server/inference/context.ts

@@ -1,7 +1,16 @@
-import { createRequestExecutionBudget, type RequestExecutionBudget } from "../../lib/request-execution-budget";
+import { CODEX_TEXT_GUARDED_BUDGET_POLICY, createRequestExecutionBudget, type RequestExecutionBudget } from "../../lib/request-execution-budget";
+import { TRANSIENT_RETRY_MAX_ATTEMPTS, type TransientSendBudget } from "../../lib/upstream-retry";
+import { GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST } from "../../oauth/generic-account-failover";
 import type { RequestLogContext } from "../request-log";
 import { attachRequestSpendTracker } from "../responses/request-spend";
 
+// Only ingress owns an expandable policy. Exact caller budgets and combo-derived scopes
+// never enter this map, even when their policy happens to equal the default profile.
+const ingressPolicies = new WeakMap<TransientSendBudget, {
+  maxTotalModelSends: number;
+  baseSendAllowance: number;
+}>();
+
 /**
  * The one construction of an ingress-owned send budget: the default guarded policy, no logical
  * request id, and this request's spend tracker as the send observer. Attaching the tracker
@@ -11,5 +20,24 @@ export function createInferenceSendBudget(
   req: Pick<Request, "headers">,
   logCtx: RequestLogContext,
 ): RequestExecutionBudget {
-  return createRequestExecutionBudget(undefined, undefined, attachRequestSpendTracker(req, logCtx));
+  const policy = { ...CODEX_TEXT_GUARDED_BUDGET_POLICY };
+  const budget = createRequestExecutionBudget(policy, undefined, attachRequestSpendTracker(req, logCtx));
+  ingressPolicies.set(budget, policy);
+  return budget;
+}
+
+/**
+ * Fund each account's normal transient ladder once, before the first physical send. The account
+ * count is clamped here as well as at the caller, so no caller can raise the ceiling past
+ * GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST × TRANSIENT_RETRY_MAX_ATTEMPTS.
+ */
+export function expandInferenceOAuthSendBudget(budget: TransientSendBudget | undefined, accounts: number): void {
+  if (!budget) return;
+  const policy = ingressPolicies.get(budget);
+  if (!policy || budget.used !== 0) return;
+  ingressPolicies.delete(budget);
+  if (accounts < 2 || !Number.isSafeInteger(accounts)) return;
+  const sends = Math.min(accounts, GENERIC_OAUTH_MAX_ACCOUNTS_PER_REQUEST) * TRANSIENT_RETRY_MAX_ATTEMPTS;
+  policy.baseSendAllowance = Math.max(policy.baseSendAllowance, sends);
+  policy.maxTotalModelSends = Math.max(policy.maxTotalModelSends, sends);
 }

src/server/responses/adapter-continuation.ts

@@ -39,7 +39,6 @@ import {
   formatAnthropicProviderForLog,
 } from "../../oauth/anthropic-routing";
 import {
-  GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST,
   hasEligibleGenericOAuthFailoverTarget,
   isGenericOAuthFailoverEnabled,
   rotateGenericOAuthAccountOn429,
@@ -84,6 +83,7 @@ export function createAdapterContinuations(
     | "commitResolvedOAuthSelection"
     | "genericFailoverAccountId"
     | "genericFailovers"
+    | "genericFailoverLimit"
     | "applyFailoverSnapshot"
     | "replayOAuthCredentialSnapshot"
     | "noteRoutedAttemptSend"
@@ -407,7 +407,7 @@ export function createAdapterContinuations(
        response.status === 429
        && transportState.genericFailoverAccountId
         && !isNonReplayableResponse(response)
-       && transportState.genericFailovers < GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST
+       && transportState.genericFailovers < transportState.genericFailoverLimit
         && isGenericOAuthFailoverEnabled(config, route.providerName)
       ) {
         // Intersection with the shared request budget. The continuation loop re-sends the

src/server/responses/adapter-dispatch.ts

@@ -54,7 +54,6 @@ import {
   formatAnthropicProviderForLog,
 } from "../../oauth/anthropic-routing";
 import {
-  GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST,
   isGenericOAuthFailoverEnabled,
   rotateGenericOAuthAccountOn429,
   failoverAccountSnapshot,
@@ -124,6 +123,7 @@ export async function prepareAdapterExchange(
     | "commitResolvedOAuthSelection"
     | "genericFailoverAccountId"
     | "genericFailovers"
+    | "genericFailoverLimit"
     | "applyFailoverSnapshot"
     | "noteRoutedAttemptSend"
   >,
@@ -856,7 +856,7 @@ export async function prepareAdapterExchange(
       while (
         upstreamResponse.status === 429
         && transportState.genericFailoverAccountId
-        && transportState.genericFailovers < GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST
+        && transportState.genericFailovers < transportState.genericFailoverLimit
         && isGenericOAuthFailoverEnabled(config, route.providerName)
       ) {
         // Intersection with the shared request budget. This arm re-sends through

src/server/responses/passthrough-dispatch.ts

@@ -138,7 +138,6 @@ import type { OAuthAccessSnapshot } from "../../oauth";
 import { publicOAuthAuthenticationErrorMessage } from "../../oauth";
 import { resolveCopilotApiBaseUrl } from "../../oauth/github-copilot";
 import {
-  GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST,
   hasEligibleGenericOAuthFailoverTarget,
   isGenericOAuthFailoverEnabled,
   rotateGenericOAuthAccountOn429,
@@ -194,6 +193,7 @@ export async function preparePassthroughExchange(
     | "refreshResolvedOAuthSelection"
     | "replayOAuthCredentialSnapshot"
     | "genericFailovers"
+    | "genericFailoverLimit"
     | "applyFailoverSnapshot"
     | "noteRoutedAttemptSend"
     | "selectionIsCurrent"
@@ -1319,7 +1319,7 @@ export async function preparePassthroughExchange(
       // have run and would cool down an account that refused nothing.
       && !isNonReplayableResponse(upstreamResponse)
      && transportState.genericFailoverAccountId
-      && transportState.genericFailovers < GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST
+      && transportState.genericFailovers < transportState.genericFailoverLimit
       && isGenericOAuthFailoverEnabled(config, route.providerName)
     ) {
       // The roster cap above is one half of the bound; the request's shared budget is the

src/server/responses/request-send-budget.ts

@@ -200,7 +200,9 @@ export function createResponsesSendBudget(
     options: { allowFinalRecoveryReserve?: boolean } = {},
   ): { attempts: number; permit?: SingleUseDispatchPermit } => {
     const base = remainingTransientSendBudget(cap);
-    if (base > 0) return { attempts: base };
+    // The hop already paid for this leg's first send. Include it in the helper's total
+    // attempts without charging it again, or the final account loses one transient attempt.
+    if (base > 0) return { attempts: Math.min(cap, base + (pendingHopPermit ? 1 : 0)) };
     // A provider-configured transient total is an exact physical-send ceiling. Once it is
     // exhausted, the request-wide recovery reserve must not silently widen it. The default stays
     // permissive so unconfigured providers retain the guarded profile's fourth recovery send.
@@ -217,8 +219,8 @@ export function createResponsesSendBudget(
   /**
    * One credential hop of this logical request, admitted by the INTERSECTION of two bounds.
    *
-   * `GENERIC_OAUTH_MAX_FAILOVERS_PER_REQUEST` and `ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST`
-   * stay exactly as they are: they bound rotation within one credential roster. What neither
+   * The snapshotted generic OAuth roster cap and `ANTHROPIC_POOL_MAX_FAILOVERS_PER_REQUEST`
+   * bound rotation within one credential roster. What neither
    * can see is everything else this request already sent, so three hops layered on a spent
    * budget still reached upstream three more times. A hop now happens only when its own layer
    * cap AND the shared budget both permit it, and the smaller of the two wins.
@@ -245,7 +247,13 @@ export function createResponsesSendBudget(
     countedExternally = false,
   ): { allowed: boolean; permit?: SingleUseDispatchPermit } => {
     if (!isRequestExecutionBudget(sendBudget)) return { allowed: true };
-    const decision = sendBudget.reserveDispatch({ sendClass, targetKey, countedExternally });
+    const decision = sendBudget.reserveDispatch({
+      sendClass,
+      // A same-provider credential hop is not a model/endpoint transition. Its diagnostic
+      // label must not replace the physical target used by the adapter's next retry.
+      targetKey: sendClass === "auth-recovery" ? sendBudget.lastTargetKey ?? targetKey : targetKey,
+      countedExternally,
+    });
     return decision.allowed ? { allowed: true, permit: decision.permit } : { allowed: false };
   };
   /**
@@ -322,6 +330,13 @@ function adapterDispatchBudgetView(
       // already paid does not make an unsafe replay safe, so that check stays with the budget.
       if (intent.replaySafe !== false) {
         const hopPermit = hop.claimHopPermit();
+        if (hopPermit && intent.targetKey !== budget.lastTargetKey) {
+          // The rotated credential can select a different regional endpoint. Replace the
+          // provisional booking synchronously so the actual destination obeys transition
+          // limits, while the physical send is still charged only once.
+          hopPermit.release();
+          return budget.reserveDispatch({ ...intent, sendClass: hopPermit.sendClass });
+        }
         // Confirmed here rather than in `use()`: the adapter reserves immediately before it
         // opens the transport, which is the same boundary the hop's own confirmation uses.
         // A permit some other leg already settled returns false, and this falls through to a