Repository navigation
Conversation
Move NodeRoom's default model ids off retired or deprecated ones, per model-defaults SPEC Part A section A.2.9: - one DEFAULT_AGENT_MODEL (gemini-3.8-flash) in convex/modelDefaults.ts replaces the gemini-3.5-flash literals (Google deprecated it 2026-10-08); env chains unchanged - top_paid default anthropic/claude-sonnet-4 -> anthropic/claude-sonnet-5.5; Haiku defaults -> claude-haiku-5-5; pi-ai default gemini-2.5-flash -> DEFAULT_AGENT_MODEL - TTS gemini-3.1-flash-tts-preview -> gemini-3.8-flash-tts (shutdown 2026-11-17) - catalog: 5.5 and gemini pricing rows, tier allowlists, fallback chains pruned; the dotted claude-id regex becomes an explicit table (claude-opus-4.7 now maps to claude-opus-4-7) - Claude 5.5 request shape: forced tool_choice downgraded to auto for Sonnet/Opus 5.5, explicit effort on the AI-SDK path, 2048 max_tokens floor, @ai-sdk/anthropic >= 3.0.120 - judge scripts pinned to gemini-3.6-flash with temperature removed - ladder: --real needs a dollar cap, per-dispatch cost reservation, receipt carries the tested commit, budget and dispatch policy Merge is gated on the L1-L4 ladder receipt (not run here: paid). The adaptiveRouter fast lane stays on gemini-3.5-flash until that receipt exists. Spec: model-defaults SPEC Part A §A.2 (NodeRoom) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Scaffold Handoff — For Your Coding Agent
Immutability CheckMode: advisory ✅ No immutable files were modified in this branch. Changed Files
Needs Adversarial Review — Do NOT Apply YetThese proposals passed the reject check but have not been approved by
Safety Boundary
Immutable files (never modify):
Scaffold files (safe to modify):
|
HomenShum
added
the
needs-live-eval
HomenShum
marked this pull request as draft
October 14, 2026 20:09
- merge gate: PR is a draft with the needs-live-eval label; the ladder receipt, the gemini-3.8-flash-tts listen check and the top_paid Environment smoke are listed as explicit blockers (PR body, CHANGELOG lane) - ladder --json-out is now reviewable under A.0.3: command, candidateIds, echoedModels, per-dispatch rows (provider-echoed model from the raw body, usage, cost, fallbackUsed), environment, budget.requestCeiling - a refused reservation or unpriced id is terminal for the meter: no fallback model, retry or candidate dispatches after it; the provider-neutral Google fallback honours crossModelFallback: false - claude-haiku-5-5 is priced tier-aware (>100K prompt tokens: $0.50/$2.50) in every cost and reservation path; claude-opus-4-7 gets pricingAsOf and an agreement row - Sonnet/Opus 5.5 send strict tool definitions where a forced tool_choice becomes auto; capture reasoner and provider parser send the explicit effort - .env.example copies of the changed defaults now match the code (and a test pins them) - request-shape tests for the TTS route, private stream route, provider parser default, judge scripts; ladder budget tests (b) and (e) follow the specified scenarios Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- long-context price tier is applied per request only (priceRequest / convexPriceRequest in reserveDispatch, meter settlement, runtime priceStep, calculateRequestCost, exactConvexUsageCost); summed usage is priced at base-tier rates (priceAggregateTokens) and the ladder sums settled dispatch costs; haiku-5-5 long-context tier gets cachedInputPer1M 0.05 - tests that fail on revert for the five DEFAULT_AGENT_MODEL call sites, every remaining judge, benchmark and smoke script through a fake transport, and both AGENT_MODEL fallbacks in qaMatrix - providerParserLive env-precedence test uses a non-default env id again - LangChain bridge passes strict: true on the forced tool_choice downgrade - probe.ts and providerParseSmoke.ts send the explicit claude effort; providerParseSmoke compares the provider to "gemini" so the Gemini smoke reaches Google - ladder pre-flight request ceiling and estimate multiply free-auto by OPENROUTER_FREE_AUTO_LIMIT - pricing-source inventory lists run.ts (OpenRouter snapshot first) and the proxy sweep Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- SPEC A.0.10: resolveModelAlias throws UnknownModelError for an id that is not priced, not declared by the alias table and not free-looking, so adapter model() and convexModel() (and the fallback lists they read) refuse it before any request; an unknown -free / :free id resolves to openrouter/free-auto. Pricing and telemetry readers use lookupModelAlias - a free primary no longer falls back to a paid model: adapter fallbackModelFor and convexModel drop the paid candidate unless allowPaidFallback is set and a reservation exists (SpendMeter / finite maxCostUsd) - ladder runRung reports the sum of its settled dispatch costs when a rung fails after charged dispatches (was hardcoded 0) - ladder request ceiling and estimate resolve the route alias before the free-auto fan-out test - probe.ts gpt-6.1-sol is asserted on the wire (URL, model, no sampling) - benchmark/run.ts pricing moved to pricingAtRun.ts; the pricing agreement test runs it and the --dry-run sweep on a dated OpenRouter /models fixture - tests that dispatched never-heard-of ids now assert the refusal; their invariants stay on the alias-declared unpriced id claude-sonnet-4.5 Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- resolver: an id the repo ships or freezes by name is known without a price row (mobile picker, SUPPORTED_MODEL_ROUTES, frozen BTB default gpt-4.1-mini, official judges); every concrete :free id the repo or the spec's 2026-10-10 live list names resolves as itself, never to openrouter/free-auto. Only truly unknown ids throw; only unknown :free ids reroute. tests/modelKnownIds.test.ts reads the owning lists, so a new id added there without a catalog entry fails. - public boundary: defaultModelPolicyForRoute refuses a modelPolicy the catalog does not know (unknown_model_policy) for createOrReuse, start and startPublicAsk; fixtures that used placeholder strings now name catalog ids. - nodeagent/free-auto: the paid direct-Gemini leg needs allowPaidFallback and a meter (same guard as fallbackModelFor); NODEAGENT_FREE_AUTO_GOOGLE_MODEL precedence unchanged. - pi-ai default is asserted on the wire through createPiAiAdapter().next(). - claude-opus-4-7 loses the unsourced cachedInputPer1M 0.50; the agreement test now pins every sourced cache rate and the absent one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- resolver (D1): REPO_DECLARED_UNPRICED_IDS now holds every OPENROUTER_TOP_PAID_AGENT_ROUTES route and
every SPEC 1.2 wire id on every transport (9 of 19 top-paid routes and 18 of 32 section 1.2 ids threw).
tests/modelKnownIds.test.ts reads resolveRouteSet for all six route sets, allAgentLlmRoutes() and the
section 1.2 list.
- operator pass-through (D1): model(id, { operatorSelected: true }) keeps an operator- or live-list-selected
id as written; set only by benchmark/run.ts, proofloop-free-model-gauge, ladder --real, spreadsheetbench-run
and chatIntakeRuntime. Server code stays fail-closed.
- PlanPreview-blocked runRoomAgent records again: the not_started marker is accepted only with
initialStatus blocked; a convex-test drives the branch.
- streamPrivateReplyText and aiSdkReasoner resolve the id first (one guard each); tests assert 0 fetches
for unknown ids.
- tests/modelRefreshAgentWire.test.ts drives runRoomAgent, runPrivateAgent and runFreeAutoJobSlice through
convex-test with a fake fetch: emitted model gemini-3.8-flash, no sampling keys.
- D2: top_paid default is nvidia/nemotron-3.5-super-vl:free (newest qualifying :free row of the public
OpenRouter list, fetched 2026-10-16T11:32:04Z, docs/eval/openrouter-free-models-2026-10-16.json);
live-unverified (no key, Homen 2026-10-16).
- .env.example: the direct Gemini leg is documented as opt-in; .tours citation line follows agentJobs.ts.
- CHANGELOG/qa/model-refresh-noderoom.md: round 5 entry with the REA/ORX record (D5).
- Not run (paid): the L1-L4 ladder receipt; tests/qaMatrix.test.ts stays red on purpose.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- operator pass-through (D1, A.0.4): operatorSelected is now set at every CLI entry point that takes an operator id, not five of them: financeModelLive, managedLockPerf, professionalRuntimeLive, halo-live-path-calibration, sfn-nonbtb-nodeagent-runner, bankerToolBenchNodeAgentGeneral and startup-diligence-provider-eval. convexModel() gains the same option (ConvexModelOptions.operatorSelected); normalizeFallbackModelIds no longer re-resolves a primary its caller already resolved. The source pin in tests/modelKnownIds.test.ts now scans every resolver call in twelve entry points (multi-line calls included). - free-looking unknown ids (A.0.10): aiSdkReasoner and streamPrivateReplyText dispatch the RESOLVED id. Streaming sends openrouter/free-auto like any known free-auto input; the capture seam serves only Anthropic/OpenAI and refuses (capture_reasoner_route_unsupported). Tests: 0 raw-name requests. - public boundary (A.0.10): defaultModelPolicyForRoute passes the AGENT_ORCHESTRATOR_MODEL / AGENT_MODEL / AGENT_TOP_PAID_MODEL / AGENT_RESEARCH_MODEL defaults, and the file-egress promotion passes configuredFileEgressModel(), through knownModelPolicy. Override precedence unchanged. 8 new convex-test cases fail without the change. - compatibility (A.0.5): claude-opus-4.1 and claude-haiku-3.5 alias to their hyphenated wire ids (price rows added at the same prices); openrouter/auto, the benchmark-lane label, is a declared known id again. - A.0.6: the gemini-3.6-flash contextWindow cites the public OpenRouter list (read 2026-10-16). - .tours citation moves agentJobs.ts:2798 -> :2799; CHANGELOG/qa/model-refresh-noderoom.md round 6 entry. - Not run (paid): the L1-L4 ladder receipt; tests/qaMatrix.test.ts stays red on purpose. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- startup-diligence eval: the contract job always carries the catalog free route, so an operator-picked model the catalog does not know is the only request and the reported model (was: unknown_model_policy, second paid request, wrong model reported). - adapter.judge(): operatorSelected option (D1/A.0.4) passed by professionalLiveCatalog and benchmark/run. - streamingModel: refuse openrouter/free-auto (and nodeagent/free-auto) before any request; it is not an OpenRouter model id. The test asserted the invalid wire value and now asserts the refusal. - pricing agreement test exercises proofloopModelTracking (source 6) and the retired opus-4-1 / haiku-3-5 rows. - provider parser tests assert no forced tool_choice, native output_config.format and max_tokens on the Anthropic generateObject path. - Not run (paid): the L1-L4 ladder receipt; tests/qaMatrix.test.ts stays red on purpose. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- A.0.6: the two rows the PR added (claude-opus-4-1, claude-haiku-3-5) carry pricingAsOf "2026-04", the month of origin/main's "official docs, April 2026" block they were copied from; the agreement test now asserts the date instead of "undated". - A.0.3 receipt honesty: SpendMeter counts requests past MAX_DISPATCH_RECORDS in droppedDispatches; the ladder report carries budget.dispatchesDropped and takes rung cost from the meter's spentUsd delta, not from the capped row array. Test: 4,097 fake requests give 4,096 rows, droppedDispatches 1, requests 4,097. - A.0.6: request-shape test sends the top_paid default (nvidia/nemotron-3.5-super-vl:free) through convexModel: model as written, tool_choice required, no sampling keys. - Not changed, escalated: free-to-paid file-egress promotion (needs a Homen decision) and the Top paid UI copy; the paid ladder receipt and the TTS listen check remain merge blockers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
- A.2.9: remove the fallback-chain keys for gemini-3.8-flash (DEFAULT_AGENT_MODEL), gemini-3.6-flash and gemini-3.5-flash-lite. Spec authorizes chains for the 5.5 ids only; a chain on the default route sent room content to OpenRouter/OpenAI on a failed default turn, which origin/main never did. Test now pins the three ids chain-less and restricts the "has a chain" assertion to the 5.5 ids. - Chat.tsx Top paid copy no longer promises a paid model: it names the operator env vars and says the server default is free when neither is set. - evals/ladder.ts: droppedDispatches > 0 exits 1 like stoppedBy (incomplete rows). - Heavy cases (ladderBudget (e), modelCostClass 4,097-request) get a 30s per-test timeout; no assertion changed. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Refreshes NodeRoom's stale default model ids (Google deprecated
gemini-3.5-flash2026-10-08; OpenRouter dropsgoogle/gemini-2.5-*2026-10-20;claude-sonnet-4retired 2026-06-15;gemini-3.1-flash-tts-previewshuts down 2026-11-17) and fixes the request shapes the new ids reject. Env chains and caller arguments are unchanged: if prod setsAGENT_MODEL,AGENT_ORCHESTRATOR_MODEL,AGENT_TOP_PAID_MODEL,VOICE_GEMINI_TTS_MODELetc., the default ids prod resolves do not change - this PR moves code fallbacks only. Qualifier: an env or fallback value the catalog does not know now fails closed (SPEC A.0.10):unknown_model_policyatconvex/agentJobs.ts:2100-2107(job start),UnknownModelErrorinconvexModel()andmodel(). A prod env var that names an id outside the catalog (and is not a free-looking id) stops working instead of being sent to a provider as written; check the values of the env names in the Env checklist before merge. Operator overrides in smoke/eval scripts stay pass-through (D1).This PR is a draft and carries the
needs-live-evallabel. Both stay until the two merge blockers below are closed. The label is a block, not a note (SPEC A.0.3). Nothing in this PR made a paid provider call.MERGE BLOCKERS (both need a provider key or a human at a speaker; none was run here)
L1-L4 ladder receipt gates the
gemini-3.8-flashdefault (and theadaptiveRouter.ts:136fast-lane literal).tests/qaMatrix.test.ts("keeps the Convex interactive fallback on a ladder-safe route") fails on purpose until it exists:gemini-3.8-flashhas no PASS row indocs/qa/production-matrix.json. The check is not weakened.Pre-flight (re-run on this head with
fetchreplaced by a function that throws; 0 fetch calls): worst case $8.0552 for 2 routes x 4 rungs, request ceiling 144, so the cap must be at least that or the run refuses to start. Actual spend is expected to be a small fraction of it. Commit the receipt in a receipt-only commit on top of the tested commit (A.0.3 form a) with a clean tree.The
--json-outreport now carries everything A.0.3 reviews:command,candidateIds,echoedModels,rows[](one per provider request:requestedModel,resolvedModel,echoedModel,usage,costUsd,fallbackUsed,rung),environment,testedCommit(40-hex),gitDirty,constantSha256,budget(maxUsd,estimateUsd,spentUsd,requests,requestCeiling,stoppedBy) anddispatchPolicy. Reject the receipt unless:requestedRoutes == candidateIds == ["gemini-3.6-flash","gemini-3.8-flash"]; rungs are exactly L1-L4 with 8 passing result rows; every row hasresolvedModel == requestedModel,fallbackUsed == falseand anechoedModelequal to the requested id (agemini-3.8-flashrequest answered as another model is the failure this field exists to expose; 3.5 is already served as 3.6);budget.stoppedBy == null;budget.requests <= budget.requestCeiling;gitDirty == false;dispatchPolicy == {maxRetries:0, crossModelFallback:false}.echoedModelis read from the raw response body (modelVersion/model): the AI SDK'sresponse.modelIdis the requested id for Google, so it could never show a re-route.Manual listen check of
gemini-3.8-flash-tts(convex/http.ts:36, SPEC A.0.7: a frozen-class id with an announced shutdown inside 60 days moves to its vendor-named replacement, and a listen test is the gate). Shape-verified here by a fake-transport test throughPOST /voice/synthesize; the audio itself is unverified. To close it: on a deployment withGOOGLE_GENERATIVE_AI_API_KEYandVOICE_TTS_PROVIDER=gemini, POST one short line to/voice/synthesize, confirmX-Voice-TTS-Model: gemini-3.8-flash-ttsand a playable WAV in theKorevoice, and record the result on this PR. The oldgemini-3.1-flash-tts-previewshuts down 2026-11-17.No longer a blocker (round 5, SPEC-v9 D2, Homen 2026-10-16): the
top_paidEnvironment smoke. No working OpenRouter key exists, so an OpenRouter-routed role that this refresh moves goes to the newest:freerow of the public listhttps://openrouter.ai/api/v1/modelsthat has the capabilities the role needs.top_paid(convex/agentJobs.ts:2118,DEFAULT_TOP_PAID_MODELinconvex/modelDefaults.ts) is nownvidia/nemotron-3.5-super-vl:free(created 2026-10-15, 1,000,000 context,tools,tool_choice,structured_outputs, text output, noexpiration_date), from the list fetched 2026-10-16T11:32:04Z (docs/eval/openrouter-free-models-2026-10-16.json). Marked: live-unverified (no key, Homen 2026-10-16). Merge is allowed with that marking. Per-role D2 scope is in the Review round 5 table below.Also not run:
prod:gate(network, audit, design and proof gates). CI'sverifyjob fails at thenpm auditstep on rootundici 8.10.0, the same version asorigin/main, so that failure is not from this PR (PR #256 bumps it); failing checks are not required for merge, which is why the block is the draft state plus the label.Runtime sites (44; migrate 26, freeze 16, defer 2 per spec)
convex/agent.ts:346gemini-3.5-flashconvex/agent.ts:345minimax/minimax-m3convex/agent.ts:783gemini-3.5-flashconvex/agent.ts:65z-ai/glm-4.7-flashconvex/agentJobs.ts:2103gemini-3.5-flashconvex/agentJobs.ts:2102minimax/minimax-m3convex/agentJobs.ts:2118anthropic/claude-sonnet-4nvidia/nemotron-3.5-super-vl:free(DEFAULT_TOP_PAID_MODEL) - claude-sonnet-4-20250514 retired 2026-06-15 (rule 1). SPEC-v9 D2: no OpenRouter key, so the target is the newest:freerow with tools, tool_choice, structured_outputs, >= 256K context, text output and noexpiration_datein the public list fetched 2026-10-16T11:32:04Z. live-unverified (no key, Homen 2026-10-16). Env chainAGENT_TOP_PAID_MODEL ?? AGENT_MODEL ?? constunchanged, so a deployment that sets either does not change. A free route takes no uploaded file: atop_paidjob in a room with a file is refused at the free-file-egress gate (or promoted toz-ai/glm-4.7-flashunder the existingFREE_AUTO_ALLOW_FILE_EGRESS_PROMOTIONopt-in) unless an env id is set. Theanthropic/claude-sonnet-5.5tool_choice: autoplusstrict: truehandling stays inconvexModelfor an operator who sets that id.convex/agentJobRunner.ts:512gemini-3.5-flashconvex/agentJobRunner.ts:514openrouter/free-autoconvex/http.ts:175gemini-3.5-flashconvex/streaming.ts:68gemini-3.5-flashconvex/http.ts:35gemini-3.1-flash-tts-previewconvex/http.ts:36) - Google shutdown 2026-11-17, vendor-named replacement, frozen-class voice exception (A.0.7).VOICE_GEMINI_TTS_MODELprecedence unchanged. The manual listen check is merge blocker 2. A fake-transport test drivesPOST /voice/synthesizeand asserts the wire id.convex/http.ts:283gpt-4o-mini-transcribeconvex/http.ts:352gpt-4o-mini-ttsconvex/okfEmbeddingProvider.ts:43gemini-embedding-2src/landing/modelRegistry.ts:37z-ai/glm-5.2src/landing/modelRegistry.ts:48anthropic/claude-fable-5src/ui/Chat.tsx:1518moonshotai/kimi-k3src/ui/Chat.tsx:1690gemini-3.5-flashsrc/nodeagent/models/modelCatalog.ts:305moonshotai/kimi-k3src/nodeagent/models/modelCatalog.ts:232gpt-5.4src/nodeagent/models/modelCatalog.ts:272claude-haiku-4.5src/nodeagent/models/modelCatalog.ts:288gemini-3-flash-previewsrc/nodeagent/models/modelCatalog.ts:342gpt-5.4-minisrc/nodeagent/models/modelCatalog.ts:187claude-haiku-4.5src/nodeagent/models/modelCatalog.ts:612claude-sonnet-4.6claude-opus-4.7now maps toclaude-opus-4-7(it was silently rewritten toclaude-opus-4-8); theopusandclaude-opusshortcuts are pinned toclaude-opus-4-8, which is what they effectively resolved to before, so their behaviour is unchanged;claude-*-5.5dotted ids map toclaude-*-5-5; every other dotted 4.x id that can sit in a persisted row is listed.src/nodeagent/models/modelCatalog.ts:641z-ai/glm-5.2src/nodeagent/models/modelCatalog.ts:806z-ai/glm-5.2AGENT_FALLBACK_MODELSprecedence unchanged.src/nodeagent/models/adapter.ts:132claude-haiku-4-5sdkToolChoiceForModeldowngradesrequiredtoauto(withstrict: truetools) for forcedToolChoice-forbidden ids, and the ladder meter'sreserveDispatch+maxOutputTokenssit inside the threewithRetryclosures with the dispatch-policy option (A.2.9 Ladder budget).src/nodeagent/models/adapter.ts:152gemini-3.5-flashsrc/nodeagent/models/adapter.ts:216(env-provided)src/nodeagent/models/piAiAdapter.ts:9gemini-2.5-flashmodel()), so the test does NOT do a pi-ai catalog lookup; it drivescreatePiAiAdapter().next()through a fake transport and asserts the wire idgemini-3.8-flashin the request URL, the absence of sampling keys, and the provider and pricing of the id (tests/modelRefreshRequestShape.test.ts).src/nodeagent/capture/reasoning.ts:23claude-haiku-4-5CAPTURE_REASONING_MODELprecedence unchanged; now sends the explicit SPEC 1.3 effort (claudeEffortProviderOptions).src/nodeagent/models/providerParserLive.ts:110gemini-3.5-flashsrc/nodeagent/models/providerParserLive.ts:111gpt-5.5src/nodeagent/models/providerParserLive.ts:112claude-haiku-4-5src/nodeagent/models/providerParserLive.ts:113z-ai/glm-5.2z-ai/glm-5.2,moonshotai/kimi-k2.7-codeand the other remaining candidates are frozen paid routes with no vendor-named successor, and no:freereplacement is added.src/nodeagent/core/adaptiveRouter.ts:82minimax/minimax-m3gemini-3.5-flashuntil the ladder receipt exists (spec: moves only after the receipt) - defer, condition = receipt file present.src/lib/models/modelRouter.ts:47nebius/zai-org/GLM-5.2src/nodeagent/models/openRouterFreeModels.ts:59cohere/north-mini-code:freebackend/app/config.py:78nvidia/nemotron-3-nano-omniGET {GMI_MAAS_BASE_URL}/models+ the backend's pytest; not covered by npm gatesbtb_noderoom_agent/harbor_adapter.py:83z-ai/glm-5.2scripts/gemini-demo-media-judge.ts:21gemini-3.5-flashscripts/design-qa/rubric-panel.ts:31gemini-3.5-flashExecutable (E) and runtime-adjacent (R) sites (20; migrate 7, freeze 12, defer 1)
src/nodeagent/models/modelCatalog.ts:63claude-opus-4.7longContext, applied by every cost and reservation path), claude-sonnet-5-5 ($2/$10), claude-opus-5-5 ($4/$20), gemini-3.8-flash ($0.75/$3.75; $1.50/$7.50 from 2027-01-01), gemini-3.6-flash ($0.75/$3.75), gemini-3.5-flash-lite ($0.30/$2.50); the claude-opus-4-7 row ($5/$25) the alias table now needs carriespricingAsOf; retired 4.x rows kept for persisted rooms; all rows are in the agreement test (A.0.6).src/nodeagent/core/creditModel.ts:57z-ai/glm-5.2src/nodeagent/models/convexModel.ts:968(any claude-*)anthropicStepalready sent no temperature/thinking/tool_choice/prefill; added the 2048max_tokensfloor for 5.5 ids (thinking counts), reading blocks by type (test), andstrict: truetool definitions where a forced choice is downgraded toauto;tool_choiceforwarding at :842/:892 sendsauto(plus strict tools) for forcedToolChoice-forbidden ids.scripts/benchmark/modelEvalConfig.ts:126deepseek/deepseek-v4-flashscripts/benchmark/run.ts:63gemini-3.1-flash-litescripts/benchmark/probe.ts:13claude-opus-4-8evals/ladder.ts:670gpt-5.4-nanoevals/professionalLiveCatalog.ts:51deepseek/deepseek-v4-flashdeepseek-v4-flashretirement is a different transport - per-transport fact, B.1); research_champion by evidence; also financeModelLive.ts:983, halo-live-path-calibration.ts:22, package.json:241,245scripts/finch-official-judge.py:20gpt-5-miniscripts/finauditing-official-judge.py:26gpt-5-miniscripts/mbabench-official-sweep.py:26google/gemini-3-flash-previewscripts/judge-demo-gif.ts:24gemini-3.5-flashscripts/walkthroughs/judge-video.ts:53gemini-3.5-flashscripts/eval/providerParseSmoke.ts:21claude-haiku-4-5scripts/startup-diligence-provider-eval.ts:150gemini-3.5-flashscripts/bankertoolbench-nodeagent-full-sweep.ps1:7z-ai/glm-5.2scripts/bankertoolbench-nodeagent-clean-sequential.mjs:8gpt-4.1-miniscripts/inference-nodetrace/run.mjs:13gpt-5.4scripts/walkthroughs/voiceover.ts:60gpt-4o-mini-ttspackage.json:82gpt-5-miniClass N (24) dispositions - docs loop, test assertions, examples, fixtures
doc (5)
src/nodeagent/models/modelCatalog.ts:1header comment ("Primary NodeBench default: z-ai/glm-5.2"): freeze - glm-5.2 is untouched by this PR.README.md:1604(npx convex env set AGENT_MODEL gemini-3.5-flash): defer - an operator command that names the ladder-recorded route. It moves togemini-3.8-flashin the same commit as the ladder receipt (merge blocker 1), exactly like theadaptiveRouter.ts:136literal; moving it first would recommend a route with no PASS row.production-agent.json:108(modelMigration.pinnedModelz-ai/glm-5.2): freeze - pins the glm-5.2 default, unchanged.docs/eval/FIRST_USER_E2E.md:209(z-ai/glm-5.2 env example): freeze - recorded run instructions, glm-5.2 unchanged.docs/historical mentions (841 docs, 20 evidence, 30 .proofloop, 8 episodes files): freeze - hash-bound evidence and archived prototypes are never rewritten.eval-fixture (1)
public/eval/proofloop-prod-proxy-benchmark-matrix.json: freeze - generated benchmark artifact derived fromsrc/eval/proofloopProdProxyBenchmarkMatrix.ts; glm-5.2 unchanged.example (7)
.env.example:65NODEAGENT_FREE_AUTO_GOOGLE_MODEL: migrate ->gemini-3.8-flash(the code default)..env.example:116GEMINI_SEO_VIDEO_JUDGE_MODEL: migrate ->gemini-3.6-flash(the judge script default)..env.example:165PROVIDER_PARSER_GEMINI_MODEL: migrate ->gemini-3.8-flash,gemini-3.6-flash; the neighbouring:167PROVIDER_PARSER_ANTHROPIC_MODEL->claude-haiku-5-5,claude-sonnet-5-5and:168PROVIDER_PARSER_OPENROUTER_MODELwithout the unlistednex-agi/nex-n2-proare part of the same copy (the inventory records them under :165).:84VOICE_GEMINI_TTS_MODEL->gemini-3.8-flash-tts(the shutdown-deadline fix). Env vars beat code defaults, so a.envbuilt from the old file would have silently restored every one of these ids..env.example:66AGENT_MODEL=gpt-5.4-nano,:103AGENT_FILE_EGRESS_MODEL,:147FREE_AUTO_JOB_MODEL: freeze - values this PR does not change.src/nodeagent/core/subagentDispatcher.ts:32('minimax/minimax-m3'in a zod.describe): freeze - research lane champion.tests/modelRefreshServerRoles.test.ts, last describe) now fails if any of these.env.examplevalues drifts from what the code defaults to.test-assertion (11)
modelAliasClaude.test.ts:11(claude-opus-4.7 -> claude-opus-4-7),phaseModel.test.ts:73(the top_paid fallback fixture is nowanthropic/claude-sonnet-5.5; claude-sonnet-4 retired 2026-06-15),agentJobsRuntime.test.ts(new: default model per route policy, asserted as literals),openRouterFreeModels.test.ts(autofollows the constant),openAiTokenLimit.test.ts(gpt-6 case),providerEgressPolicy.test.ts(re-run, passes unchanged), and in this roundproviderParserLive.test.ts(env id is the non-defaultgemini-3.5-flash-lite, expecting[gemini-3.8-flash, gemini-3.5-flash-lite, gemini-3.6-flash], so env precedence is proven; a second case with a default env id proves de-duplication).providerParseSmoke.tsalso compared the provider to"google"whilegetProviderForModelreturns"gemini", so the Gemini smoke never reached Google; fixed (pre-existing on main).convexModelFreeAutoMode.test.ts(two tests moved fromvendor/unknown-paid-modelto the alias-declaredclaude-sonnet-4.5, because the unknown id now throws).kimiK3Default.test.ts:22,chatModelPickerPolish.test.tsx:168(claude-sonnet-4-6 is still a live catalog row),openRouterConvexBenchmark.test.ts,finchOfficialJudgeCostGuard.test.ts(frozen official judge),modelEvalMatrix.test.ts(promotion tags).Hash-bound evidence files untouched.
Review round 3 (Codex round 2 defects and the two round-1 findings that stayed open)
adapter.fallbackModelForandconvexModeldrop a paid fallback for a free primary (isFreeModelId) unless the caller setsallowPaidFallback: trueand a reservation exists: adapter = aSpendMeter(meteredDispatchreserves the paid model's worst case before the request), Convex = a finitemaxCostUsdonnext()(the failover controller estimates each candidate against it and skips what does not fit). Tests: a concrete:freemodel that is down under the default policy,allowPaidFallbackwithout a meter, a meter without the flag, both together, a paid fallback the meter cannot afford,fallbackModelIds, the deploymentAGENT_FALLBACK_MODELSlist, and paid -> paid / paid -> free unchanged (tests/modelCostClass.test.ts). No production caller setsallowPaidFallback: default is deny.resolveModelAliasthrowsUnknownModelErrorfor an id that is not priced, not declared by the alias table, not declared by the repo (round 4: the picker, the benchmark routes, the frozen defaults and judges) and not free-looking; an unknown-free/:freeid resolves toopenrouter/free-auto.model(),convexModel(),judge(), the adaptive router and the fallback lists therefore refusegemini-9-flash-unknown,acme/paid-unknown-model,claude-opus-4.9before any request (fake transport: 0 fetch calls), metered or not. Readers that must tolerate historical ids (priceRun,reserveDispatch,creditModel,proofloopModelTracking, the egress policy,benchmark/run.ts, the interop normalizer, the LangChain strict check) call the newlookupModelAlias, which is the old behaviour.runRungreports the sum of its dispatch costs. Test drivesmain()with a charged answer thenECONNRESET: row cost equals the dispatch sum andbudget.spentUsd(reverting tocost: 0fails it).ladderRequestCeilingandestimateLadderUsdtest/free-auto$/onresolveModelAlias(route):free,free-auto,openrouter-auto,openrouter-free-auto,nodeagent-autonow countOPENROUTER_FREE_AUTO_LIMITper step;openrouter/freeand concrete aliases count 1.gpt-6.1-solrequest shapeExisting tests that dispatched never-heard-of ids were retargeted, not loosened:
evals/ladderBudget.test.ts(g),tests/convexModelFreeAutoMode.test.ts(two tests) andtests/modelAliasClaude.test.ts. The premise of each (an id the catalog does not know reaches a provider) is the defect, so each now asserts the refusal, and their original invariants (zero requests, meter untouched, terminal stop, conservative reservation, hard-budget block) are kept on the one kind of id the adapter can still route without a price row: an id the alias table declares (claude-sonnet-4.5->claude-sonnet-4-5). The old ids stay in comments in the diff.Review round 4 (Codex round-2 defects against
6dabf77f):freeids reroutedmodelCatalog.REPO_DECLARED_UNPRICED_IDSholds the mobile picker ids (anthropic/claude-fable-5,openai/gpt-5.5,google/gemini-3.5-flash), the frozen benchmark routes (ibm-granite/granite-4.1-8b,inclusionai/ring-2.6-1t,xiaomi/mimo-v2.5,google/gemini-3.1-flash-lite,poolside/laguna-*:free), the frozen BTB default (gpt-4.1-mini,openai/gpt-4.1-mini) and the frozen official judges (gpt-5-mini,google/gemini-3-flash-preview);openRouterFreeModels.OPENROUTER_KNOWN_FREE_IDSholds the catalog fallback list, the preferred list and the spec's 15-row live:freelist. No price was invented: a metered run still refuses these as unpriced.tests/modelKnownIds.test.tsreadsMODEL_REGISTRY,SUPPORTED_MODEL_ROUTES,llmModelCatalog(Chat.tsx resolves every tier-map id at render), the free ids and the BTB default out of their owning files and assertsresolveModelAlias,model()andconvexModel()return each as itself (103 cases). Re-probe oforigin/main's resolver (verbatim) against head over 47 of these ids: 0 differ;gemini-9-flash-unknown,acme/paid-unknown-model,claude-opus-4.9still throw;vendor/not-in-catalog:freestill reroutes. Round 5 correction: the claim that frozen and deferred sites "behave as onorigin/main" was false. The probe covered 47 ids, not the repo's route lists: 9 of the 19OPENROUTER_TOP_PAID_AGENT_ROUTESand 18 of the 32 SPEC 1.2 wire ids still threw, andscripts/benchmark/run.ts,ladder:free --free-auto-topandproofloop-free-model-gauge.tsthrew on or rewrote operator-selected ids. See Review round 5, defects 1 and 7.tests/agentJobsRuntime.test.ts: all three mutations rejectgemini-not-in-catalogwithunknown_model_policyand persist no job; a catalog id, an alias and a declared-but-unpriced id still start. Reverting the check fails all three.nodeagent/free-autopaid Gemini legcreatePiAiAdapter().next()is driven through the fake transport: the URL carriesgemini-3.8-flash, no sampling keys; an explicitmodelNamestill picks its own provider. The fiveDEFAULT_AGENT_MODELcall sites stay pinned bytests/modelRefreshCallSites.test.ts(the shared constant is tested on the wire). There is no pi-ai package in this repo; the adapter wrapsmodel().claude-opus-4-7no longer carriescachedInputPer1M(frontier.json lists only $5 / $25). The false "pi-ai catalog" claim in the agreement test comment is gone, and the test now pins every sourced cache rate (sonnet-5-5 $0.10, opus-5-5 $0.20, gemini-3.8-flash $0.075, gemini-3.5-flash-lite $0.03, haiku-5-5 $0.01) and asserts the absent one.Spec discrepancies for a human decision (SPEC section C) - not decided in this PR
nodeagent/free-auto-> paidgemini-3.8-flash: resolved in round 4. The spec'sadapter.ts:152row fixes only theNODEAGENT_FREE_AUTO_GOOGLE_MODELenv precedence, and that is unchanged. The leg itself is now gated like every other free -> paid fallback (adapter.fallbackModelFor): it needscrossModelFallback, an explicitallowPaidFallback: trueand a meter whose awaited reservation (meteredDispatch) precedes the request. Under the default policy a free-route outage is the answer and no paid request is made (tests/modelAdapterProviderNeutralRoute.test.ts: default policy, flag without meter, meter without flag; reverting the guard fails it). No production caller setsallowPaidFallback, so this removes the paid leg in production until a human decides to authorize it.claude-sonnet-4.5,claude-opus-4.5,claude-opus-4.6and what they resolve to). A.0.5/A.0.6 keep old strings valid for persisted rows, A.0.10 says unknown throws. They stay routable; a metered run refuses them (UnpricedModelError), an unmetered caller prices them at the guessed $1/$5. Decision: add verified price rows (no vendor price is in the spec) or drop the aliases.defaultModelPolicyForRoute(the one funnelcreateOrReuse,start,startPublicAskand the two internal starters share) refuses amodelPolicythe catalog does not know withunknown_model_policybefore any job row exists. The earlier worry about live clients persistingscripted/fastdid not hold for client senders (the model picker sends catalog ids, plusopenrouter/free-auto), but this statement missed the one internal sender,runRoomAgent's PlanPreview-blocked branch, which recordsmodelPolicy: "not_started"and was rejected by the new check (fixed in round 5, defect 2: the marker is accepted only withinitialStatus: "blocked");scripted/fast/test-modelwere test fixtures, now catalog ids. The one script that used a label (scripts/startup-diligence-live-eval.ts, contract-only mode, runner stubbed) now carriesopenrouter/free-auto; the evidence labelstartup-contract-evalon the attempt rows is unchanged. Rows already persisted are untouched.getProviderForModelis still a classifier (any slash id classifies as OpenRouter) because free-auto candidates discovered live (for exampleopenrouter/owl-alpha) must classify. The gate is the resolver and the two boundaries, not the classifier.:freeid resolves toopenrouter/free-auto, as A.0.10 asks. Round 4 corrects the first version of this: every:freeid the repo names (the catalog fallback list, the preferred list, the benchmark routes) and every row of the spec's 2026-10-10 live:freelist now resolves as itself, sonvidia/nemotron-3-super-120b-a12b:free,google/gemma-4-26b-a4b-it:freeandpoolside/laguna-*:freeare no longer rewritten. Only an id nothing in the repo or the spec names is rerouted.convex/agentJobs.ts, behind theFREE_AUTO_ALLOW_FILE_EGRESS_PROMOTIONenv opt-in) turns a free-auto job into paidz-ai/glm-4.7-flashwith noallowPaidFallbackand no reservation. Its site is FROZEN in the spec, so it goes to the human (SPEC section C).Review round 5 (Codex round-3 defects against
fec0c331)REPO_DECLARED_UNPRICED_IDSnow holds all 19OPENROUTER_TOP_PAID_AGENT_ROUTESand every SPEC 1.2 wire id on every transport (verified:freeids are inOPENROUTER_KNOWN_FREE_IDS). Before: 9 of 19 top-paid routes and 18 of 32 section 1.2 ids threw, somodel-eval-matrix --route-set top-paidbroke.tests/modelKnownIds.test.tsreadsOPENROUTER_TOP_PAID_AGENT_ROUTES,allAgentLlmRoutes(),resolveRouteSet()fortop-paid,supported,champions,free,research,collaboration, and a section 1.2 list (typed in the test) and assertsresolveModelAlias,model()andconvexModel()return each id as itself. Mutation run: removingqwen/qwen3.7-maxandx-ai/grok-build-0.1from the registry fails their cases. No price was invented; a metered run still refuses these as unpriced.unknown_model_policydefaultModelPolicyForRouteaccepts the internal markernot_startedonly together withinitialStatus: "blocked"(no model will run, so there is no id to check). Any other use of the marker is stillunknown_model_policy(tested).tests/agentJobsRuntime.test.tsdrivesrunRoomAgent's blocked branch (convex-test database, fake ctx): call ordercredits:reserve,agentJobs:createOrReuse,messages:sendAgent,credits:settle, resultplan_blocked, job rowblockedwithmodelPolicy: "not_started". Mutation run: removing the exemption fails it.AGENT_MODELresolveModelAlias(modelId), at the top ofstreamPrivateReplyText(it throwsUnknownModelError, asconvexModel()does for the same env value).tests/modelRefreshServerRoles.test.ts:AGENT_MODEL=gemini-not-in-catalogdriven throughPOST /stream-private-replymakes 0 fetch calls, and the route puts the refusal in the stream as visible text; a direct call withclaude-not-in-catalog-9rejects with 0 calls. Mutation run: removing the guard fails it.captureaction dispatches any Anthropic or OpenAI idresolveModelAlias(modelId), at the top ofaiSdkReasonercoversconvex/capturesNode.ts,captureSource.ts:32andcaptureSourceFirecrawlTool.ts:29.tests/modelRefreshRequestShape.test.ts:claude-not-in-catalog-9,gpt-not-in-catalog,o3-not-in-catalog,claude-opus-9each throwUnknownModelErrorwith 0 fetch calls; the defaultclaude-haiku-5-5and the effort tests are unchanged. Mutation run: removing the guard fails all four.tests/modelRefreshAgentWire.test.tsdrivesrunRoomAgent(public_ask),runPrivateAgentandagentJobRunner.runFreeAutoJobSlice(a claimed job whosemodelPolicyis empty) on a convex-test database with a fakefetch, asserts the emitted model (gemini-3.8-flash, or theAGENT_MODELoverride) in every request URL and no sampling keys, and that an unknownAGENT_MODELsends 0 requests. Codex's mutation (modelNameForEgressalways returningconfiguredFileEgressModel()) plus a runner default ofgemini-3.5-flashnow fails 5 of its 6 tests; both were reverted and the tree restored. Thetests/modelRefreshCallSites.test.tsheader no longer claims the sites are unreachable.top_paidtop_paiddefault isnvidia/nemotron-3.5-super-vl:free(see MERGE BLOCKERS and the row above), selected by a test that re-applies the stated criteria to the recorded list (docs/eval/openrouter-free-models-2026-10-16.json, fetched 2026-10-16T11:32:04Z). live-unverified (no key, Homen 2026-10-16). The Environment-smoke blocker is removed. Per-role D2 decision (D2a: D2 chooses the target of roles this refresh moves under rule 1):agentJobs.tstop_paidapplies (moved from the retiredclaude-sonnet-4);providerParserLive.ts:113does not apply (only drops the unlistednex-agi/nex-n2-pro, picks no new target, remaining candidates are frozen paid routes with no vendor successor);modelRegistry.ts:48does not apply (defer, id is active); the frozenz-ai/glm-5.2,kimi-k3rows and the research laneminimax/minimax-m3stay frozen. Behaviour change disclosed: atop_paidjob in a room with an uploaded file is refused at the free-file-egress gate unlessAGENT_TOP_PAID_MODEL/AGENT_MODELis set.model(id, { operatorSelected: true }): the alias table still applies, but an id the catalog does not know goes to the provider as written, and a concrete:freeid is not rewritten to the auto router. Set only inscripts/benchmark/run.ts(3 call sites),scripts/proofloop-free-model-gauge.ts,evals/ladder.ts(--real, soladder:free --free-auto-toptoo),scripts/spreadsheetbench-run.ts,evals/chatIntakeRuntime.ts; server code never sets it and stays fail-closed. Test:openrouter/owl-alpha:free,qwen/qwen3.9-coder:free,acme/paid-unknown-modelkeep their names; without the option the unknown paid id still throws and the unknown:freeid still goes to the verified free route; a source pin on the five scripts.CHANGELOG/qa/model-refresh-noderoom.md..env.examplecommentallowPaidFallbackand a spend meter (adapter.ts), and no production caller sets either.piAiAdapter.ts:9row, the round-3 "fixed at the resolver and both boundaries" row, the round-4 "behave as onorigin/main" claim and the discrepancy 3 sender claim are corrected above.LADDER_MAX_COST_USD=9 npm run ladder:real -- gemini-3.6-flash,gemini-3.8-flash --levels=1-4 --record --json-out docs/eval/ladder/model-refresh-noderoom.json(pre-flight worst case $8.0552 for 144 requests; actual spend expected far lower).tests/qaMatrix.test.ts:114fails by design until the receipt exists. The manual listen check ofgemini-3.8-flash-ttsis also outstanding.Review round 6 (Codex round-4 defects against
1c98671f)operatorSelected: trueis now passed at every call site that takes an operator id:evals/financeModelLive.ts,evals/managedLockPerf.ts(bothrealModelcalls),evals/professionalRuntimeLive.ts,scripts/halo-live-path-calibration.ts,scripts/startup-diligence-provider-eval.ts, and two more the same search found,scripts/sfn-nonbtb-nodeagent-runner.ts(SFN_MODEL_ID) andsrc/eval/bankerToolBenchNodeAgentGeneral.ts(--model-id).convexModel()had no such option:ConvexModelOptions.operatorSelectedhas the same contract asModelAdapterOptions.operatorSelected(alias table applies, an unknown id goes to the provider as written; server code never sets it), andnormalizeFallbackModelIdsno longer re-resolves a primary its caller already resolved (it would have re-thrown). Tests:convexModel("gemini-not-in-catalog", { operatorSelected: true })puts/models/gemini-not-in-catalog:generateContenton the wire (fake transport) and throwsUnknownModelErrorwithout the option;modelandconvexModelkeepvendor/unknown-xandfoo/bar:free. The source pin is now a scan of EVERY resolver call (balanced parentheses, so multi-line calls count) in 12 entry points instead of a five-file regex; removing the option from the startup-diligence call fails it, and removing theconvexModelbranch fails 2 files.streamPrivateReplyTextnow routesresolveModelAlias(modelId), sogemini-3.8-flash:free,foo/bar:free,claude-not-in-catalog:freeandgpt-not-in-catalog-freego to OpenRouter asopenrouter/free-auto(what the same path already does for a known free-auto input) and the raw string appears in no request.aiSdkReasonerserves only Anthropic and OpenAI, so it dispatches the resolved id and refuses (capture_reasoner_route_unsupported:openrouter/free-auto, 0 requests) when the resolved route is not served; a dotted display id now goes out hyphenated (claude-opus-4.8becomesclaude-opus-4-8). Non-free unknowns still throwUnknownModelErrorwith 0 requests. Tests: 4 cases inmodelRefreshServerRoles, 5 inmodelRefreshRequestShape; reverting either guard to the raw id fails 5 each.defaultModelPolicyForRouteranknownModelPolicyonly on an explicitmodelPolicy. TheAGENT_TOP_PAID_MODEL,AGENT_RESEARCH_MODEL,AGENT_ORCHESTRATOR_MODELandAGENT_MODELdefaults andconfiguredFileEgressModel()now pass it too; precedence is unchanged.tests/agentJobsRuntime.test.ts(convex-test, real handlers): each env var set togemini-not-in-catalogis refused withunknown_model_policyand no job row bystartand, where the route applies, bystartPublicAsk(7 cases), and an unknownAGENT_FILE_EGRESS_MODELis refused when a free job is promoted to it. All 8 fail without the change.openrouter/auto(A.0.5)claude-*id; the explicit table missed the two retained retired ids.claude-opus-4.1now maps toclaude-opus-4-1andclaude-haiku-3.5toclaude-haiku-3-5(the wire ids main sent; both models are retired, so this is parity, not a claim that they work), with hyphenated price rows at the dotted rows' prices so they still price. A test enumerates every dottedclaude-*price key and asserts a hyphenated, priced wire id, and two builders (convexModel,model) are asserted on the wire.openrouter/auto, the labelchooseNodeAgentRoutegives the benchmark lane, is declared known again (unpriced: a metered run still refuses it); a test resolves the model of every router decision (8 tasks x risk x latency x free) as itself.gemini-3.6-flashcontextWindow provenance (A.0.6)frontier.jsonmarks the 3.6 context UNVERIFIED; the value 1048576 iscontext_lengthforgoogle/gemini-3.6-flashon the public OpenRouter list (GET /api/v1/models, no key, read 2026-10-16), now cited in the row. The dollar prices were already sourced.prod:gatenot run.Review round 7 (Codex round-5 defects against
74c20c49)convexModel()only. The contract eval passed the provider's resolved id toagentJobs.createOrReuse, whose public boundary refuses an id outside the catalog, so the candidate loop moved on togemini-3.8-flash. The contract job now always carries the catalog free route (the runner is stubbed; the model measured stays on the attempt rows).tests/modelRefreshServerRoles.test.tsdrivesrunStartupDiligenceProviderEval({ model: 'gemini-not-in-catalog', noWrite: true })with a fake fetch: 1 request, no errors, the report names that model only. Reverting the line gives 2 requests and fails it.judge()has no operator path (D1, A.0.4)judge(id, prompt, { operatorSelected }), same meaning asModelAdapterOptions.operatorSelected; passed byevals/professionalLiveCatalog.tsandscripts/benchmark/run.ts.openrouter-free-discoverkeeps the strict resolver (fixed id). Test: an unknown Gemini id andacme/new-model:freego out as written with the option, and without it the unknown id still throws with 0 requests.openrouter/free-autoopenrouter/free,openrouter/auto).streamPrivateReplyTextthrowsprivate_stream_free_auto_unsupportedbefore any request for the sentinel and fornodeagent/free-auto; the comment is corrected. The round-6 test that asserted the invalid wire value now asserts 0 requests (4 free-looking ids plus the sentinel itself). This supersedes the round-6 item 2 statement that the stream sendsopenrouter/free-auto.proofloopModelCostFieldsForRun(which reaches the privateestimateModelCostUsd) is called for every refresh row, the long-context aggregate case and the two retired rows.tool_choiceassertion ongenerateObjecttool_choice, notools,output_config.format.type === "json_schema"andmax_tokens >= 64000on every Anthropic request (Haiku and Sonnet 5.5).claude-opus-4-1andclaude-haiku-3-5carry main's prices; the agreement test pins them under both spellings in every source. (Superseded in round 8: they are now dated2026-04.)LADDER_MAX_COST_USD=9 npm run ladder:real -- gemini-3.6-flash,gemini-3.8-flash --levels=1-4 --record --json-out docs/eval/ladder/model-refresh-noderoom.json, worst case $8.0552 for 144 requests, not run) and the manual listen check ofgemini-3.8-flash-tts.convex/agentJobs.ts:2481-2485; D2 sends the freetop_paiddefault through it).runLiveProviderParserhas no server caller.Review round 8 (Codex round-6 defects against
4b2af806)LADDER_MAX_COST_USD=9 npm run ladder:real -- gemini-3.6-flash,gemini-3.8-flash --levels=1-4 --record --json-out docs/eval/ladder/model-refresh-noderoom.json(worst case $8.0552, ceiling 144 requests; a $1 cap refuses before any request).tests/qaMatrix.test.tsstays red on purpose. Draft +needs-live-evalstay.agentJobs.ts:2481-2485, A.0.10)FREE_AUTO_ALLOW_FILE_EGRESS_PROMOTION, default 0). Gating it behindallowPaidFallbackplus an awaited reservation is not possible inside a Convex mutation (NodeRoom has no per-job financial reservation there; the runner's primary dispatch reserves nothing, onlyspendLimits), and removing it would retire an operator opt-in that three test files pin. Options a/b/c are recorded as SPEC section C item 10. Disclosed consequence of D2:top_paidnow defaults to a free model, so atop_paidjob in a room with an uploaded file is blocked by the free-file-egress gate or, with the flag on, promoted to the paidconfiguredFileEgressModel()(defaultz-ai/glm-4.7-flash).claude-opus-4-1andclaude-haiku-3-5carrypricingAsOf: "2026-04", the month in the "official docs, April 2026" header of origin/main's block holding the dotted rows they were copied from (claude-opus-4.1,claude-haiku-3.5). Main gives only the month, so no day is invented.tests/modelPricingAgreement.test.tsasserts the date where it used to assert "undated".SpendMeter.droppedDispatchescounts requests pastMAX_DISPATCH_RECORDS; the ladder budget block carriesdispatchesDropped(non-zero meansrowsis incomplete: reject the receipt) and the spend line prints it. Rungcostis now the meter'sspentUsddelta over the rung, not the sum of the capped rows. Test (tests/modelCostClass.test.ts): 4,097 fake requests giverequests4097, 4096 rows,droppedDispatches1, andspentUsdexceeds the rows' sum by exactly one request. A one-request ladder run throughmain()with a fakefetch(only googleapis.com allowed, 1 call) wrotedispatchesDropped: 0and a rung cost equal to the row sum. It cannot change the 144-request receipt.src/ui/Chat.tsx:872)AGENT_TOP_PAID_MODELandAGENT_MODELunset, the code default is the freenvidia/nemotron-3.5-super-vl:free(D2), so a user who picks Top paid gets a free route. No copy was changed because the wording is a product decision.top_paidtargettests/modelRefreshRequestShape.test.tssendsDEFAULT_TOP_PAID_MODELthroughconvexModel().nextwith a fake transport: URL is OpenRouter chat completions,modelisnvidia/nemotron-3.5-super-vl:freeas written,tool_choiceisrequired, no sampling keys; the constant is also pinned to the literal.agentJobs.ts:2118; the "What" claim now carries the fail-closed qualifier.REA and ORX (SPEC-v9 D5), fetched 2026-10-16
REA reads the mechanisms this PR touches: an explicit alias table instead of a regex, a known-id registry that fails
closed, tiered per-request pricing, a spend meter. Facts, with what was adopted or rejected.
model_prices_and_context_window.json(raw.githubusercontent.com/BerriAI/litellm/main, 4,194 entries, 2026-10-16T14:44Z)gemini/gemini-3.8-flash,vertex_ai/gemini-3.8-flash,claude-haiku-5-5,bedrock_mantle/anthropic.claude-haiku-5-5). Tier prices are separate keys on the same row (input_cost_per_token_above_100k_tokens,*_above_200k_tokens,cache_read_input_token_cost_above_100k_tokens);claude-haiku-5-5carries 5e-07 / 2.5e-06 / 5e-08 above 100K.deprecation_dateis per row and differs by transport (azure_ai/claude-sonnet-52027-06-30,vertex_ai/claude-sonnet-52026-12-24, nativeclaude-sonnet-5null).max_output_tokensis 128000 forclaude-haiku-5-5.us.anthropic.*rows at 1.1x) and batch keys, which no NodeRoom path uses.https://models.dev/api.json(231 providers, 2026-10-16T14:44Z)anthropic/claude-haiku-5-5: cost 0.1 / 0.5,cache_read0.01,cache_write0.125, onetiersentry{type: "context", size: 100000}at 0.5 / 2.5 / 0.05 / 0.625;limit.output128000;release_date2026-10-07;tool_callandstructured_outputtrue. Resellers re-price the same model (poe/google/gemini-3.8-flashinput 0.7576 against 0.75 at the vendor).release_dateagrees with the spec.GET /api/v1/models(public, no key; 465 rows, 2026-10-16T11:32:04Z; earlier reads 2026-10-10 and 2026-10-14 in the spec)id,created,context_length, per-token stringpricing,supported_parameters(tools,tool_choice,structured_outputs) andexpiration_date; 18 of 465 rows carry anexpiration_date, including two of the newest:freerows.supported_parameters, theexpiration_dateexclusion andcreatedordering for D2; the per-token to per-1M conversion is already pinned bytests/modelPricingAgreement.test.ts.@ai-sdk/anthropic3.0.120 (installed source,dist/index.mjsgetModelCapabilities, read 2026-10-16)claude-haiku-5-5matches no 5.x branch (onlyclaude-opus-5-5,claude-opus-5andclaude-sonnet-5do), so it falls into the genericclaude-branch:maxOutputTokens128000,isKnownModel: false. The SDK reports the flag and still dispatches.isKnownModelidea, enforced instead of reported:resolveModelAliasthrowsUnknownModelError, andmodelMaxOutputTokenskeeps its own 2048 floor and caps for the 5.5 ids rather than trusting the SDK fallback.claude-haiku-5-5ended up in the fallback branch).ORX, primary sources behind the ids, prices and dates in this PR (ids and prices were not taken from recall):
frontier.json(read 2026-10-10, which citesplatform.claude.com/docs/en/models/overviewand the release notes); the Anthropic deprecations page re-read live 2026-10-14 (stored copy
_scratch/anthropic-deprecations-2026-10-14-v8.htmlin the model-defaults workspace):claude-haiku-4-5-20251001Active,floor 2026-10-15;
claude-sonnet-4-5-20250929deprecated, retires 2026-11-30.gemini-3.5-flashdeprecation (2026-10-08) andgemini-3.1-flash-tts-previewshutdown(2026-11-17):
frontier.json(read 2026-10-10), Google's deprecations table as quoted in SPEC 1.1.expiration_dateand the D2 selection: the public list above, recorded indocs/eval/openrouter-free-models-2026-10-16.json(all 15:freerows with the criteria columns).:freeroute and the Claude 5.5 ids over OpenRouter arerequest-shape tested against a fake transport only.
Pricing sources (A.0.6) - every source is enumerated and in
tests/modelPricingAgreement.test.tsmodelCatalog.modelPricing- the only hand-typed table. Every row this PR adds or relies on now carriespricingAsOf(2026-10-10), includingclaude-opus-4-7($5/$25, whichclaude-opus-4.7now resolves to) and is in the agreement cases.claude-haiku-5-5andanthropic/claude-haiku-5.5also carry the >100K-prompt tier ($0.50/$2.50) aslongContext.getModelPricing- accessor over (1).adapter.priceRequest,priceRequestStrict,reserveDispatch- tier-aware throughmodelCatalog.priceTokens, and ONLY for one request's prompt (the ladder meter uses the strict price and the reservation).adapter.priceRunprices usage summed over requests at the row's base tier (priceAggregateTokens): a sum of 60K + 60K prompts is not a 120K prompt.convexModel.convexPriceRequest,exactConvexUsageCost,convexAttemptCostReservationUsd- per-request, tier-aware;convexPriceRunis the base-tier aggregate.calculateRequestCost(useCachedInput)is per-request and reads the tier's own cache rate.creditModelprivatepriceRunandproofloopModelTrackingcost estimate - aggregate (base tier,priceAggregateTokens).scripts/benchmark/pricingAtRun.ts(moved out ofrun.tsin round 3, which runs the benchmark on import) -pricingAtRunForPREFERS the OpenRouter/modelssnapshot (priceFromOpenRouterInfo, dated by that run'sfetchedAt+routeSnapshotId, recorded on every row'spricingAtRun) and falls back to (1) throughcatalogPricingAtRun, then to a flaggedfallback_estimate($1/$5). It is not purely a catalog reader.scripts/proofloop-proxy-model-sweep.ts(:131/:238) - prices from the live OpenRouter catalog fetched when the sweep runs (datedfetchedAtper row); no fallback table.Chat.tsxprice chips - read (1), show the base tier.Round 3 (review defect 5): sources 6 and 7 are no longer excluded. Their per-token to per-1M conversion is code that can disagree with (1), and A.0.6 allows no exclusion.
tests/modelPricingAgreement.test.tsnow runs source 6 (pricingAtRunFor/catalogPricingAtRun) and source 7 (the sweep script itself in--dry-run, fakefetch) on a dated OpenRouter/modelsfixture (2026-10-10, literal per-token strings) and asserts{inPerM, outPerM}(and the cache-read rate) equal the catalog row for every OpenRouter-transport refresh row. Reverted to check:prompt * 1_000_000->* 100_000inpricingAtRun.ts, andparsed * 1_000_000->* 100_000in the sweep: each fails its agreement test (Codex's 10x undercharge now fails). The no-hand-typed-table check stays, now pointed atpricingAtRun.tsand assertingrun.tsimports it.The tier fix is behaviour, not just data: 120K input + the 8192-token cap now reserves $0.08048 (was $0.016096), and the pre-flight estimate prices each step by its prompt size.
Round 2 (review defect 2): the tier is chosen per REQUEST only (
reserveDispatch, meter settlement, runtimepriceStep=priceRequest/convexPriceRequest,calculateRequestCost,exactConvexUsageCost). Aggregated usage is priced at base-tier rates (priceRun,convexPriceRun,creditModel,proofloopModelTracking), and the ladder sumsmeter.dispatches[].costUsd(metered) or the runtime's per-stepusage.costUsd(scripted) instead of repricing summed usage (round 8: a metered rung's cost is the meter'sspentUsddelta).priceRun('claude-haiku-5-5', 120000, 2000)was $0.065 while two settled 60K/1K requests cost $0.013; a run of those two now reports $0.013.HAIKU_5_5_LONG_CONTEXTalso carries the vendor'scachedInputPer1M: 0.05, so cached tokens above 100K no longer bill at $0.50/M.Breaking-parameter fixes (spec section 1.3)
sdkToolChoiceForModel(adapter.ts),convexModel.openAiCompatibleToolChoice(OpenRouteranthropic/claude-*-5.5) and the LangChain bridge sendautoinstead of a forcedrequired/anyfor Sonnet/Opus 5.5 (and Fable 5.1); Haiku 5.5 keeps a forced choice. Keyed by exact wire id inmodelCatalog.forbidsForcedToolChoice. New this round: wherever that downgrade happens the tool definitions carrystrict: true(needsStrictTools: AI-SDK adapter, direct AnthropicanthropicStep, OpenRouteropenAiTool), which is SPEC 1.3's replacement for the forced choice. It is applied only on the downgrade, so no other path changes. Round 2: the LangChain bridge now passes{ strict: true }throughbindToolson the downgrade too (asserted in the LangChain request-shape test).modelMaxOutputTokens(id)(sharedsrc/nodeagent/models/outputTokens.ts): 2048 floor for 5.5 ids because thinking counts towardmax_tokens;anthropicStepreads content blocks by type (test).effort(haiku medium / sonnet high / opus medium) on every AI-SDK Anthropic path that moved to a 5.5 id: the adapter,capture/reasoning.ts,providerParserLive, and, new in round 2,scripts/benchmark/probe.tsandscripts/eval/providerParseSmoke.tsthrough one helper (claudeEffortProviderOptions);@ai-sdk/anthropic^3.0.84->^3.0.120.openAiTokenLimitregex also matchesgpt-6(precondition only; no gpt-6 id lands here).gemini-3.6-flashwithtemperatureremoved (deprecated on 3.6+);GEMINI_JUDGE_MODELenv chains unchanged.resolveModelAlias: the dotted-id regex rewrite is deleted for an explicit table;claude-opus-4.7now maps toclaude-opus-4-7(was silently-4-8). Theopusandclaude-opusshortcuts are pinned toclaude-opus-4-8(their previous effective target, behaviour preserved). Old 4.x strings stay valid for persisted rows.--realrefuses to start withoutLADDER_MAX_COST_USD/--max-cost-usd; job-wide meter;reserveDispatchinside everywithRetrydispatch (maxRetries: 0on the SDK, one reservation per physical request);priceRunStrictthrowsUnpricedModelErrorinstead of a guessed $1/$5;dispatch: { maxRetries, crossModelFallback }option. New this round: a refused reservation or unpriced id is terminal for the meter (the failed dispatch can no longer fall through toAGENT_FALLBACK_MODEL, a retry, or a free-auto candidate), and the provider-neutralnodeagent/free-autoGoogle fallback honourscrossModelFallback: false(it previously always went to paid Gemini).ladder:freegets--max-cost-usd=1because a cap is now mandatory. Round 2: the pre-flight request ceiling and USD estimate multiply a free-auto route byopenRouterFreeAutoLimit()(OPENROUTER_FREE_AUTO_LIMIT, default 8, max 20), the fan-out the adapter enforces, not by--free-auto-top.Test proof
New request-shape tests build a real request through the repo's own builders with a fake
fetch, asserting the literal wire id (never imported from the code under test) and that forbidden keys are absent:tests/modelRefreshRequestShape.test.ts-convexModel(direct Anthropic, OpenRouter),adapter.model,aiSdkReasoner, the LangChain bridge, pi-ai default; now alsostricttools and effort.tests/modelRefreshServerRoles.test.ts(new) - the roles that had no fake-transport test: Gemini TTS (drivesPOST /voice/synthesize), private stream (drivesPOST /stream-private-reply, nothing injected: the route's ownAGENT_MODEL ?? DEFAULT_AGENT_MODELreachesgeminiStream), provider parser (runLiveProviderParserwith no model argument, so the default candidate reaches the wire; effort and wire id for haiku and sonnet), judge scripts (rubric-panel, visual-polish, mp4-workflow-review, SEO judge and walkthroughs/judge-video are executed with fixtures and a fakefetch; round 2 also drivesjudge-demo-gif,judge-state-captures,spreadsheetbench-chart-visual-grade,gemini-demo-media-judge,gemini-ui-review,painJudge, the chart-visual probe, walkthrough-review (CLI default and shipped config),scripts/benchmark/probe.ts,providerParseSmokeand the startup-diligence provider eval through a fake transport; the ASTtemperaturechecks stay), and.env.exampleagainst the code defaults.tests/modelPricingAgreement.test.ts(A.0.6 agreement, tier-aware),tests/modelCatalogRefresh.test.ts.evals/ladderBudget.test.tsdrives the actual adapter path. Round 1 tightened (b) to the specified sequence (charged response, then ECONNRESET) and (e) to usage that stays within every reservation: the cap is the pre-flight estimate itself, the context outgrows the scripted dry run, and the test asserts the grid stops before the cap (spentUsd < maxUsd, a refused reservation,budget.requestsequals the fetch count and is below the ceiling). Also covered: terminal-stop behaviour, per-dispatch receipt rows (echoedModel,fallbackUsed) and the report fields.tests/modelRefreshAgentWire.test.ts(new, round 5) -runRoomAgent(public_ask),runPrivateAgentandrunFreeAutoJobSlice(claimed job with nomodelPolicy) run through their real handlers on a convex-test database with a fakefetch: the model in the request URL isgemini-3.8-flash(or theAGENT_MODELoverride) and no sampling key is sent; an unknownAGENT_MODELsends 0 requests. Mutation check run:modelNameForEgressalways returning the file-egress model and the runner default changed togemini-3.5-flashfail 5 of its 6 tests.tests/modelRefreshCallSites.test.ts(round 2; round 5 correction) - the round-2 claim that these sites were "not reachable without a live Convex deployment" was wrong (convex-test drives the handlers; see the wire test above). This file now only pins where the default is spelled for the sites the wire tests do not drive (streaming.tsis driven bytests/modelRefreshServerRoles.test.ts,Chat.tsxis a React closure): the expression that owns the default;tests/qaMatrix.test.tsnow checks EVERYAGENT_MODEL ?? ...fallback inconvex/agent.ts, not the first. Re-inlininggemini-3.5-flashat each of the five fails a named test.tests/modelAdapterProviderNeutralRoute.test.ts-crossModelFallback: falsenever reaches paid Gemini.Each new assertion was checked by reverting the line it protects (TTS constant,
DEFAULT_AGENT_MODELand each of its five call sites, a judge script'stemperature/model, the parser default and effort, strict tools, the tier, the terminal stop, the fallback guard): the matching test fails every time. Round 3 correction: that claim did not hold for thescripts/benchmark/probe.tsgpt-6.1-solrow (the test answered OpenAI with a 404 and asserted nothing about the request; Codex swapped the id and it stayed green). It is now asserted on the wire (POST https://api.openai.com/v1/responses,body.model === "gpt-6.1-sol", no sampling keys, no tools) and reverting the id togpt-6.1-solxfails it. The round-3 reversions (cost-class guards in the adapter and inconvexModel, the resolver throw, the failed-rung cost, the alias fan-out, both pricing conversions, the probe id) were each run and each fails its named test.Check output (
npm run check= typecheck + convex tsc + vitest)Round 6, head
74c20c49(the commit that contains the code; a later body-only edit does not change it).tsc --noEmitexit 0,convex/tsconfig.jsonexit 0, then the fullvitest rununder machine load: 12 failed / 3215 passed (3227), 398 files. Two are deterministic and not caused by this PR:tests/qaMatrix.test.ts(the ladder receipt blocker above; fails on purpose, the check is not weakened) andtests/proofStaleness.test.ts(docs/eval/noderoom-fresh-user-vertical-proof.jsonis 47.9 days old against a 30-day window; it also fails onorigin/main). The other ten are 5 s to 30 s timeouts under load (ladderBudgetx3,createRoomAtomicity,dockerSandboxProbe,notebookProcessingTarget,proofloopHooks,proofloopOrchestratorx2,proofloopPromoteOfficialScore). Re-run serially with--no-file-parallelism --testTimeout=300000, nine of them pass (every test in the six other files).proofloopPromoteOfficialScorehas its own 15 s timeout, which the flag does not override; it passes alone in 14.1 s (12.4 s with this PR stashed, so it is close to its limit on this machine).walkthroughCitationspasses with the.toursline moved to:2799.Round 7, head
4b2af806.tsc --noEmitexit 0,convex/tsconfig.jsonexit 0, fullvitest run: 2 failed / 3230 passed (3232), 398 files, no load timeouts this run. Both failures are the known ones:tests/qaMatrix.test.ts(the ladder receipt gate; fails on purpose, not weakened) andtests/proofStaleness.test.ts(also fails onorigin/main). Mutation check run and reverted: the startup-diligencemodelPolicyline (the new test fails: 2 requests). No paid provider call was made.Mutation checks run and reverted (each fails its named tests): the startup-diligence
operatorSelectedargument (1 test), theconvexModeloption branch (2), the capture reasoner's resolved-id dispatch and route refusal (5), the streaming resolved-id routing (5), the two dotted aliases plusopenrouter/auto(5), and, for the boundary, theknownModelPolicywrapping of the env defaults and the promotion (all 8 new cases). No paid provider call was made.Check output, round 8
Head
8dcdb2ce.npm run check:tsc --noEmitandconvex/tsconfig.jsonpass; vitest 2 failed / 3233 passed (3235), 398 files. The two failures are the known ones:tests/qaMatrix.test.ts(ladder receipt, fails on purpose, not weakened) andtests/proofStaleness.test.ts(also fails onorigin/main). No paid provider call was made.Not in this PR / for the human
backend/app/config.py:78-80,llm.py:37on GMI MaaS): defer - listGET {GMI_MAAS_BASE_URL}/models, confirm the ids, run the backend pytest.src/landing/modelRegistry.ts:48(anthropic/claude-fable-5): defer until the forced-tool-choice downgrade is observed live (no key exists, so this stays deferred; D2 does not move it).README.md:1604andadaptiveRouter.ts:136: move togemini-3.8-flashwith the ladder receipt.gpt-5-mini(retires 2026-12-11, URGENT decision),google/gemini-3-flash-preview, benchmark judgegemini-3.1-flash-lite(2027-05-07).undiciinpackage.json/package-lock.jsonand PR fix(evals): reject unconfigured real-mode runs #260 ownsevals/runEval.ts; this PR touches neither. The lockfile change here adds a nested@ai-sdk/anthropic/node_modules/undici6.29.0 entry; re-runnpm ciafter whichever PR merges second.Spec: model-defaults SPEC Part A section A.2 (NodeRoom)
Review round 9 (against
8dcdb2ce)gemini-3.8-flash,gemini-3.6-flash,gemini-3.5-flash-litechain keys;configuredConvexModelFallbacks('gemini-3.8-flash', {})is[]again as on main. Test pins the three ids chain-less; the "has a chain" assertion covers the 5.5 ids only.droppedDispatches > 0exits 1 likestoppedBy. No dedicated test (needs 4,097 real requests throughmain()); the 144-request receipt cannot reach it.Check at round 9: tsc and convex tsc pass; full vitest 3234 passed, 2 failed (the known
qaMatrixgemini-3.8-flash-not-in-ladder, red until the paid receipt exists, andproofStaleness, red on main). No paid calls made; paid ladder receipt and TTS listen check remain merge blockers.