getainode · webdevtodayjason · Sep 22, 2026 · Sep 22, 2026
diff --git a/.gitignore b/.gitignore
@@ -69,3 +69,9 @@ node_modules/
 
 # graft's local graph cache — regenerable, not committed (run `graft build`).
 /graft/
+
+# The decision bench's question sets: item text fetched by `ainode-bench decide
+# download` and verified against the committed manifests' state hashes. Never committed,
+# because upstream does not republish item text either and the upstream licences are not
+# ours to relicense (bench/decide/sets/README.md).
+bench/decide/cache/
diff --git a/AGENTS.md b/AGENTS.md
@@ -24,7 +24,8 @@ State / architecture / decisions / "why": Obsidian Vault, `AINode` (cluster ops:
 - **Every request a bench run makes carries the node's API key, one helper puts it there, and a refusal stops the section instead of scoring it** (`ainode/bench/auth.py`). Every section takes `--api-key` and falls back to `$AINODE_API_KEY` before its own `ainode` placeholder, and `auth.bearer(key)` is the ONLY spelling of the header in the package, so a new transport either passes a key or passes nothing. The key is never printed, never written into a record and never in a dry run's environment line: a run reports the SOURCE (`--api-key`, `$AINODE_API_KEY`, `the default`) and nothing else. A 401 answers "this node wants an API key (pass --api-key or set AINODE_API_KEY)" and a 429 names the limit that refused it; both stop before the first score and write no file, because a refusal recorded as ten model failures is the #153 false zero in a new place (out-of-process bench runs 401'd silently once a fresh install started requiring a key, #245). The hosted decision backend is exempt from the raise (`Backend.local = False`): its 401 is TypeSafe's, and its key resolution is a separate function so neither party's credential can reach the other. `tests/test_bench_auth.py` WALKS THE SOURCE for both halves, and `tests/conftest.py::no_bench_preflight` keeps the opening GET inside the suite.
 - **The harness bench (`ainode/bench/harness/`, `scripts/ainode-bench.py harness`) must never let a harness see the hidden tests.** `bench/harness/tasks/<slug>/tests/` is copied into the working directory only after the agent CLI has exited and removed again before the next attempt; a task with a `*_test.py` at its root is a load error. Those vendored files are also excluded from pytest collection (`norecursedirs` in `pyproject.toml`) and from ruff (`extend-exclude`) because they import a module that only exists inside a run, and because they are upstream's text kept verbatim. Adapter `command()` / `env()` / `config()` stay pure functions of the request so `tests/test_bench_harness.py` can pin every harness's exact argv; `run()` lives once in the base class. **An adapter for an agent that keeps state under `$HOME` points it at the run's own directory** (`DSH_HOME`, the XDG vars, `CLAUDE_CONFIG_DIR`): a bench run never reads or writes the operator's own agent profile, both so runs cannot poison each other and so a personal `settings.json` full of hooks is not inside the measurement. Adapters and flags: `bench/harness/README.md`.
 - **The agentic rubric (`ainode/bench/agentic/`, `scripts/ainode-bench.py agentic`) scores on mechanical verdicts only.** No judge model and no pass read by eye: a probe is decided by a regex, a parsed tool call, an executed subprocess or a compared call trace. Every checker is a module-level function of plain values (the reply text, the calls, the delivered tool results) so `tests/test_bench_agentic.py` can canned-response all of them with no network, and a probe that raises is one recorded failure rather than a dead run. **Group C executes model-written code on the machine driving the bench** (temporary directory, 60-second timeout, `sys.executable`), the same trade the harness bench makes when it runs an agent's edit. A run writes an `agentic` block and no `results` block, and a group `--groups` or `--quick` left out is absent from the score, never a zero. New rubric numbers go in that block: the hand-typed `rubric` key in three older records is a historical claim and must not be added to a new record. Probes and flags: `bench/agentic/README.md`.
-- **The decision bench (`ainode/bench/decide/`, `scripts/ainode-bench.py decide`) scores typed decisions against labels, and its confidence numbers are the product.** Accuracy is the weakest number in the block: a wrong answer at 0.95 is the failure mode, so every block carries the Brier score on the labeled option, an expected calibration error with the five-bin reliability table behind it, and the count of wrong answers surviving a 0.8 and a 0.9 gate. **A probability nobody reported is absent, never assumed** (such a row is in the accuracy, out of the calibration, and counted in `no_confidence`), **an item that failed is one row with an `error`** and never a wrong answer, and **cost is a posted vendor rate over reported tokens or `0`** for a local backend, never an estimate. `bench/decide/items.json` is repo data versioned next to its results: loading is strict, a malformed item is a load error rather than a skipped item, and a set's items must share one kind, question and option set because a set is one measurement. Backends are split `request()` / `parse()` as pure functions so `tests/test_bench_decide.py` pins every request shape and response shape with canned payloads and no network. **The TypeSafe key is never printed, never written into a record and never put in a note**: a run reports only which of `--api-key`, `$TYPESAFE_API_KEY` or `~/.jev_api_key` it came from. Sets, metrics and flags: `bench/decide/README.md`.
+- **The decision bench runs TWO measurements under one subcommand, and a record says which one produced it.** The **Jevals recipe** (`ainode/bench/decide/{jevals,sets,suite}.py`, `--suite`/`--questions` plus `--transport decide|systemone`) scores the public question sets the independent boards use, with their formulas, so an AINode-served model can be read next to Jev and its clones; the **legacy 110-item path** (`--backend`) is the bullet below. Mixing the two flag sets is an error rather than a guess. **`bench/decide/JEVALS.md` is authoritative for what a recipe number means**: it records every formula against the page and the date it was read (jevals.com/methodology and jevals.com/policy, JevBench's README, the typed-decisions card, all 2026-09-21) and names every deviation, and a formula change here is a change there. Rules with no exemption: **the model is never shown the answer** (labels live in a question file's separate `labels` map, a question carrying an answer-key-shaped field is a load error, and `suite.wire_leaks` re-checks the assembled body and REFUSES to send rather than producing a flattering score, skipping only the caller's own `state`); **every state is verified against its `state_sha256` on load**, which is what makes a number comparable to a board number and is how the three Jevals state constructions were recovered in the first place (`{"message": ...}` for Banking77, not `text`; `{"question": ..., "context": [...]}` for PubMedQA, not `context.contexts`); **item text is never committed**, only the manifests under `bench/decide/sets/`, because upstream does not republish it either and the upstream licences are not ours to relicense (`decide download` writes the gitignored `bench/decide/cache/`); **every metrics block names its recipe** (`recipe`, `recipe_of_record`, `probability_source` of `logprob`/`native`/`verbalized`) because two boards use one word for different arithmetic; **a set holding more than one ANSWER SPACE is broken down by answer space** (one `(type, options)` pair) with a per-primitive roll-up, and its Decision Score is the mean over the spaces, because the label prior is the base rates of the labels in ONE option list and pooling two of them builds the baseline that defines 0 over an answer space neither question has (`typed-decisions` asks `action` with four options in one workflow and five in another, so this is a wrong number and not a presentation choice); **a set in a listed system's published training data carries that finding with its primary source** (`sets.CONTAMINATION`); and **`decide.overall` on such a record carries no `brier`, `ece`, `bins` or `thresholds`**, because those names mean the legacy definitions and the table renders them "not measured" rather than borrowing a number computed another way. `tests/test_bench_decide_jevals.py` pins the three anchors of the scale against hand arithmetic (a calibrated system that is exactly the prior scores 0, a confidently wrong one scores negative, a perfect one scores 100), both transports against a loopback server, and that no wire body ever carried an answer key.
+- **The legacy 110-item decision path (`--backend ainode|chat|jev`) scores typed decisions against labels, and its confidence numbers are the product.** Accuracy is the weakest number in the block: a wrong answer at 0.95 is the failure mode, so every block carries the Brier score on the labeled option, an expected calibration error with the five-bin reliability table behind it, and the count of wrong answers surviving a 0.8 and a 0.9 gate. **A probability nobody reported is absent, never assumed** (such a row is in the accuracy, out of the calibration, and counted in `no_confidence`), **an item that failed is one row with an `error`** and never a wrong answer, and **cost is a posted vendor rate over reported tokens or `0`** for a local backend, never an estimate. `bench/decide/items.json` is repo data versioned next to its results: loading is strict, a malformed item is a load error rather than a skipped item, and a set's items must share one kind, question and option set because a set is one measurement. Backends are split `request()` / `parse()` as pure functions so `tests/test_bench_decide.py` pins every request shape and response shape with canned payloads and no network. **The TypeSafe key is never printed, never written into a record and never put in a note**: a run reports only which of `--api-key`, `$TYPESAFE_API_KEY` or `~/.jev_api_key` it came from. Sets, metrics and flags: `bench/decide/README.md`.
 - **The speech bench (`ainode/bench/speech/`, `scripts/ainode-bench.py speech`) scores against committed audio, and both halves of that are load-bearing.** A word error rate is only comparable over the same bytes, so the ten clips live in `bench/speech/clips/` as repo data (1.4 MB) rather than being synthesised per run, and **`clips.CLIPS_VERSION` is bumped on any edit to a text, a voice or a WAV**; `--generate-clips` rebuilds the set with macOS `say` plus `afconvert` and is a maintenance step a run never takes. The reference is the exact string handed to `say`, fixed before the run, and **nothing adjusts a reference after a transcript is seen**: a reference edited to match what a model said makes the rate a statement about the editor. The normaliser is part of the measurement, so it is versioned (`metrics.NORMALIZER_VERSION`) and the record carries BOTH rates, `wer` with number words folded to digits and `wer_orthographic` with case and punctuation only, because a transcript that heard every word and wrote "9" for "nine" is not a hearing error and one number alone hides which kind it was. Nothing is folded that changes a word: no stopword list, no stemming, no synonym map, no per-clip exception. `wer` is pooled over words, never a mean of per-clip rates. **A clip that failed is one row with an `error` and nulls for every number**, counted out of every rate, percentile and factor, never folded in as a 100 percent error rate: a transport failure inside a figure a reader takes as the model's is the one mistake this section can make. It is the one bench whose request body is not JSON (`client.py` assembles the multipart itself), so a run through a node's `:3000/v1` exercises the fleet's own audio path; the block is `bench/SCHEMA.md`.
 - **One proxy handler serves every forwarded inference path**: `proxy_to_vllm` is registered for `POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`, `/v1/messages/count_tokens`, `/v1/responses`, `/v1/rerank`, `/v1/score`, `/v1/audio/transcriptions`, `/v1/audio/translations`, `/tokenize` and `/detokenize` (`GET /v1/models` is the federated union and does not forward; `POST /v1/embeddings` keeps its own handler because it validates the body first, and `/v1/decide` plus `/v1/systemone` compose their own completions). Add a path by registering it on that handler, never by writing a second proxy: routing on the body's `model`, transport failover, the multimodal ordering below, SSE passthrough and header passthrough are all protocol-agnostic and already there. Two of the paths are NOT under `/v1` because vLLM does not serve them there (`/tokenize`, `/detokenize`), so the table is the authority on the path and not a prefix rule. **The two audio paths are the ones whose body is NOT JSON**: OpenAI's speech-to-text API is a `multipart/form-data` upload with the model id as a form field, so the handler reads it with `api/multipart.py::form_fields` over the body it already buffered (never `request.multipart()`, which consumes the stream the proxy still has to forward) and forwards the bytes UNCHANGED under the caller's own `Content-Type`: a multipart body is only parseable against the boundary in its own header, so re-encoding the parts hands the engine a body the forwarded header no longer describes. A multipart body that names no `model` is a 400 naming the field, never a fallback to this node's own model, which would send someone's audio to a chat engine. They are also the first pair whose existence is MODEL-CONDITIONAL: vLLM attaches its speech-to-text router only when the served model reports the `transcription` task, so no chat or pooling engine's `/openapi.json` lists them and the check below cannot be run on one. For a path like that, the evidence is the router's own declaration in the engine image the recipe pins (`entrypoints/openai/speech_to_text/api_router.py`), and the `openapi.json` check still applies the first time such a model actually serves. Forward `request.path_qs`, not `request.path`: Claude Code posts to `/v1/messages?beta=true`. This is a route table and not a catch-all: an unregistered path stays a 404, which is why a new path is added only after `curl http://<node>:<port>/openapi.json` on a real engine says the engine answers it.
 - **`/v1/decide`'s response shape is a contract, not an implementation detail** (`api/decide.py`). Its bench is written against the exact shape (`model`, `node`, `latency_ms`, `decisions[key] = {answer, confidence, distribution, latency_ms}`, `usage = {prompt_tokens, completion_tokens, calls}`), so a `200` always carries every question asked: a bad request is a `400` and an engine that cannot answer is a `503`, never a partial `decisions` block. Probabilities are keyed by the caller's OPTION strings, never by the letters used to constrain the engine. It is the one `/v1` path deliberately NOT on `proxy_to_vllm`, because it composes N grammar-constrained chat completions of its own from one request and has no caller body to forward; it still routes through the proxy's own `_routing_candidates` and the shared `app["client_session"]`, so never give it its own routing rule or HTTP stack. The constraint field is vLLM 0.27.1's `structured_outputs: {"choice": [...]}`: the legacy `guided_choice` is accepted by that image and then silently ignored, so sending it instead would produce free prose with no error. The engine-facing half of a decision request is `decide.py::run_questions`, shared with `/v1/systemone`: one path to the engines and one way the probabilities are read, so a route decides the status and the shape and nothing else.

diff --git a/README.md b/README.md
@@ -637,10 +637,10 @@ sets and the protocol are in [`bench/decide/README.md`](bench/decide/README.md).
 
 <!-- BEGIN decide-bench-table (generated by scripts/render-bench-table.py) -->
 
-| Backend/Model | Placement | Items | Accuracy | Brier | ECE | Wrong at 0.9 | p50 ms | Cost | Date | Run |
-|---|---|---|---|---|---|---|---|---|---|---|
-| chat / Ornith 1.5 35B-A3B | Spark-1-DGX, TP=1 | 110 | 0.964 | 0.027 | 0.023 | 2 of 106 | 659 | $0 | 2026-09-18 | [Ornith stacked Spark-1, chat fallback](https://github.com/getainode/ainode/blob/main/bench/results/20260918-030249-ornith-1_5-35b-a3b-nvfp4-ornith-stacked-spark-1-chat-fallback-decide.json) |
-| jev / jev-1.13.0 | typesafe.ai hosted | 110 | 0.964 | 0.024 | 0.049 | 0 of 80 | 306 | $0.0016 | 2026-09-18 | [jev-latest, 110 items](https://github.com/getainode/ainode/blob/main/bench/results/20260918-030240-jev-1_13_0-jev-latest-110-items-decide.json) |
+| Backend/Model | Placement | Items | Accuracy | Decision Score | Brier | ECE | Hand-off at 95% | Wrong at 0.9 | p50 ms | Cost | Date | Run |
+|---|---|---|---|---|---|---|---|---|---|---|---|---|
+| chat / Ornith 1.5 35B-A3B | Spark-1-DGX, TP=1 | 110 | 0.964 | not measured | 0.027 | 0.023 | not measured | 2 of 106 | 659 | $0 | 2026-09-18 | [Ornith stacked Spark-1, chat fallback](https://github.com/getainode/ainode/blob/main/bench/results/20260918-030249-ornith-1_5-35b-a3b-nvfp4-ornith-stacked-spark-1-chat-fallback-decide.json) |
+| jev / jev-1.13.0 | typesafe.ai hosted | 110 | 0.964 | not measured | 0.024 | 0.049 | not measured | 0 of 80 | 306 | $0.0016 | 2026-09-18 | [jev-latest, 110 items](https://github.com/getainode/ainode/blob/main/bench/results/20260918-030240-jev-1_13_0-jev-latest-110-items-decide.json) |
 
 <!-- END decide-bench-table -->