From 475f42a2d3e60d2e82c7f23a9e4dfeb1e82cbf04 Mon Sep 17 00:00:00 2001 From: webdevtodayjason Date: Mon, 21 Sep 2026 20:04:56 -0500 Subject: [PATCH] Score decisions on the public Jevals sets, with the boards' own formulas The decision bench could only score AINode's own 110-item set, so an AINode-served model had no number anybody outside this repo could read. This adds the independent recipe as a second measurement under the same subcommand: the public question sets the Jevals boards use, their formulas, five repeats per question, and two transports so the same run can target /v1/decide or any server speaking the Jev /v1/systemone wire format. The recipe is recorded in bench/decide/JEVALS.md against the pages it came from and the date they were read, with every deviation named. Four manifests carry the sets without any item text; `decide download` fetches the text and verifies every state against its published hash, which is also how the three Jevals state constructions were recovered (Banking77's state key is `message`, not `text`, and PubMedQA's is `context` holding the list). Co-Authored-By: Claude Fable 5.1 --- .gitignore | 6 + AGENTS.md | 3 +- README.md | 8 +- ainode/bench/decide/__init__.py | 54 +- ainode/bench/decide/cli.py | 329 +++- ainode/bench/decide/jevals.py | 1193 ++++++++++++ ainode/bench/decide/sets.py | 740 ++++++++ ainode/bench/decide/suite.py | 1051 +++++++++++ bench/SCHEMA.md | 177 ++ bench/decide/JEVALS.md | 540 ++++++ bench/decide/README.md | 143 +- bench/decide/sets/README.md | 70 + bench/decide/sets/banking77.json | 2059 +++++++++++++++++++++ bench/decide/sets/helpsteer2.json | 1844 +++++++++++++++++++ bench/decide/sets/pubmedqa.json | 1835 ++++++++++++++++++ bench/decide/sets/typed-decisions.json | 2348 ++++++++++++++++++++++++ scripts/render-bench-table.py | 83 +- tests/test_bench_decide.py | 25 +- tests/test_bench_decide_jevals.py | 1722 +++++++++++++++++ 19 files changed, 14196 insertions(+), 34 deletions(-) create mode 100644 ainode/bench/decide/jevals.py create mode 100644 ainode/bench/decide/sets.py create mode 100644 ainode/bench/decide/suite.py create mode 100644 bench/decide/JEVALS.md create mode 100644 bench/decide/sets/README.md create mode 100644 bench/decide/sets/banking77.json create mode 100644 bench/decide/sets/helpsteer2.json create mode 100644 bench/decide/sets/pubmedqa.json create mode 100644 bench/decide/sets/typed-decisions.json create mode 100644 tests/test_bench_decide_jevals.py diff --git a/.gitignore b/.gitignore index a616c8db..7287a16c 100644 --- a/.gitignore +++ b/.gitignore @@ -69,3 +69,9 @@ node_modules/ # graft's local graph cache — regenerable, not committed (run `graft build`). /graft/ + +# The decision bench's question sets: item text fetched by `ainode-bench decide +# download` and verified against the committed manifests' state hashes. Never committed, +# because upstream does not republish item text either and the upstream licences are not +# ours to relicense (bench/decide/sets/README.md). +bench/decide/cache/ diff --git a/AGENTS.md b/AGENTS.md index 0c906e59..b7b61b78 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,8 @@ State / architecture / decisions / "why": Obsidian Vault, `AINode` (cluster ops: - **Every request a bench run makes carries the node's API key, one helper puts it there, and a refusal stops the section instead of scoring it** (`ainode/bench/auth.py`). Every section takes `--api-key` and falls back to `$AINODE_API_KEY` before its own `ainode` placeholder, and `auth.bearer(key)` is the ONLY spelling of the header in the package, so a new transport either passes a key or passes nothing. The key is never printed, never written into a record and never in a dry run's environment line: a run reports the SOURCE (`--api-key`, `$AINODE_API_KEY`, `the default`) and nothing else. A 401 answers "this node wants an API key (pass --api-key or set AINODE_API_KEY)" and a 429 names the limit that refused it; both stop before the first score and write no file, because a refusal recorded as ten model failures is the #153 false zero in a new place (out-of-process bench runs 401'd silently once a fresh install started requiring a key, #245). The hosted decision backend is exempt from the raise (`Backend.local = False`): its 401 is TypeSafe's, and its key resolution is a separate function so neither party's credential can reach the other. `tests/test_bench_auth.py` WALKS THE SOURCE for both halves, and `tests/conftest.py::no_bench_preflight` keeps the opening GET inside the suite. - **The harness bench (`ainode/bench/harness/`, `scripts/ainode-bench.py harness`) must never let a harness see the hidden tests.** `bench/harness/tasks//tests/` is copied into the working directory only after the agent CLI has exited and removed again before the next attempt; a task with a `*_test.py` at its root is a load error. Those vendored files are also excluded from pytest collection (`norecursedirs` in `pyproject.toml`) and from ruff (`extend-exclude`) because they import a module that only exists inside a run, and because they are upstream's text kept verbatim. Adapter `command()` / `env()` / `config()` stay pure functions of the request so `tests/test_bench_harness.py` can pin every harness's exact argv; `run()` lives once in the base class. **An adapter for an agent that keeps state under `$HOME` points it at the run's own directory** (`DSH_HOME`, the XDG vars, `CLAUDE_CONFIG_DIR`): a bench run never reads or writes the operator's own agent profile, both so runs cannot poison each other and so a personal `settings.json` full of hooks is not inside the measurement. Adapters and flags: `bench/harness/README.md`. - **The agentic rubric (`ainode/bench/agentic/`, `scripts/ainode-bench.py agentic`) scores on mechanical verdicts only.** No judge model and no pass read by eye: a probe is decided by a regex, a parsed tool call, an executed subprocess or a compared call trace. Every checker is a module-level function of plain values (the reply text, the calls, the delivered tool results) so `tests/test_bench_agentic.py` can canned-response all of them with no network, and a probe that raises is one recorded failure rather than a dead run. **Group C executes model-written code on the machine driving the bench** (temporary directory, 60-second timeout, `sys.executable`), the same trade the harness bench makes when it runs an agent's edit. A run writes an `agentic` block and no `results` block, and a group `--groups` or `--quick` left out is absent from the score, never a zero. New rubric numbers go in that block: the hand-typed `rubric` key in three older records is a historical claim and must not be added to a new record. Probes and flags: `bench/agentic/README.md`. -- **The decision bench (`ainode/bench/decide/`, `scripts/ainode-bench.py decide`) scores typed decisions against labels, and its confidence numbers are the product.** Accuracy is the weakest number in the block: a wrong answer at 0.95 is the failure mode, so every block carries the Brier score on the labeled option, an expected calibration error with the five-bin reliability table behind it, and the count of wrong answers surviving a 0.8 and a 0.9 gate. **A probability nobody reported is absent, never assumed** (such a row is in the accuracy, out of the calibration, and counted in `no_confidence`), **an item that failed is one row with an `error`** and never a wrong answer, and **cost is a posted vendor rate over reported tokens or `0`** for a local backend, never an estimate. `bench/decide/items.json` is repo data versioned next to its results: loading is strict, a malformed item is a load error rather than a skipped item, and a set's items must share one kind, question and option set because a set is one measurement. Backends are split `request()` / `parse()` as pure functions so `tests/test_bench_decide.py` pins every request shape and response shape with canned payloads and no network. **The TypeSafe key is never printed, never written into a record and never put in a note**: a run reports only which of `--api-key`, `$TYPESAFE_API_KEY` or `~/.jev_api_key` it came from. Sets, metrics and flags: `bench/decide/README.md`. +- **The decision bench runs TWO measurements under one subcommand, and a record says which one produced it.** The **Jevals recipe** (`ainode/bench/decide/{jevals,sets,suite}.py`, `--suite`/`--questions` plus `--transport decide|systemone`) scores the public question sets the independent boards use, with their formulas, so an AINode-served model can be read next to Jev and its clones; the **legacy 110-item path** (`--backend`) is the bullet below. Mixing the two flag sets is an error rather than a guess. **`bench/decide/JEVALS.md` is authoritative for what a recipe number means**: it records every formula against the page and the date it was read (jevals.com/methodology and jevals.com/policy, JevBench's README, the typed-decisions card, all 2026-09-21) and names every deviation, and a formula change here is a change there. Rules with no exemption: **the model is never shown the answer** (labels live in a question file's separate `labels` map, a question carrying an answer-key-shaped field is a load error, and `suite.wire_leaks` re-checks the assembled body and REFUSES to send rather than producing a flattering score, skipping only the caller's own `state`); **every state is verified against its `state_sha256` on load**, which is what makes a number comparable to a board number and is how the three Jevals state constructions were recovered in the first place (`{"message": ...}` for Banking77, not `text`; `{"question": ..., "context": [...]}` for PubMedQA, not `context.contexts`); **item text is never committed**, only the manifests under `bench/decide/sets/`, because upstream does not republish it either and the upstream licences are not ours to relicense (`decide download` writes the gitignored `bench/decide/cache/`); **every metrics block names its recipe** (`recipe`, `recipe_of_record`, `probability_source` of `logprob`/`native`/`verbalized`) because two boards use one word for different arithmetic; **a set holding more than one ANSWER SPACE is broken down by answer space** (one `(type, options)` pair) with a per-primitive roll-up, and its Decision Score is the mean over the spaces, because the label prior is the base rates of the labels in ONE option list and pooling two of them builds the baseline that defines 0 over an answer space neither question has (`typed-decisions` asks `action` with four options in one workflow and five in another, so this is a wrong number and not a presentation choice); **a set in a listed system's published training data carries that finding with its primary source** (`sets.CONTAMINATION`); and **`decide.overall` on such a record carries no `brier`, `ece`, `bins` or `thresholds`**, because those names mean the legacy definitions and the table renders them "not measured" rather than borrowing a number computed another way. `tests/test_bench_decide_jevals.py` pins the three anchors of the scale against hand arithmetic (a calibrated system that is exactly the prior scores 0, a confidently wrong one scores negative, a perfect one scores 100), both transports against a loopback server, and that no wire body ever carried an answer key. +- **The legacy 110-item decision path (`--backend ainode|chat|jev`) scores typed decisions against labels, and its confidence numbers are the product.** Accuracy is the weakest number in the block: a wrong answer at 0.95 is the failure mode, so every block carries the Brier score on the labeled option, an expected calibration error with the five-bin reliability table behind it, and the count of wrong answers surviving a 0.8 and a 0.9 gate. **A probability nobody reported is absent, never assumed** (such a row is in the accuracy, out of the calibration, and counted in `no_confidence`), **an item that failed is one row with an `error`** and never a wrong answer, and **cost is a posted vendor rate over reported tokens or `0`** for a local backend, never an estimate. `bench/decide/items.json` is repo data versioned next to its results: loading is strict, a malformed item is a load error rather than a skipped item, and a set's items must share one kind, question and option set because a set is one measurement. Backends are split `request()` / `parse()` as pure functions so `tests/test_bench_decide.py` pins every request shape and response shape with canned payloads and no network. **The TypeSafe key is never printed, never written into a record and never put in a note**: a run reports only which of `--api-key`, `$TYPESAFE_API_KEY` or `~/.jev_api_key` it came from. Sets, metrics and flags: `bench/decide/README.md`. - **The speech bench (`ainode/bench/speech/`, `scripts/ainode-bench.py speech`) scores against committed audio, and both halves of that are load-bearing.** A word error rate is only comparable over the same bytes, so the ten clips live in `bench/speech/clips/` as repo data (1.4 MB) rather than being synthesised per run, and **`clips.CLIPS_VERSION` is bumped on any edit to a text, a voice or a WAV**; `--generate-clips` rebuilds the set with macOS `say` plus `afconvert` and is a maintenance step a run never takes. The reference is the exact string handed to `say`, fixed before the run, and **nothing adjusts a reference after a transcript is seen**: a reference edited to match what a model said makes the rate a statement about the editor. The normaliser is part of the measurement, so it is versioned (`metrics.NORMALIZER_VERSION`) and the record carries BOTH rates, `wer` with number words folded to digits and `wer_orthographic` with case and punctuation only, because a transcript that heard every word and wrote "9" for "nine" is not a hearing error and one number alone hides which kind it was. Nothing is folded that changes a word: no stopword list, no stemming, no synonym map, no per-clip exception. `wer` is pooled over words, never a mean of per-clip rates. **A clip that failed is one row with an `error` and nulls for every number**, counted out of every rate, percentile and factor, never folded in as a 100 percent error rate: a transport failure inside a figure a reader takes as the model's is the one mistake this section can make. It is the one bench whose request body is not JSON (`client.py` assembles the multipart itself), so a run through a node's `:3000/v1` exercises the fleet's own audio path; the block is `bench/SCHEMA.md`. - **One proxy handler serves every forwarded inference path**: `proxy_to_vllm` is registered for `POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`, `/v1/messages/count_tokens`, `/v1/responses`, `/v1/rerank`, `/v1/score`, `/v1/audio/transcriptions`, `/v1/audio/translations`, `/tokenize` and `/detokenize` (`GET /v1/models` is the federated union and does not forward; `POST /v1/embeddings` keeps its own handler because it validates the body first, and `/v1/decide` plus `/v1/systemone` compose their own completions). Add a path by registering it on that handler, never by writing a second proxy: routing on the body's `model`, transport failover, the multimodal ordering below, SSE passthrough and header passthrough are all protocol-agnostic and already there. Two of the paths are NOT under `/v1` because vLLM does not serve them there (`/tokenize`, `/detokenize`), so the table is the authority on the path and not a prefix rule. **The two audio paths are the ones whose body is NOT JSON**: OpenAI's speech-to-text API is a `multipart/form-data` upload with the model id as a form field, so the handler reads it with `api/multipart.py::form_fields` over the body it already buffered (never `request.multipart()`, which consumes the stream the proxy still has to forward) and forwards the bytes UNCHANGED under the caller's own `Content-Type`: a multipart body is only parseable against the boundary in its own header, so re-encoding the parts hands the engine a body the forwarded header no longer describes. A multipart body that names no `model` is a 400 naming the field, never a fallback to this node's own model, which would send someone's audio to a chat engine. They are also the first pair whose existence is MODEL-CONDITIONAL: vLLM attaches its speech-to-text router only when the served model reports the `transcription` task, so no chat or pooling engine's `/openapi.json` lists them and the check below cannot be run on one. For a path like that, the evidence is the router's own declaration in the engine image the recipe pins (`entrypoints/openai/speech_to_text/api_router.py`), and the `openapi.json` check still applies the first time such a model actually serves. Forward `request.path_qs`, not `request.path`: Claude Code posts to `/v1/messages?beta=true`. This is a route table and not a catch-all: an unregistered path stays a 404, which is why a new path is added only after `curl http://:/openapi.json` on a real engine says the engine answers it. - **`/v1/decide`'s response shape is a contract, not an implementation detail** (`api/decide.py`). Its bench is written against the exact shape (`model`, `node`, `latency_ms`, `decisions[key] = {answer, confidence, distribution, latency_ms}`, `usage = {prompt_tokens, completion_tokens, calls}`), so a `200` always carries every question asked: a bad request is a `400` and an engine that cannot answer is a `503`, never a partial `decisions` block. Probabilities are keyed by the caller's OPTION strings, never by the letters used to constrain the engine. It is the one `/v1` path deliberately NOT on `proxy_to_vllm`, because it composes N grammar-constrained chat completions of its own from one request and has no caller body to forward; it still routes through the proxy's own `_routing_candidates` and the shared `app["client_session"]`, so never give it its own routing rule or HTTP stack. The constraint field is vLLM 0.27.1's `structured_outputs: {"choice": [...]}`: the legacy `guided_choice` is accepted by that image and then silently ignored, so sending it instead would produce free prose with no error. The engine-facing half of a decision request is `decide.py::run_questions`, shared with `/v1/systemone`: one path to the engines and one way the probabilities are read, so a route decides the status and the shape and nothing else. diff --git a/README.md b/README.md index 76f9018f..6b2eaaf3 100644 --- a/README.md +++ b/README.md @@ -637,10 +637,10 @@ sets and the protocol are in [`bench/decide/README.md`](bench/decide/README.md). -| Backend/Model | Placement | Items | Accuracy | Brier | ECE | Wrong at 0.9 | p50 ms | Cost | Date | Run | -|---|---|---|---|---|---|---|---|---|---|---| -| chat / Ornith 1.5 35B-A3B | Spark-1-DGX, TP=1 | 110 | 0.964 | 0.027 | 0.023 | 2 of 106 | 659 | $0 | 2026-09-18 | [Ornith stacked Spark-1, chat fallback](https://github.com/getainode/ainode/blob/main/bench/results/20260918-030249-ornith-1_5-35b-a3b-nvfp4-ornith-stacked-spark-1-chat-fallback-decide.json) | -| jev / jev-1.13.0 | typesafe.ai hosted | 110 | 0.964 | 0.024 | 0.049 | 0 of 80 | 306 | $0.0016 | 2026-09-18 | [jev-latest, 110 items](https://github.com/getainode/ainode/blob/main/bench/results/20260918-030240-jev-1_13_0-jev-latest-110-items-decide.json) | +| Backend/Model | Placement | Items | Accuracy | Decision Score | Brier | ECE | Hand-off at 95% | Wrong at 0.9 | p50 ms | Cost | Date | Run | +|---|---|---|---|---|---|---|---|---|---|---|---|---| +| chat / Ornith 1.5 35B-A3B | Spark-1-DGX, TP=1 | 110 | 0.964 | not measured | 0.027 | 0.023 | not measured | 2 of 106 | 659 | $0 | 2026-09-18 | [Ornith stacked Spark-1, chat fallback](https://github.com/getainode/ainode/blob/main/bench/results/20260918-030249-ornith-1_5-35b-a3b-nvfp4-ornith-stacked-spark-1-chat-fallback-decide.json) | +| jev / jev-1.13.0 | typesafe.ai hosted | 110 | 0.964 | not measured | 0.024 | 0.049 | not measured | 0 of 80 | 306 | $0.0016 | 2026-09-18 | [jev-latest, 110 items](https://github.com/getainode/ainode/blob/main/bench/results/20260918-030240-jev-1_13_0-jev-latest-110-items-decide.json) | diff --git a/ainode/bench/decide/__init__.py b/ainode/bench/decide/__init__.py index c25e7fdb..77673506 100644 --- a/ainode/bench/decide/__init__.py +++ b/ainode/bench/decide/__init__.py @@ -14,13 +14,27 @@ calibration error with the reliability table behind it, and a count of the wrong answers that survive a 0.8 and a 0.9 gate. - ``items.py`` the labeled set, loaded strictly from bench/decide/items.json - ``metrics.py`` accuracy, Brier, calibration, thresholds, latency, cost +Two measurements live here, because they ask that of the same endpoints and write the +same record. The **Jevals recipe** scores the three public question sets the independent +Jevals boards use, with their formulas, so an AINode-served model can be read next to Jev +and its clones; the **legacy 110-item path** scores AINode's own hand-built set, five +shapes of the job a router or a triage step actually does. A number from one is not a +number from the other, and a record says which produced it (``decide.mode``). + + ``jevals.py`` the Jevals recipe as pure functions: Decision Score, ECE, + hand-off at 95 percent, the gate, flips, the losses + ``sets.py`` the three public sets: committed manifests, the download, the + question files, the state-hash check + ``suite.py`` the Jevals run: the two transports, the repeats, the record + ``items.py`` the legacy labeled set, loaded strictly from bench/decide/items.json + ``metrics.py`` the legacy metrics: accuracy, Brier, calibration, thresholds ``backends.py`` ainode (POST /v1/decide), chat (lettered options), jev (hosted) - ``runner.py`` the loop, the tables, the record + ``runner.py`` the legacy loop, the tables, the record ``cli.py`` ``scripts/ainode-bench.py decide ...`` -Stdlib only, like the rest of ``ainode/bench``. See ``bench/decide/README.md``. +Stdlib only, like the rest of ``ainode/bench``. See ``bench/decide/README.md`` for the +flags and ``bench/decide/JEVALS.md`` for the recipe, the date it was read and every +deviation from it. """ from ainode.bench.decide.backends import ( @@ -59,6 +73,14 @@ summarize_sets, threshold_counts, ) +from ainode.bench.decide.jevals import ( + PUBLISHED_GATES, + REPEATS, + confidence_swing, + decision_score, + handoff, + pick_flips, +) from ainode.bench.decide.runner import ( DEFAULT_CONCURRENCY, SOURCE, @@ -68,8 +90,30 @@ row_for, run_items, ) +from ainode.bench.decide.sets import ( + SUITES, + SetError, + answer_key_leaks, + load_questions, + load_suite_questions, + state_sha256, +) +from ainode.bench.decide.suite import ( + MODE, + TRANSPORTS, + DecideTransport, + SystemOneTransport, + Transport, + build_transport, + wire_leaks, +) -__all__ = ["BACKENDS", "BINS", "CHOICE", "DEFAULT_CONCURRENCY", +__all__ = ["BACKENDS", "BINS", "CHOICE", "DEFAULT_CONCURRENCY", "MODE", + "PUBLISHED_GATES", "REPEATS", "SUITES", "TRANSPORTS", "DecideTransport", + "SetError", "SystemOneTransport", "Transport", "answer_key_leaks", + "build_transport", "confidence_swing", "decision_score", "handoff", + "load_questions", "load_suite_questions", "pick_flips", "state_sha256", + "wire_leaks", "JEV_INPUT_USD_PER_MTOK", "JEV_MODEL", "JEV_URL", "KINDS", "NOUL", "SOURCE", "THRESHOLDS", "Backend", "BackendError", "ChatBackend", "DecideBackend", "Decision", "Item", "ItemError", "ItemSet", diff --git a/ainode/bench/decide/cli.py b/ainode/bench/decide/cli.py index 606428e8..4edee497 100644 --- a/ainode/bench/decide/cli.py +++ b/ainode/bench/decide/cli.py @@ -5,28 +5,49 @@ whether it can hold an agent loop together, but whether its typed decisions can be trusted by code that acts on them. Accuracy, calibration, latency, cost. - scripts/ainode-bench.py decide --backend jev --label "jev-latest, 110 items" - - scripts/ainode-bench.py decide --backend chat \\ +Two measurements live under this one word, because they ask that of the same endpoints +and write the same record. **The Jevals recipe** scores the three public question sets +the independent Jevals boards use, with their formulas, so an AINode-served model can be +read next to Jev and its clones; **the legacy 110-item path** scores AINode's own hand +built set, which is five shapes of the job a router or a triage step actually does. +``--suite``/``--questions`` picks the first and ``--backend`` the second, and mixing them +is an error rather than a guess. + + # the Jevals recipe. Fetch the question sets once; the item text is not committed + scripts/ainode-bench.py decide download + + scripts/ainode-bench.py decide --suite all --transport decide \\ --endpoint http://100.122.26.9:3000/v1 \\ --ainode http://100.122.26.9:3000 \\ --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 \\ - --label "Ornith stacked Spark-1, chat fallback" + --label "Ornith on Spark-1, Jevals 0.1.0" + + # any server that speaks the Jev wire format, TypeSafe's hosted Jev included + scripts/ainode-bench.py decide --suite all --transport systemone \\ + --endpoint https://api.typesafe.ai/v1 --label "jev-latest, Jevals 0.1.0" + # a private blind set, in the same shape, never committed + scripts/ainode-bench.py decide --questions /path/to/blind.json \\ + --transport systemone --endpoint http://kev-host:8080/v1 --label blind-1 + + # the legacy 110-item path + scripts/ainode-bench.py decide --backend jev --label "jev-latest, 110 items" scripts/ainode-bench.py decide --backend chat --compare jev ... # side by side -``--dry-run`` prints the item counts, the backend and one example request and -touches nothing: no request, no file. The API key is never printed, only where it -came from. +``--dry-run`` prints the plan, the sets and one example request per question shape and +touches nothing: no request, no file. The API key is never printed, only where it came +from. """ from __future__ import annotations import argparse import json import pathlib +import sys import time from ainode.bench import auth +from ainode.bench.decide import jevals, sets, suite from ainode.bench.decide.backends import ( BACKENDS, DEFAULT_API_KEY, @@ -104,6 +125,40 @@ def build_parser() -> argparse.ArgumentParser: "Never printed and never written into a record") p.add_argument("--dry-run", action="store_true", help="print the plan and one example request, write nothing") + + # The Jevals-recipe mode. A second measurement in the same subcommand, because it + # asks the same question of the same endpoints and writes the same record; what it + # changes is the question sets (the three public ones the independent boards use), + # the repeats and the formulas. See bench/decide/JEVALS.md. + jev = p.add_argument_group( + "the Jevals recipe (suite 0.1.0)", + "score the same public question sets the independent Jevals boards use, with " + "their formulas, so an AINode-served model can be read next to Jev and its " + "clones. `decide download` fetches the item text first; it is not committed") + jev.add_argument("--suite", default="", + help="comma list of " + ", ".join(sets.SUITES) + ", or `all`. " + "Turns on the Jevals recipe and needs --transport") + jev.add_argument("--questions", default="", + help="run a question file in the same shape instead of a suite, so " + "a private blind set is scored by the same code without being " + "committed. Repeatable as a comma list") + jev.add_argument("--transport", default="", choices=list(suite.TRANSPORTS), + help="decide (AINode's POST /v1/decide) or systemone (POST " + "/v1/systemone in the Jev wire format: TypeSafe's hosted Jev, " + "or any server that speaks it). The key is resolved from the " + "endpoint's HOST, so a fleet key can never reach a vendor") + jev.add_argument("--repeats", type=int, default=jevals.REPEATS, + help=f"answers per question (default {jevals.REPEATS}, which is the " + "suite's figure; a board listing needs a complete run at 5)") + jev.add_argument("--limit", type=int, default=0, + help="take only the first N questions of each set. A transport " + "proof, not a suite result, and the record says so") + jev.add_argument("--price-in", type=float, default=0.0, + help="USD per million input tokens for this endpoint, from its " + "posted rate. Without it the cost column reads $0 rather than " + "an estimate") + jev.add_argument("--price-out", type=float, default=0.0, + help="USD per million output tokens for this endpoint") return p @@ -273,10 +328,260 @@ def progress(done, count, row): return block, title, path -def main(argv=None, out_dir=None) -> int: +# --------------------------------------------------------------- the Jevals recipe + +def download_main(argv, out=say) -> int: + """``decide download [suite,...]``: fetch the item text, verify it, write the cache. + + The only networked call in :mod:`ainode.bench.decide.sets`, and the only thing that + writes under ``bench/decide/cache/``, which is gitignored. Item text is not + committed: Jevals does not republish it either, the three upstream licences are the + item text's and not ours to relicense, and every state is checked against its + published ``state_sha256`` on the way in, so a download proves itself rather than + being trusted. + """ + wanted = name_list(argv[0]) if argv and not argv[0].startswith("-") else None + if wanted == ["all"]: + wanted = None + try: + out("\n ainode-bench decide download -> " + str(sets.cache_dir())) + for suite_id in (wanted or sets.SUITES): + summary = sets.manifest_summary(suite_id) + out(f" {summary['id']:<12} {summary['type']:<7} " + f"{summary['items']:3d} items {summary['options']:3d} options " + f"{summary['dataset']} {summary['split']} @ " + f"{summary['hf_revision'][:8]} {summary['license']}") + paths = sets.download(wanted, progress=lambda line: out(f" {line}")) + except sets.SetError as exc: + out(f"\n {exc}") + return 1 + out(f"\n {len(paths)} question file(s) written and hash-verified. They are " + "gitignored on purpose (bench/decide/JEVALS.md).") + return 0 + + +def suite_docs(args, p): + """The question files this run scores: named suites, named files, or an error.""" + docs = [] + if args.suite: + wanted = name_list(args.suite) + if wanted == ["all"]: + wanted = list(sets.SUITES) + docs += sets.load_suite_questions(wanted) + for path in name_list(args.questions): + docs.append(sets.load_questions(path)) + if not docs: + p.error("--suite or --questions names nothing to run") + seen = set() + for doc in docs: + name = doc.get("set") or doc["id"] + if name in seen: + p.error(f"two question files both call themselves {name!r}; a set is one " + "measurement and two of them cannot share a name") + seen.add(name) + return docs + + +def suite_settings(args, transport, docs) -> dict: + return {"mode": suite.MODE, + "transport": transport.name, + "endpoint": transport.endpoint, + "model_requested": args.model or "", + "suite": name_list(args.suite), + "questions_files": name_list(args.questions), + "sets": [(doc.get("set") or doc["id"]) for doc in docs], + "repeats": args.repeats, + "limit": args.limit or None, + "concurrency": args.concurrency, + "timeout_s": args.timeout} + + +def suite_dry_run(args, transport, docs, out=say) -> int: + """The plan, one example request per primitive, and what the key source was.""" + work = suite.plan(docs, args.repeats, args.limit) + out(f"\n ainode-bench decide the Jevals recipe, suite {suite.RECIPE['suite']}") + out(f" recipe : {suite.RECIPE['source']} read {suite.RECIPE['read']}, " + f"recorded in {suite.RECIPE['doc']}") + out(f" transport: {transport.name} {transport.endpoint}") + out(f" model : {transport.model or 'server default'}") + out(f" key : from {transport.key_source or 'the default'} (never printed)") + out(f" cost : ${transport.input_usd_per_mtok:g}/M input, " + f"${transport.output_usd_per_mtok:g}/M output") + out(f" plan : {len(work)} decisions, {args.repeats} repeats, concurrency " + f"{args.concurrency}") + for doc in docs: + summary = suite.set_summary(doc, args.limit) + source = summary["source"] + out(f" {summary['id']:<12} {summary['type']:<7} " + f"{summary['questions']:3d} questions {summary['options']:3d} options " + f"seed {summary['seed']}") + if source: + out(f" {source.get('dataset', '')} " + f"{source.get('split', '')} @ {str(source.get('hf_revision', ''))[:8]}" + f" {source.get('license', '')}") + # One example per primitive the set holds, so a mixed set shows every shape it + # will really send rather than whichever question happens to be first. + shown = set() + for question in doc["questions"]: + spec = suite.spec_for(doc, question) + if spec["type"] in shown: + continue + shown.add(spec["type"]) + presented = suite.presented_options(spec["options"], question["id"], + doc.get("seed"), 0, spec["type"]) + request = transport.request(doc, question, presented) + line = request.curl_safe() + out(f" example ({spec['type']}): {line[:400]}" + f"{' ...' if len(line) > 400 else ''}") + leaks = suite.wire_leaks(request.payload) + out(f" wire : " + f"{'CARRIES ' + ', '.join(leaks) if leaks else 'no answer key'}") + out("\n dry run: nothing was requested and no file was written") + return 0 + + +def run_suite(args, out_dir, stamp, log=say): + """One transport over the question sets, five repeats each. Writes one record.""" p = build_parser() - args = p.parse_args(argv) + docs = suite_docs(args, p) + transport = suite.build_transport( + args.transport, endpoint=args.endpoint, model=args.model, + api_key=args.api_key, timeout=args.timeout, + input_usd_per_mtok=args.price_in, output_usd_per_mtok=args.price_out) + names = [(doc.get("set") or doc["id"]) for doc in docs] + + log(f"\n ainode-bench decide the Jevals recipe, suite {suite.RECIPE['suite']}: " + f"{', '.join(names)}") + log(f" recipe : {suite.RECIPE['source']} read {suite.RECIPE['read']} " + f"({suite.RECIPE['doc']} names every deviation)") + log(" metrics : accuracy with its guessing floor, Decision Score against the " + "label prior, ECE over 10 bins with the reliability table, hand-off share at " + "95%, the published gate, pick flips and confidence swing, p50/p95 latency, " + "questions per second, malformed answers, cost") + if args.dry_run: + return suite_dry_run(args, transport, docs), None + log(f" {transport.name} {transport.model or 'server default'} " + f"{transport.endpoint}") + log(f" key from {transport.key_source or 'the default'} (never printed)") + if transport.local: + refused = auth.preflight(args.endpoint, transport.api_key) + if refused: + raise auth.EndpointRefused(refused) + + def progress(done, count, decision): + if done == count or done % 50 == 0: + mark = "err " if decision.get("error") else ( + "bad " if decision.get("malformed") else + ("ok " if decision.get("pick") == decision.get("label") else "MISS")) + log(f" {done:5d}/{count} last {decision['id']:<16} " + f"r{decision['repeat']} {mark}") + + decisions, seconds = suite.run(transport, docs, repeats=args.repeats, + concurrency=args.concurrency, limit=args.limit, + progress=progress) + block = suite.build_decide_block(transport, docs, decisions, seconds, + args.repeats, args.concurrency, args.limit, + model_reported=suite.reported_model(transport)) + model_block, placement, warnings = describe_suite(args, transport) + for warning in warnings: + log(f" warn : {warning}") + notes = suite.build_notes(transport, docs, decisions, seconds, args.repeats, + args.limit) + list(warnings) + record = build_record(args.label, model_block, placement, block, + suite_settings(args, transport, docs), notes, stamp) + + out_dir.mkdir(parents=True, exist_ok=True) + path = record_path(out_dir, stamp, model_block.get("id") or transport.name, + args.label, transport.name) + path.write_text(json.dumps(record, indent=1) + "\n") + title = (f"{transport.name} " + f"{model_block.get('name') or model_block.get('id')} " + f"({len(decisions)} decisions in {round(seconds)}s)") + suite.print_table(block["jevals"], title, out=log) + suite.print_wrong(decisions, out=log) + log(f"\n saved {path}") + return 0, path + + +def describe_suite(args, transport): + """``(model_block, placement, warnings)`` for a Jevals-recipe record. + + The same rule the other sections follow: a hosted endpoint gets the one honest + placement string there is, and a local run that was not told where the control plane + is records NO placement rather than stamping the endpoint's host as the node. + """ + model_id = suite.reported_model(transport) + if not transport.local: + return ({"id": model_id, "name": model_id, "vendor": "typesafe.ai"}, + dict(HOSTED_PLACEMENT), []) + if not args.ainode: + return {"id": model_id}, {}, [] + from ainode.bench.fleet import describe_via_http, resolve_serving_node + + base = args.ainode.rstrip("/") + key = transport.api_key + node_name, engine_port, gpu_name, resolve_warn = resolve_serving_node( + base, args.model, api_key=key) + model_block, placement, _node_id, warnings = describe_via_http( + base, base, args.model, api_key=key) + if node_name: + placement["node"] = node_name + placement["port"] = engine_port + if gpu_name: + placement["gpu"] = gpu_name + warnings = list(warnings) + if resolve_warn: + warnings.insert(0, resolve_warn) + return model_block, placement, warnings + + +def main(argv=None, out_dir=None) -> int: + words = list(argv) if argv is not None else sys.argv[1:] out_dir = pathlib.Path(out_dir) if out_dir else pathlib.Path.cwd() / "bench" / "results" + # One positional, dispatched before argparse sees it, the way `ainode-bench` + # dispatches its own subcommands: `decide download` fetches the question sets and + # asks a model nothing at all, so it shares none of the run's flags. + if words and words[0] == "download": + return download_main(words[1:]) + + p = build_parser() + args = p.parse_args(words) + + if args.concurrency < 1: + p.error("--concurrency must be at least 1") + if args.timeout <= 0: + p.error("--timeout must be positive") + if not args.dry_run and not args.label: + p.error("--label is required") + + if args.suite or args.questions: + if args.backend: + p.error("--suite/--questions run the Jevals recipe and pick their wire with " + "--transport; --backend is the legacy 110-item path") + if not args.transport: + p.error("--transport is required with --suite/--questions; pick from " + + ", ".join(suite.TRANSPORTS)) + if args.repeats < 1: + p.error("--repeats must be at least 1") + if args.limit < 0: + p.error("--limit cannot be negative") + stamp = time.strftime("%Y%m%d-%H%M%S", time.gmtime()) + try: + code, path = run_suite(args, out_dir, stamp) + except (sets.SetError, BackendError) as exc: + return p.error(str(exc)) + except auth.EndpointRefused as exc: + # The node refused, before or during the run. Nothing is scored and no + # record is written: a Decision Score computed over answers nobody gave is + # worse than no record. + return auth.stop(str(exc), out=say) + if path: + say("\n render the README table with: python3 " + "scripts/render-bench-table.py") + return code + if args.transport: + p.error("--transport belongs to --suite/--questions; the legacy 110-item path " + "picks its wire with --backend") if not args.backend: p.error("--backend is required; pick from " + ", ".join(BACKENDS)) @@ -331,5 +636,7 @@ def run() -> int: return 130 -__all__ = ["HOSTED_PLACEMENT", "SOURCE", "build_parser", "describe", "dry_run", - "main", "name_list", "run", "run_backend", "settings_for"] +__all__ = ["HOSTED_PLACEMENT", "SOURCE", "build_parser", "describe", + "describe_suite", "download_main", "dry_run", "main", "name_list", "run", + "run_backend", "run_suite", "settings_for", "suite_docs", "suite_dry_run", + "suite_settings"] diff --git a/ainode/bench/decide/jevals.py b/ainode/bench/decide/jevals.py new file mode 100644 index 00000000..406c3b04 --- /dev/null +++ b/ainode/bench/decide/jevals.py @@ -0,0 +1,1193 @@ +"""The Jevals recipe, suite 0.1.0, as functions over plain decision dicts. + +The recipe is recorded in ``bench/decide/JEVALS.md`` with the URL and the date it was +read, and every formula here cites the section it came from. Nothing in this module +talks to a model, a node or a file: a metric is a pure function of the decisions, which +is what lets ``tests/test_bench_decide_jevals.py`` score hand-computed examples with no +network. + +Why a second metrics module beside :mod:`ainode.bench.decide.metrics` rather than a +rewrite of it: the two measure different things and older records carry the first one. +``metrics.py`` scores the 110-item AINode set one pass per item, with a one-term Brier +on the labeled option and five bins. This module scores the three public Jevals sets +five passes per item, with the recipe's own multiclass Brier, its ranked probability +score, its ten bins, its Decision Score against the label prior and its hand-off share. +A record can carry both; a number from one is not a number from the other. + +The unit of measurement here is a **decision**: one (item, repeat) pair. A decision dict +is what :mod:`ainode.bench.decide.suite` builds: + + ``id`` the item it belongs to + ``set`` which suite set (``pubmedqa``, ``banking77``, ``helpsteer2``) + ``repeat`` 0-based repeat index + ``order`` which presented option order this repeat used + ``type`` ``noul``, ``choice`` or ``score`` + ``options`` the task's options in CANONICAL order, never the presented one + ``label`` the labeled option, as an option string + ``vector`` probability per option, already normalized + ``pick`` the option the system picked, or None (which counts as wrong) + ``malformed`` the answer could not be read as a vector over these options + ``one_hot`` the system reported no probabilities, so the vector is its pick + ``confidence`` the probability of the pick, or None + ``wall_ms`` measured round trip + ``tokens_in`` prompt tokens the endpoint reported, or None + ``tokens_out`` completion tokens the endpoint reported, or None + ``error`` a transport or protocol failure, or None + +Two exclusion rules run through everything below, both the recipe's: + + * **A transport failure is never scored.** A decision with an ``error`` is out of the + accuracy, out of the losses, out of every rate, and counted on its own as ``failed``. + A board cannot be built while any (item, repeat) is missing, so a run with failures + reports them rather than quietly scoring 299 items as 300. + * **A malformed or refused answer IS scored**, as the uniform distribution and a wrong + pick. It counts in the Decision Score and the accuracy and is excluded from the ECE, + the flip rates and the gate. +""" +from __future__ import annotations + +import math + +#: Question types, Jev's three primitives. +NOUL = "noul" +CHOICE = "choice" +SCORE = "score" +TYPES = (NOUL, CHOICE, SCORE) + +#: Repeats per item. Five is the suite's figure, and a listing on a board needs a +#: complete run of every task at five. +REPEATS = 5 + +#: Bins for the calibration gap. Ten equal-width bins, which is the recipe's number and +#: not ``metrics.BINS``' five. +BINS = 10 + +#: The grid the gate and the hand-off threshold are searched on. +GRID = 0.01 + +#: Hand-off at 95 percent: the accuracy a threshold has to reach, and the smallest +#: number of decisions that may stand behind it. +HANDOFF_ACCURACY = 0.95 +HANDOFF_MIN_DECISIONS = 100 + +#: The shared gate's rule: pooled error at most this, over at least this many decisions. +GATE_MAX_ERROR = 0.05 +GATE_MIN_DECISIONS = 100 + +#: The frozen gates jevals.com publishes for suite 0.1.0. They are pooled across every +#: listed system and frozen from the first release, so one run cannot recompute them: +#: a run reports its coverage AT these, and its own one-system gate separately. +PUBLISHED_GATES = {CHOICE: 0.96, SCORE: None, NOUL: 0.91} +PUBLISHED_GATE_SOURCE = "jevals.com/methodology, suite 0.1.0, frozen" + +#: Comparisons on the 0.01 grid absorb float representation error rather than dropping +#: a stated 0.96 that does not survive being written down as a double. +EPSILON = 1e-9 + + +# ------------------------------------------------------------------ small helpers + +def round_half_up(value: float) -> int: + """``Math.round`` semantics, which is what the published bin formula assumes. + + Python's ``round`` breaks a tie to even (``round(0.5) == 0``), so a confidence of + exactly 0.105 would bin differently here than on the board. The ECE formula is + published as ``min(9, floor(round(100*c) / 10))`` over a JavaScript harness, so the + rounding has to be the JavaScript one. + """ + return int(math.floor(float(value) + 0.5)) + + +def is_probability(value) -> bool: + """A finite number in [0, 1]. Booleans are not numbers here.""" + if isinstance(value, bool) or not isinstance(value, (int, float)): + return False + number = float(value) + return math.isfinite(number) and -EPSILON <= number <= 1.0 + EPSILON + + +def uniform(options) -> dict: + """The vector a refused or malformed answer is scored as.""" + if not options: + return {} + share = 1.0 / len(options) + return {option: share for option in options} + + +#: JevBench's two sum tolerances. Its headline renormalizes anything inside a 2 percent +#: band and publishes a strict column under the 0.001 tolerance its v1 froze, because +#: the loose band mostly catches three-decimal rounding on a nine-option question. Both +#: rates are reported here under its own names. +SUM_TOLERANCE = 0.02 +SUM_TOLERANCE_STRICT = 0.001 + + +def normalize_vector(raw, options): + """``(vector, None)`` or ``(None, why)``: the recipe's malformed rules, in order. + + Malformed when the map is missing, names an unknown option, repeats one, carries a + value that is not a finite number in [0, 1], or sums to 0. Otherwise the listed + values are divided by their sum, which covers both published normalizations for a + full vector: a sum above 1 is scaled down, and a sum below 1 (Jev rounds to two + decimals, so 0.99 happens) is scaled up. + + An option the answer did not name is 0 rather than absent, because every metric + below reads the vector as a distribution over the task's whole option set. + """ + if not isinstance(raw, dict) or not raw: + return None, "no probability map" + allowed = set(options) + seen = set() + values = {} + for key, value in raw.items(): + name = key if isinstance(key, str) else str(key) + if name not in allowed: + return None, f"unknown option {name!r}" + if name in seen: + return None, f"duplicate option {name!r}" + seen.add(name) + if not is_probability(value): + return None, f"option {name!r} has probability {value!r}" + values[name] = min(1.0, max(0.0, float(value))) + total = sum(values.values()) + if total <= 0: + return None, "probabilities sum to 0" + return {option: values.get(option, 0.0) / total for option in options}, None + + +def pick_from_vector(vector, options, presented=None, qtype: str = CHOICE, + stated=None): + """The most probable option, with the recipe's tie rules. + + ``choice``: ties go to the option the system listed first, which is its own stated + answer when that is one of the winners and otherwise the first winner in the order + the options were PRESENTED in. ``score``: ties go to the lower level, which is the + first winner in canonical (level) order. ``noul``: a yes/no answer of exactly 0.5 + has no pick and counts as wrong, so this returns None. + """ + if not vector: + return None + top = max(vector.get(option, 0.0) for option in options) + winners = [option for option in options if vector.get(option, 0.0) >= top - EPSILON] + if not winners: + return None + if qtype == NOUL: + # Exactly 0.5 either way is the published no-pick case. Two options, equal mass. + if len(winners) > 1: + return None + return winners[0] + if qtype == SCORE: + return winners[0] + if stated in winners: + return stated + order = list(presented or options) + for option in order: + if option in winners: + return option + return winners[0] + + +# ------------------------------------------------------------------ partitions + +def scored(decisions) -> list: + """Decisions the recipe scores: everything that came back at all. + + A transport failure is not one of them. That is the line between "the model was + wrong" and "the request never happened", and collapsing it is how a refused run + turns into a set of confident misses. + """ + return [d for d in decisions if not d.get("error")] + + +def failed(decisions) -> list: + return [d for d in decisions if d.get("error")] + + +def calibratable(decisions) -> list: + """Decisions the ECE, the gate, the hand-off and the flip rates run over. + + Malformed answers are out (the recipe says so) and so are one-hot answers, because + a system that reports no probabilities has a confidence of 1.00 by construction and + a board shows it a dash rather than a perfect gate. + """ + return [d for d in scored(decisions) + if not d.get("malformed") and not d.get("one_hot") + and d.get("confidence") is not None] + + +def by_item(decisions) -> dict: + """``{item id: [decisions in repeat order]}``, which is the loss's unit.""" + out = {} + for decision in decisions: + out.setdefault(decision["id"], []).append(decision) + for rows in out.values(): + rows.sort(key=lambda d: d.get("repeat", 0)) + return out + + +def is_correct(decision) -> bool: + """A pick equal to the label. No pick is wrong, which malformed answers rely on.""" + pick = decision.get("pick") + return pick is not None and pick == decision.get("label") + + +# ------------------------------------------------------------------ the two losses + +def brier_loss(vector, options, label) -> float: + """``sum_k (p_k - y_k)^2``: the multiclass Brier score, for choice and noul. + + Not the one-term version :mod:`ainode.bench.decide.metrics` uses. Two options with + all the mass on the wrong one scores 2.0, and a uniform answer over K options + scores ``1 - 1/K``. + """ + total = 0.0 + for option in options: + target = 1.0 if option == label else 0.0 + total += (float(vector.get(option, 0.0)) - target) ** 2 + return total + + +def rps_loss(vector, options, label) -> float: + """``sum_{k float: + """The loss the recipe uses for that primitive.""" + if qtype == SCORE: + return rps_loss(vector, options, label) + return brier_loss(vector, options, label) + + +def decision_loss(decision) -> float: + """One decision's loss. A malformed answer's vector is already the uniform one.""" + vector = decision.get("vector") or uniform(decision["options"]) + return loss_for(decision.get("type", CHOICE), vector, decision["options"], + decision["label"]) + + +def mean_item_loss(decisions): + """``L``: the mean over items of the mean over that item's repeats. + + Per item first, so an item answered five times weighs the same as one answered + once, which is what makes the item-cluster bootstrap on the board meaningful and + what keeps a partly failed item from pulling the mean. + """ + groups = by_item(scored(decisions)) + if not groups: + return None + per_item = [sum(decision_loss(d) for d in rows) / len(rows) + for rows in groups.values() if rows] + if not per_item: + return None + return sum(per_item) / len(per_item) + + +# ------------------------------------------------------------------ the label prior + +def label_prior(decisions): + """The baseline that defines 0: the base rates of the EVALUATED items. + + One vector, over the options, from how often each label occurs among the items this + run scored. Counted per item and not per decision, so five repeats of one item do + not make its label five times as common. + """ + groups = by_item(scored(decisions)) + if not groups: + return None, None + options = None + counts = {} + for rows in groups.values(): + first = rows[0] + options = options or list(first["options"]) + counts[first["label"]] = counts.get(first["label"], 0) + 1 + total = sum(counts.values()) + if not options or not total: + return None, None + return {option: counts.get(option, 0) / total for option in options}, options + + +def prior_loss(decisions): + """``L_prior``: the same loss, for the label prior, on the same items.""" + prior, options = label_prior(decisions) + if prior is None: + return None + groups = by_item(scored(decisions)) + qtype = next(iter(groups.values()))[0].get("type", CHOICE) + losses = [loss_for(qtype, prior, options, rows[0]["label"]) + for rows in groups.values()] + if not losses: + return None + return sum(losses) / len(losses) + + +def prior_accuracy(decisions): + """The guessing floor: how often the label prior's own pick is right. + + Its pick is the most common label, so this is that label's share of the evaluated + items. Every metrics block here carries it, because an accuracy of 0.62 on a set + whose majority class is 0.62 is a model that has learned nothing, and the reporting + rule this bench follows is that the floor travels with the figure. + """ + prior, options = label_prior(decisions) + if prior is None: + return None + groups = by_item(scored(decisions)) + qtype = next(iter(groups.values()))[0].get("type", CHOICE) + if qtype == NOUL and len(set(prior.values())) == 1: + # A 50/50 prior has no pick under the yes/no rule, so it is right never. + return 0.0 + guess = pick_from_vector(prior, options, presented=options, + qtype=CHOICE if qtype == NOUL else qtype) + if guess is None: + return 0.0 + hits = sum(1 for rows in groups.values() if rows[0]["label"] == guess) + return hits / len(groups) + + +def decision_score(system_loss, baseline_loss): + """``100 * (1 - L_system / L_prior)``. 100 is perfect, 0 is the base rates. + + Negative is worse than the base rates and is returned as it is, never clamped. None + when either loss is missing, or when the prior's loss is 0, which happens only on a + set where every item carries the same label and no system can do better than it. + """ + if system_loss is None or baseline_loss is None or baseline_loss <= 0: + return None + return 100.0 * (1.0 - (system_loss / baseline_loss)) + + +# ------------------------------------------------------------------ accuracy, ECE + +def accuracy(decisions): + """Correct picks over ALL scored decisions, items times repeats. + + Malformed and refused answers are in the denominator and count as wrong, which is + the recipe's rule and the reason a system cannot buy accuracy by refusing. + """ + rows = scored(decisions) + if not rows: + return None + return sum(1 for d in rows if is_correct(d)) / len(rows) + + +def bin_index(confidence: float, bins: int = BINS) -> int: + """``min(9, floor(round(100*c) / 10))``, so a confidence of 1.00 lands in the last.""" + return min(bins - 1, max(0, round_half_up(100.0 * float(confidence)) // (100 // bins))) + + +def reliability(decisions, bins: int = BINS) -> list: + """The table behind the ECE: per bin, how many decisions, how often right, how sure. + + An empty bin keeps its row with ``count: 0`` and nulls, so the shape of the table + does not change between runs and a reader can see which part of the range a system + never used. + """ + buckets = [[] for _ in range(bins)] + for decision in calibratable(decisions): + buckets[bin_index(decision["confidence"], bins)].append(decision) + table = [] + for index, bucket in enumerate(buckets): + block = {"lo": round(index / bins, 2), "hi": round((index + 1) / bins, 2), + "count": len(bucket), "accuracy": None, "confidence": None} + if bucket: + block["accuracy"] = sum(1 for d in bucket if is_correct(d)) / len(bucket) + block["confidence"] = sum(float(d["confidence"]) + for d in bucket) / len(bucket) + table.append(block) + return table + + +def ece(decisions, bins: int = BINS): + """``sum_b (n_b/N) * |accuracy_b - mean confidence_b|`` as a ratio, or None. + + None when nothing could be calibrated, which is the board's dash: a system with no + probabilities has no calibration gap, and reporting 0 for it would make the most + opaque row look like the most honest one. + """ + rows = calibratable(decisions) + if not rows: + return None + total = 0.0 + for block in reliability(rows, bins): + if block["count"]: + total += abs(block["accuracy"] - block["confidence"]) * block["count"] + return total / len(rows) + + +def ece_points(decisions, bins: int = BINS): + """The ECE the way a board prints it: in points, so 0.058 reads as 5.8.""" + value = ece(decisions, bins) + return None if value is None else value * 100.0 + + +# ------------------------------------------------------------------ gate, hand-off + +def grid_thresholds(step: float = GRID) -> list: + """``[0.00, 0.01, ..., 1.00]``, built off integers so the steps are exact.""" + count = int(round(1.0 / step)) + return [index / count for index in range(count + 1)] + + +def above(decisions, threshold: float) -> list: + """Calibratable decisions whose confidence clears the threshold.""" + return [d for d in calibratable(decisions) + if float(d["confidence"]) >= threshold - EPSILON] + + +def handoff(decisions, target: float = HANDOFF_ACCURACY, + min_decisions: int = HANDOFF_MIN_DECISIONS, step: float = GRID): + """Hand-off at 95 percent: the system's own threshold and the share it can take. + + The LOWEST confidence on the grid at which the decisions clearing it are at least + ``target`` correct, with at least ``min_decisions`` of them. ``share`` is those + decisions over ALL of this system's decisions, refused and malformed included, + which is the published denominator and the reason the share is not simply coverage + among the answers it was confident about. + + None when no threshold qualifies, which is the board's dash for a row whose accuracy + never reaches 95 percent. The threshold is chosen on the same decisions it is + measured on, so the share is optimistic; it is optimistic the same way for every + system, which is what makes it comparable. + """ + total = len(scored(decisions)) + if not total: + return None + for threshold in grid_thresholds(step): + kept = above(decisions, threshold) + if len(kept) < min_decisions: + continue + correct = sum(1 for d in kept if is_correct(d)) + if correct / len(kept) >= target - EPSILON: + return {"threshold": round(threshold, 2), "n": len(kept), + "accuracy": correct / len(kept), "share": len(kept) / total, + "decisions": total} + return None + + +def gate_local(decisions, max_error: float = GATE_MAX_ERROR, + min_decisions: int = GATE_MIN_DECISIONS, step: float = GRID): + """The published gate rule applied to THIS run's decisions alone. + + The board's gate is pooled over every listed system and frozen from the first + release, so it is not a thing one run can recompute; this is the same arithmetic + over one system, and the record labels it as such. None when no threshold on the + grid gets the pooled error to ``max_error`` over at least ``min_decisions``. + """ + for threshold in grid_thresholds(step): + kept = above(decisions, threshold) + if len(kept) < min_decisions: + continue + wrong = sum(1 for d in kept if not is_correct(d)) + if wrong / len(kept) <= max_error + EPSILON: + return round(threshold, 2) + return None + + +def gate_coverage(decisions, gate): + """Coverage and accuracy at a gate: the two numbers a board row shows beside it. + + ``coverage`` is over ALL scored decisions, the same denominator the hand-off share + uses, so a system that refuses half the set cannot read as covering everything it + answered. None when the primitive has no gate (``score`` has none in 0.1.0) or when + nothing here can be gated. + """ + if gate is None: + return None + total = len(scored(decisions)) + if not total or not calibratable(decisions): + return None + kept = above(decisions, float(gate)) + block = {"gate": float(gate), "n": len(kept), "coverage": len(kept) / total, + "accuracy": None} + if kept: + block["accuracy"] = sum(1 for d in kept if is_correct(d)) / len(kept) + return block + + +# ------------------------------------------------------------------ flips + +def _picks_at(rows, repeats): + """The picks at those repeat indices, or None when any of them cannot be compared. + + A malformed or failed answer in one of the compared repeats takes the whole item out + of the rate, numerator and denominator both, which is the recipe's exclusion. Half + an item's picks would otherwise read as a flip. + """ + wanted = [] + for index in repeats: + match = [d for d in rows if d.get("repeat") == index] + if not match: + return None + decision = match[0] + if decision.get("error") or decision.get("malformed"): + return None + if decision.get("pick") is None: + return None + wanted.append(decision["pick"]) + return wanted + + +def flip_rate(decisions, repeats): + """Share of items whose pick is not the same across those repeat indices. + + ``(rate, n)``, where ``n`` is how many items could be compared at all. ``(None, 0)`` + when none could, which is the honest answer for a run with fewer repeats than the + rate needs rather than a 0 that reads as perfect determinism. + """ + comparable = 0 + flipped = 0 + for rows in by_item(decisions).values(): + picks = _picks_at(rows, repeats) + if picks is None: + continue + comparable += 1 + if len(set(picks)) > 1: + flipped += 1 + if not comparable: + return None, 0 + return flipped / comparable, comparable + + +def repeat_flip_rate(decisions): + """Nondeterminism: repeats 0 and 1 are byte-identical requests.""" + return flip_rate(decisions, (0, 1)) + + +def pick_flips(decisions): + """``(rate, n)`` over ALL repeats: share of questions whose pick changed at least once. + + The board's repeat flip rate looks only at repeats 0 and 1 and its order flip rate + only at the four distinct orders, so neither answers "did this question ever get two + different answers out of five". This does, and it is the number a caller who has to + trust one answer wants: a question that flipped once in five is a question this model + does not actually have an opinion about. + + A question is comparable when at least two of its repeats came back with a readable + pick; a failed or malformed repeat drops out of that question's comparison, and a + question with fewer than two comparable repeats drops out of the rate entirely + rather than counting as stable. + """ + comparable = 0 + flipped = 0 + for rows in by_item(decisions).values(): + picks = [d["pick"] for d in rows + if not d.get("error") and not d.get("malformed") + and d.get("pick") is not None] + if len(picks) < 2: + continue + comparable += 1 + if len(set(picks)) > 1: + flipped += 1 + if not comparable: + return None, 0 + return flipped / comparable, comparable + + +def confidence_swing(decisions): + """How far one question's stated confidence moved across its repeats. + + ``{"max": .., "mean": .., "question": id, "over": n}``. The max is the largest + spread any single question showed, which is the honest headline for "how stable is + the number I would gate on": a model whose confidence on one question ran from 0.51 + to 0.99 across five identical-shaped requests has a gate that means something + different on every call. The mean is beside it because one pathological question + should not be read as the whole set. + + Over the calibratable decisions only, grouped per question, and a question with + fewer than two of them contributes nothing. ``None`` when no question had two. + """ + swings = [] + for item_id, rows in by_item(calibratable(decisions)).items(): + confidences = [float(d["confidence"]) for d in rows] + if len(confidences) < 2: + continue + swings.append((max(confidences) - min(confidences), item_id)) + if not swings: + return None + worst, worst_id = max(swings) + return {"max": worst, "mean": sum(s for s, _ in swings) / len(swings), + "question": worst_id, "over": len(swings)} + + +def order_flip_rate(decisions): + """Choice only: the four distinct option orders, repeats 0, 2, 3 and 4. + + It mixes option-order sensitivity with nondeterminism, so it is only readable next + to the repeat flip rate. Returns ``(None, 0)`` for a set whose options are never + reordered, which is every ``noul`` and ``score`` set. + """ + rows = scored(decisions) + if rows and rows[0].get("type") != CHOICE: + return None, 0 + return flip_rate(decisions, (0, 2, 3, 4)) + + +# ------------------------------------------------------------------ latency, cost + +def percentile(values, fraction: float): + """Nearest-rank: sort, take the ``ceil(fraction * n)``th, no interpolation. + + So a p95 is always a request that really took that long, which is the same rule + ``metrics.percentile`` follows and the one the recipe states. + """ + ordered = sorted(v for v in values if v is not None) + if not ordered: + return None + index = math.ceil(fraction * len(ordered)) - 1 + return ordered[min(max(index, 0), len(ordered) - 1)] + + +def latency(decisions) -> dict: + """p50 and p95 of the round trip, over the decisions that came back.""" + walls = [d.get("wall_ms") for d in scored(decisions) + if d.get("wall_ms") is not None] + return {"p50_ms": percentile(walls, 0.5), "p95_ms": percentile(walls, 0.95)} + + +def tokens(decisions) -> dict: + """Summed reported usage. A decision the endpoint reported nothing for adds 0.""" + return {"in": sum(int(d.get("tokens_in") or 0) for d in decisions), + "out": sum(int(d.get("tokens_out") or 0) for d in decisions)} + + +def cost_usd(counts: dict, input_usd_per_mtok: float = 0.0, + output_usd_per_mtok: float = 0.0) -> float: + """Reported tokens at a posted rate. Both rates 0 means the column reads $0. + + Never an estimate: an endpoint that reports no usage contributes no tokens, and a + backend nobody bills per token for is $0 rather than a guess at electricity. + """ + return (counts.get("in", 0) * float(input_usd_per_mtok) + + counts.get("out", 0) * float(output_usd_per_mtok)) / 1e6 + + +def usd_per_1k(cost: float, decisions_count: int): + """``$ per 1k decisions``: the board's cost column. None over no decisions.""" + if not decisions_count: + return None + return cost / decisions_count * 1000.0 + + +def questions_per_second(decisions_count: int, seconds: float): + """Decisions divided by the run's own wall clock, or None when it took no time. + + It is a throughput of the RUN and not of the endpoint: it moves with + ``--concurrency`` and with whatever else the node was serving, which is why the + record carries the concurrency next to it. + """ + if not seconds or seconds <= 0 or not decisions_count: + return None + return decisions_count / float(seconds) + + +# ------------------------------------------------- against a gold DISTRIBUTION + +#: Predicted probabilities are floored here before a log, so one confident miss is a +#: large KL rather than an infinite one that makes the mean unreadable. The floor is in +#: the record beside the number. +KL_FLOOR = 1e-6 + + +def with_gold(decisions) -> list: + """Scored decisions whose set ships a gold DISTRIBUTION, not just a label.""" + return [d for d in scored(decisions) if isinstance(d.get("gold"), dict) + and d["gold"]] + + +def soft_accuracy(decisions): + """Mean gold probability of the option the system picked. + + The figure a soft-labelled set asks for instead of accuracy: on a question where the + teacher itself split 0.55/0.45, picking the 0.45 option is most of a right answer and + exact-match accuracy calls it a miss. A pick the gold gave no mass contributes 0, and + a decision with no pick (malformed, or a yes/no at exactly 0.5) contributes 0 too. + """ + rows = with_gold(decisions) + if not rows: + return None + total = 0.0 + for decision in rows: + pick = decision.get("pick") + if pick is not None: + total += float(decision["gold"].get(pick, 0.0)) + return total / len(rows) + + +def total_variation(decisions): + """Mean total-variation distance between the answer and the gold: ``0.5 * sum|p-g|``. + + 0 is an exact match of the teacher's spread and 1 is disjoint. Unlike the KL it is + bounded and symmetric, so it is the one to read when a system is confident and the + gold is not. + """ + rows = with_gold(decisions) + if not rows: + return None + total = 0.0 + for decision in rows: + vector = decision.get("vector") or uniform(decision["options"]) + gold = decision["gold"] + total += 0.5 * sum(abs(float(vector.get(option, 0.0)) + - float(gold.get(option, 0.0))) + for option in decision["options"]) + return total / len(rows) + + +def kl_from_gold(decisions, floor: float = KL_FLOOR): + """Mean ``sum_k g_k * log(g_k / p_k)``, gold first, predictions floored. + + This is the number that separates "picks the right label" from "reproduces the + teacher's uncertainty": a one-hot answer that happens to be right scores well on + accuracy and badly here. A zero gold term contributes nothing, which is the usual + convention; a zero PREDICTED probability under positive gold would be infinite, so + predictions are floored at ``floor`` and the floor is recorded with the figure. + """ + rows = with_gold(decisions) + if not rows: + return None + total = 0.0 + for decision in rows: + vector = decision.get("vector") or uniform(decision["options"]) + gold = decision["gold"] + for option in decision["options"]: + g = float(gold.get(option, 0.0)) + if g <= 0: + continue + p = max(float(vector.get(option, 0.0)), floor) + total += g * math.log(g / p) + return total / len(rows) + + +def brier_from_gold(decisions): + """Mean ``sum_k (p_k - g_k)^2`` against the gold distribution. + + Explicitly OUR definition. A soft-gold set's own card may print a column called + Brier without publishing the arithmetic behind it, so this number is not that + number and must not be put in its column. + """ + rows = with_gold(decisions) + if not rows: + return None + total = 0.0 + for decision in rows: + vector = decision.get("vector") or uniform(decision["options"]) + gold = decision["gold"] + total += sum((float(vector.get(option, 0.0)) - float(gold.get(option, 0.0))) ** 2 + for option in decision["options"]) + return total / len(rows) + + +def ordinal_mae(decisions): + """JevBench's ordinal MAE: the probability-weighted level against the labeled one. + + ``mean |E[level] - label level|`` over the score decisions. Reported BESIDE argmax + accuracy and never instead of it, which is the rule JevBench states: a model that + spreads its mass either side of the right level is wrong on accuracy and close here, + and both facts matter to something that sorts by the number. ``None`` when nothing + here is a score question. + """ + rows = [d for d in scored(decisions) if d.get("type") == SCORE] + if not rows: + return None + total = 0.0 + for decision in rows: + options = decision["options"] + vector = decision.get("vector") or uniform(options) + expected = sum(index * float(vector.get(option, 0.0)) + for index, option in enumerate(options)) + total += abs(expected - options.index(decision["label"])) + return total / len(rows) + + +def schema_validity(decisions, tolerance: float = SUM_TOLERANCE): + """Share of answers that were a usable distribution over the exact label set. + + JevBench's name and JevBench's rule: a distribution has to cover the label set, sit + in [0, 1] and sum to 1. ``tolerance`` is how far the sum may be off before the answer + is invalid rather than renormalized. ``None`` when nothing came back. + """ + rows = scored(decisions) + if not rows: + return None + valid = 0 + for decision in rows: + if decision.get("malformed"): + continue + total = decision.get("sum_before_normalize") + if total is None: + # A one-hot answer has no stated sum to check; it is a valid answer that + # simply carries no spread, which `one_hot` already says. + valid += 1 + continue + if abs(float(total) - 1.0) <= tolerance + EPSILON: + valid += 1 + return valid / len(rows) + + +def within_one_level(decisions): + """Score questions only: share of picks within one level of the labeled one. + + An ordinal set's accuracy punishes a one-level miss exactly as hard as a four-level + miss, which is what the ranked probability score exists to fix for the loss; this is + the same correction for the pick. None when nothing here is a score question. + """ + rows = [d for d in scored(decisions) if d.get("type") == SCORE] + if not rows: + return None + hits = 0 + for decision in rows: + options = decision["options"] + pick = decision.get("pick") + if pick is None or pick not in options: + continue + if abs(options.index(pick) - options.index(decision["label"])) <= 1: + hits += 1 + return hits / len(rows) + + +def gold_block(decisions) -> dict: + """The against-the-gold-distribution block, or ``{}`` when the set ships no gold.""" + rows = with_gold(decisions) + if not rows: + return {} + return {"over": len(rows), + "soft_accuracy": soft_accuracy(decisions), + "total_variation": total_variation(decisions), + "kl": kl_from_gold(decisions), + "kl_floor": KL_FLOOR, + "brier_vs_gold": brier_from_gold(decisions), + "definition": "soft_accuracy is the gold probability of the pick; " + "total_variation is 0.5*sum|p-g|; kl is sum g*log(g/p) with " + "p floored at kl_floor; brier_vs_gold is sum (p-g)^2. These " + "are AINode's definitions, not a set card's column names"} + + +# ------------------------------------------------------------------ the block + +def _round(value, places=4): + return None if value is None else round(float(value), places) + + +#: Which recipe a metrics block's figures follow. Every block carries it, because two +#: numbers under one name and two recipes are the way a comparison becomes a lie. +RECIPE_JEVALS = "jevals-0.1.0" + + +def types_present(decisions) -> list: + """The primitives among these decisions, in ``TYPES`` order.""" + seen = {d.get("type") for d in scored(decisions)} + return [t for t in TYPES if t in seen] + + +def answer_spaces(decisions) -> dict: + """``{name: [decisions]}`` grouped by ANSWER SPACE, in first-seen order. + + An answer space is one ``(type, options)`` pair, and it is the unit the Decision Score + is really defined over: the label prior is the base rates of the labels IN that option + list, so pooling two questions with different option lists would build a prior over an + answer space neither of them has. A set asking five differently typed questions over + one state has five of these, and the set's own card says to read every score against + its own question rather than against the mean. + + The name is the question's own ``space`` (a mixed manifest writes ``group/question``), + and ``type#n`` in first-seen order when the questions do not carry one, with the option + list recorded beside the block so a reader can see which space it was. + """ + out = {} + keys = {} + for decision in scored(decisions): + key = (decision.get("type"), tuple(decision.get("options") or ())) + if key not in keys: + name = decision.get("space") + if not name: + seen = sum(1 for k in keys if k[0] == key[0]) + name = f"{key[0]}#{seen + 1}" + keys[key] = name + out[name] = [] + out[keys[key]].append(decision) + return out + + +def summarize(decisions, input_usd_per_mtok: float = 0.0, + output_usd_per_mtok: float = 0.0, seconds: float = 0.0, + bins: int = BINS, recipe: str = RECIPE_JEVALS) -> dict: + """One set's metrics block: every number the recipe defines, over its decisions. + + The guessing floor travels with the figures rather than sitting in a footnote: + ``prior_accuracy`` is the base-rate answer's accuracy and ``loss_prior`` is the + baseline the Decision Score is measured against, so a reader never has to go and + find what 0 meant on this set. + + **A set holding more than one primitive is broken down by primitive**, because the + two losses are different arithmetic, the label prior is per answer space, and the + board's gate is per primitive. Such a block carries ``type: "mixed"``, a ``types`` + map of one full block each, and a ``decision_score`` that is the plain mean of the + per-primitive scores, which is the recipe's rule for a tab holding more than one + task. Its own accuracy and latency are over all the decisions, because those do mean + the same thing across primitives. + """ + rows = scored(decisions) + groups = by_item(rows) + present = types_present(decisions) + spaces = answer_spaces(decisions) + if len(spaces) > 1: + return _summarize_multi(decisions, present, spaces, input_usd_per_mtok, + output_usd_per_mtok, seconds, bins, recipe) + qtype = present[0] if present else None + system_loss = mean_item_loss(decisions) + baseline_loss = prior_loss(decisions) + token_counts = tokens(rows) + cost = cost_usd(token_counts, input_usd_per_mtok, output_usd_per_mtok) + gate = PUBLISHED_GATES.get(qtype) if qtype else None + repeat_flips, repeat_flip_n = repeat_flip_rate(decisions) + order_flips, order_flip_n = order_flip_rate(decisions) + any_flips, any_flip_n = pick_flips(decisions) + swing = confidence_swing(decisions) + block = { + "recipe": recipe, + "type": qtype, + "items": len(groups), + "repeats": max((len(r) for r in groups.values()), default=0), + "decisions": len(rows), + "failed": len(failed(decisions)), + "malformed": sum(1 for d in rows if d.get("malformed")), + "one_hot": sum(1 for d in rows if d.get("one_hot")), + "calibrated_over": len(calibratable(decisions)), + "accuracy": _round(accuracy(decisions)), + # JevBench calls this majority_class_accuracy and Jevals calls its baseline the + # label prior. Same number on a set like these, and the record carries both + # names so a reader of either board knows what it is. + "prior_accuracy": _round(prior_accuracy(decisions)), + "majority_class_accuracy": _round(prior_accuracy(decisions)), + "schema_validity": _round(schema_validity(decisions)), + "schema_validity_strict": _round( + schema_validity(decisions, SUM_TOLERANCE_STRICT)), + "decision_score": _round(decision_score(system_loss, baseline_loss), 2), + "loss": _round(system_loss, 6), + "loss_prior": _round(baseline_loss, 6), + "ece_points": _round(ece_points(decisions, bins), 2), + "bins": [{**b, "accuracy": _round(b["accuracy"]), + "confidence": _round(b["confidence"])} + for b in reliability(decisions, bins)], + "handoff_95": None, + "gate": None, + "gate_local": gate_local(decisions), + "pick_flip_rate": _round(any_flips), + "pick_flip_over": any_flip_n, + "confidence_swing": (None if swing is None else + {"max": _round(swing["max"]), + "mean": _round(swing["mean"]), + "question": swing["question"], + "over": swing["over"]}), + "repeat_flip_rate": _round(repeat_flips), + "repeat_flip_over": repeat_flip_n, + "order_flip_rate": _round(order_flips), + "order_flip_over": order_flip_n, + "questions_per_second": _round(questions_per_second(len(rows), seconds), 3), + "tokens": token_counts, + "cost_usd": round(cost, 6), + "usd_per_1k_decisions": _round(usd_per_1k(cost, len(rows)), 6), + } + hand = handoff(decisions) + if hand: + block["handoff_95"] = {"threshold": hand["threshold"], + "share": _round(hand["share"]), + "n": hand["n"], + "accuracy": _round(hand["accuracy"])} + coverage = gate_coverage(decisions, gate) + if coverage: + block["gate"] = {"threshold": coverage["gate"], + "source": PUBLISHED_GATE_SOURCE, + "coverage": _round(coverage["coverage"]), + "n": coverage["n"], + "accuracy": _round(coverage["accuracy"])} + elif qtype and gate is None: + block["gate"] = {"threshold": None, "source": PUBLISHED_GATE_SOURCE, + "coverage": None, "n": 0, "accuracy": None} + gold = gold_block(decisions) + if gold: + block["vs_gold"] = {key: (_round(value, 6) if isinstance(value, float) + else value) + for key, value in gold.items()} + if qtype == SCORE: + block["within_one_level"] = _round(within_one_level(decisions)) + block["ordinal_mae"] = _round(ordinal_mae(decisions)) + block.update(latency(decisions)) + return block + + +def _summarize_multi(decisions, present, spaces, input_usd_per_mtok, + output_usd_per_mtok, seconds, bins, recipe) -> dict: + """A set holding more than one ANSWER SPACE: one block per space, plus the means. + + The Decision Score is the plain mean over the spaces, which is the recipe's rule for + more than one task applied to the unit the score is defined over. ``types`` rolls the + spaces up per primitive the way a board shows one, and the pooled ``loss`` and + ``loss_prior`` are null because a loss over two different answer spaces is not a + number. + """ + rows = scored(decisions) + per_space = {} + for name, mine in spaces.items(): + block = summarize(mine, input_usd_per_mtok=input_usd_per_mtok, + output_usd_per_mtok=output_usd_per_mtok, + seconds=seconds, bins=bins, recipe=recipe) + block["options"] = list(mine[0].get("options") or ()) + per_space[name] = block + space_scores = [b["decision_score"] for b in per_space.values()] + mean_score = (round(sum(space_scores) / len(space_scores), 2) + if space_scores and all(s is not None for s in space_scores) + else None) + per_type = {} + for qtype in present: + mine = [b for b in per_space.values() if b["type"] == qtype] + scores = [b["decision_score"] for b in mine] + per_type[qtype] = { + "recipe": recipe, "type": qtype, "spaces": len(mine), + "decisions": sum(b["decisions"] for b in mine), + "accuracy": _round(accuracy([d for d in decisions + if d.get("type") == qtype])), + "prior_accuracy": _round( + sum(b["prior_accuracy"] or 0.0 for b in mine) / len(mine) + if mine else None), + "decision_score": (round(sum(scores) / len(scores), 2) + if scores and all(s is not None for s in scores) + else None), + "ece_points": _round(ece_points([d for d in decisions + if d.get("type") == qtype], bins), 2), + } + token_counts = tokens(rows) + cost = cost_usd(token_counts, input_usd_per_mtok, output_usd_per_mtok) + any_flips, any_flip_n = pick_flips(decisions) + swing = confidence_swing(decisions) + block = { + "recipe": recipe, + "type": "mixed" if len(present) > 1 else (present[0] if present else None), + "spaces": per_space, + "types": per_type, + "items": len(by_item(rows)), + "repeats": max((len(r) for r in by_item(rows).values()), default=0), + "decisions": len(rows), + "failed": len(failed(decisions)), + "malformed": sum(1 for d in rows if d.get("malformed")), + "one_hot": sum(1 for d in rows if d.get("one_hot")), + "calibrated_over": len(calibratable(decisions)), + "accuracy": _round(accuracy(decisions)), + "prior_accuracy": _round( + sum(b["prior_accuracy"] or 0.0 for b in per_space.values()) + / len(per_space) if per_space else None), + "majority_class_accuracy": _round( + sum(b["prior_accuracy"] or 0.0 for b in per_space.values()) + / len(per_space) if per_space else None), + "schema_validity": _round(schema_validity(decisions)), + "schema_validity_strict": _round( + schema_validity(decisions, SUM_TOLERANCE_STRICT)), + "decision_score": mean_score, + "decision_score_is": "the plain mean of the per-answer-space Decision Scores, " + "which is the recipe's rule for more than one task applied " + "to the unit the score is defined over", + "loss": None, + "loss_prior": None, + "ece_points": _round(ece_points(decisions, bins), 2), + "bins": [{**b, "accuracy": _round(b["accuracy"]), + "confidence": _round(b["confidence"])} + for b in reliability(decisions, bins)], + "handoff_95": None, + "gate": None, + "gate_local": gate_local(decisions), + "pick_flip_rate": _round(any_flips), + "pick_flip_over": any_flip_n, + "confidence_swing": (None if swing is None else + {"max": _round(swing["max"]), + "mean": _round(swing["mean"]), + "question": swing["question"], + "over": swing["over"]}), + "repeat_flip_rate": _round(repeat_flip_rate(decisions)[0]), + "repeat_flip_over": repeat_flip_rate(decisions)[1], + "order_flip_rate": None, + "order_flip_over": 0, + "questions_per_second": _round(questions_per_second(len(rows), seconds), 3), + "tokens": token_counts, + "cost_usd": round(cost, 6), + "usd_per_1k_decisions": _round(usd_per_1k(cost, len(rows)), 6), + } + hand = handoff(decisions) + if hand: + block["handoff_95"] = {"threshold": hand["threshold"], + "share": _round(hand["share"]), "n": hand["n"], + "accuracy": _round(hand["accuracy"])} + gold = gold_block(decisions) + if gold: + block["vs_gold"] = {key: (_round(value, 6) if isinstance(value, float) + else value) + for key, value in gold.items()} + if SCORE in present: + block["within_one_level"] = _round(within_one_level(decisions)) + block["ordinal_mae"] = _round(ordinal_mae(decisions)) + block.update(latency(decisions)) + return block + + +def summarize_sets(decisions, set_names, **kw) -> dict: + """One block per set, in the order given, and nothing for a set with no decisions. + + A set nobody ran is absent rather than a block of nulls, the same rule the rest of + ``bench/`` follows: a measurement nobody took is not a zero. + """ + out = {} + for name in set_names: + mine = [d for d in decisions if d.get("set") == name] + if mine: + out[name] = summarize(mine, **kw) + return out + + +def mean_decision_score(blocks) -> dict: + """The plain mean of the set Decision Scores, and which sets went into it. + + The recipe's rule for a tab with more than one task. It is only reported when every + set asked for has a score, because a mean over two of three sets is a different + number wearing the same name. + """ + scores = [(name, block.get("decision_score")) for name, block in blocks.items()] + have = [(name, value) for name, value in scores if value is not None] + out = {"sets": [name for name, _ in scores], + "scored": [name for name, _ in have], "mean_decision_score": None} + if have and len(have) == len(scores): + out["mean_decision_score"] = round(sum(v for _, v in have) / len(have), 2) + return out + + +__all__ = ["BINS", "CHOICE", "EPSILON", "GATE_MAX_ERROR", "GATE_MIN_DECISIONS", + "GRID", "HANDOFF_ACCURACY", "HANDOFF_MIN_DECISIONS", "KL_FLOOR", "NOUL", + "PUBLISHED_GATES", "PUBLISHED_GATE_SOURCE", "RECIPE_JEVALS", "REPEATS", + "SCORE", "SUM_TOLERANCE", "SUM_TOLERANCE_STRICT", "TYPES", "above", + "accuracy", "bin_index", "brier_from_gold", "brier_loss", "by_item", + "calibratable", "confidence_swing", "cost_usd", "decision_loss", + "decision_score", "ece", "ece_points", "failed", "flip_rate", + "gate_coverage", "gate_local", "gold_block", "grid_thresholds", "handoff", + "is_correct", "is_probability", "kl_from_gold", "label_prior", "latency", + "loss_for", "mean_decision_score", "mean_item_loss", "normalize_vector", + "order_flip_rate", "ordinal_mae", "percentile", "pick_flips", + "pick_from_vector", "prior_accuracy", "prior_loss", "questions_per_second", + "reliability", "repeat_flip_rate", "round_half_up", "rps_loss", + "schema_validity", "scored", "soft_accuracy", "summarize", "summarize_sets", + "answer_spaces", "tokens", "total_variation", "types_present", "uniform", + "usd_per_1k", "with_gold", "within_one_level"] diff --git a/ainode/bench/decide/sets.py b/ainode/bench/decide/sets.py new file mode 100644 index 00000000..b3d90610 --- /dev/null +++ b/ainode/bench/decide/sets.py @@ -0,0 +1,740 @@ +"""The three public Jevals sets: the committed manifests, the download, the questions. + +``bench/decide/sets/.json`` is Jevals' own suite file, committed verbatim +(CC-BY-4.0, cited in ``bench/decide/JEVALS.md``). It names the dataset, the config, the +split, the pinned revision, the licence, the seed, the instructions, the criteria, the +option list, and for each of the 300 items an id, an upstream row index, the label and a +``state_sha256``. What it does NOT carry is any dataset item text: Jevals does not +republish item text, every item links to its upstream row, and the three upstream +licences are the item text's licence and not ours to relicense. This bench follows that +exactly, which also keeps three sets of 300 long biomedical abstracts and chat +transcripts out of the repo. + +So the item text is **downloaded, never committed**: + + python3 scripts/ainode-bench.py decide download # all three + python3 scripts/ainode-bench.py decide download pubmedqa # one + +writes ``bench/decide/cache/.questions.json`` (gitignored) and nothing else. That +file is a question file: the Jev question shape (``type``, ``instructions``, +``criteria``) plus one entry per question holding an id, a state and the published hash, +which is the same shape ``--questions `` takes, so a private blind set can be +scored by the same code without ever being committed. + +**A label is never inside a question.** The file's labels live in one separate +``labels`` map, keyed by question id, and a question object carrying anything +answer-key-shaped is a load error. That is not tidiness: the transport is handed a +question and builds the wire body out of it, so a label it never sees is a label it +cannot leak into the prompt, and the rule is checked from both ends +(:func:`answer_key_leaks` here, ``suite.wire_leaks`` on the assembled body). + +**Every state is checked against the published ``state_sha256`` before it is written**, +and a mismatch is a load error naming the item. That check is what makes a number here +comparable to a board number: it proves the bytes this bench put in front of the model +are the bytes Jevals put in front of theirs, which is a stronger guarantee than pinning +a dataset revision, and it is how the state construction for all three sets was +recovered in the first place (see JEVALS.md, "What we verified rather than assumed"). + +Stdlib only: ``urllib`` for the transport, ``gzip`` for the one set whose upstream file +is compressed, ``hashlib`` for the check. +""" +from __future__ import annotations + +import gzip +import hashlib +import json +import os +import pathlib +import time +import urllib.error +import urllib.parse +import urllib.request + +#: The sets this bench knows, in the order a run lists them. The first three are the +#: Jevals boards' one-task-per-primitive sets; the fourth is the one public set already +#: in the ``/v1/systemone`` request shape that ships gold DISTRIBUTIONS rather than only +#: labels, and it is mixed-primitive (five questions over one shared state). +SUITES = ("pubmedqa", "banking77", "helpsteer2", "typed-decisions") + +#: Which published recipe a set's own third-party numbers follow, so a record can say +#: which recipe each figure is comparable to. This bench computes the Jevals formulas on +#: all four sets; a set's own card may print differently named columns, and naming the +#: recipe of record is how those two are kept apart. +RECIPE_OF_RECORD = {"pubmedqa": "jevals-0.1.0", "banking77": "jevals-0.1.0", + "helpsteer2": "jevals-0.1.0", + "typed-decisions": "typed-decisions-card"} + +#: Sets a listed system is known to have TRAINED on, with the primary source that says +#: so, so a record flags its own contamination rather than leaving a reader to find out. +#: A row for one of these systems on that set measures memorisation and not decision +#: quality. Only training exposure a project's own card or README states goes in here; a +#: guess from a name does not. +CONTAMINATION = { + "banking77": [ + {"system": "Kev (jaredpalmer/kev)", + "evidence": "the kev-9b model card front matter lists " + "legacy-datasets/banking77 under `datasets:`, and the README's " + "decision-v7 recipe is 10,000 examples from ten public datasets", + "source": "https://raw.githubusercontent.com/jaredpalmer/kev/main/" + "docs/model-cards/kev-9b.md"}, + {"system": "Laya (convaiinnovations/laya)", + "evidence": "its write-up lists banking intents among the training groups, and " + "Banking77 is also in its own published eval list", + "source": "https://raw.githubusercontent.com/NandhaKishorM/laya/main/" + "BENCHMARKS.md"}, + ], + "pubmedqa": [ + {"system": "decider (Mapika/decider)", + "evidence": "PubMedQA is named among the held-out datasets of its 94-task " + "regression set, so it sits inside that project's development loop " + "even though the card calls it held out", + "source": "https://raw.githubusercontent.com/Mapika/decider/main/README.md"}, + ], +} + +ENV_SETS = "AINODE_DECIDE_SETS" +ENV_CACHE = "AINODE_DECIDE_CACHE" + +_REPO = pathlib.Path(__file__).resolve().parents[3] +_REPO_SETS = _REPO / "bench" / "decide" / "sets" +_REPO_CACHE = _REPO / "bench" / "decide" / "cache" +_HOME_SETS = pathlib.Path.home() / ".ainode" / "bench" / "decide" / "sets" +_HOME_CACHE = pathlib.Path.home() / ".ainode" / "bench" / "decide" / "cache" + +#: Hugging Face's rows API, which answers JSON and needs no pip install. It does not +#: take a revision, which is exactly why every row is hash-checked: a drifted row fails +#: the check and is a load error, where a revision pin would only have been a promise. +ROWS_API = "https://datasets-server.huggingface.co/rows" +#: Its page ceiling. +ROWS_PAGE = 100 +#: ``https://huggingface.co/datasets//resolve//``, which DOES pin a +#: revision and is used for the one set whose upstream file is stdlib-readable. +RESOLVE = "https://huggingface.co/datasets/{dataset}/resolve/{revision}/{path}" + +HTTP_TIMEOUT = 120 +HTTP_RETRIES = 6 +HTTP_BACKOFF = 3.0 +#: Ceiling on one backoff wait, including one the server asked for. +HTTP_BACKOFF_MAX = 60.0 +#: Courtesy pause between pages of a public API nobody is paying us to hammer. The rows +#: API rate limits an unauthenticated caller partway through a 31-page fetch at a quarter +#: of a second, so this is deliberately slower than it needs to be. +PAGE_PAUSE = 1.0 + + +class SetError(RuntimeError): + """A manifest, a download or a question file that cannot be used as asked.""" + + +# ------------------------------------------------------------------ the state bytes + +def state_json(state) -> str: + """The state as the bytes its hash is over: compact, insertion order, real UTF-8. + + ``JSON.stringify`` with no arguments, which is what the recipe says the hash is + taken of: no spaces, no key sorting (insertion order is the order the state fields + are listed in), and non-ASCII written as itself rather than as an escape. All three + differ from ``json.dumps``' defaults, and getting any one of them wrong moves every + hash. + """ + if isinstance(state, str): + return state + return json.dumps(state, ensure_ascii=False, separators=(",", ":")) + + +def state_sha256(state) -> str: + return hashlib.sha256(state_json(state).encode("utf-8")).hexdigest() + + +# ------------------------------------------------------------------ paths + +def sets_dir() -> pathlib.Path: + """``$AINODE_DECIDE_SETS``, else the repo's manifests, else the installed copy.""" + override = os.environ.get(ENV_SETS) + if override: + return pathlib.Path(override).expanduser() + if _REPO_SETS.is_dir(): + return _REPO_SETS + return _HOME_SETS + + +def cache_dir() -> pathlib.Path: + """``$AINODE_DECIDE_CACHE``, else ``bench/decide/cache`` beside the manifests.""" + override = os.environ.get(ENV_CACHE) + if override: + return pathlib.Path(override).expanduser() + if _REPO_SETS.is_dir(): + return _REPO_CACHE + return _HOME_CACHE + + +def manifest_path(suite_id: str) -> pathlib.Path: + return sets_dir() / f"{suite_id}.json" + + +def questions_path(suite_id: str) -> pathlib.Path: + return cache_dir() / f"{suite_id}.questions.json" + + +# ------------------------------------------------------------------ the manifests + +def load_manifest(suite_id: str) -> dict: + """One committed suite file, validated down to the fields the loader relies on. + + Two manifest shapes, one loader. A **single-primitive** manifest (the three Jevals + sets) states one ``instructions``, one ``options`` list and one ``criteria`` for the + whole set. A **mixed** one (``typed-decisions``) states ``question_schemas`` per + workflow instead, because it asks five differently typed questions over one state and + a set-level option list would be meaningless. + """ + path = manifest_path(suite_id) + if not path.is_file(): + raise SetError(f"no suite manifest at {path}; known suites: " + f"{', '.join(SUITES)}") + try: + doc = json.loads(path.read_text()) + except json.JSONDecodeError as exc: + raise SetError(f"{path} is not valid JSON: {exc}") from exc + for key in ("id", "version", "primitive", "dataset", "config", "split", + "hf_revision", "license", "items", "state_fields", "n_items"): + if key not in doc: + raise SetError(f"{path} is missing '{key}'") + if doc["id"] != suite_id: + raise SetError(f"{path} calls itself {doc['id']!r}, not {suite_id!r}") + schemas = doc.get("question_schemas") + if schemas is None: + for key in ("instructions", "options"): + if key not in doc: + raise SetError(f"{path} states no 'question_schemas' and is missing " + f"'{key}'") + options = doc["options"] + if not isinstance(options, list) or len(options) < 2: + raise SetError(f"{path} declares fewer than two options") + else: + if not isinstance(schemas, dict) or not schemas: + raise SetError(f"{path} has a 'question_schemas' that is not a map") + for group, block in schemas.items(): + if not isinstance(block, dict) or not block: + raise SetError(f"{path}: question_schemas[{group!r}] holds no questions") + for qkey, spec in block.items(): + for key in ("type", "instructions", "options"): + if key not in spec: + raise SetError(f"{path}: {group}/{qkey} is missing '{key}'") + if len(spec["options"]) < 2: + raise SetError(f"{path}: {group}/{qkey} declares fewer than two " + "options") + items = doc["items"] + if not isinstance(items, list) or len(items) != int(doc["n_items"]): + raise SetError(f"{path} says n_items={doc['n_items']} and holds " + f"{len(items) if isinstance(items, list) else 'none'}") + for item in items: + for key in ("item_id", "row_idx", "state_sha256", "target"): + if key not in item: + raise SetError(f"{path}: an item is missing '{key}'") + width = len(manifest_spec(doc, item)["options"]) + if not 0 <= int(item["target"]) < width: + raise SetError(f"{path}: item {item['item_id']} has target " + f"{item['target']}, outside its {width} options") + return doc + + +def manifest_spec(doc: dict, item: dict) -> dict: + """The typed question for one manifest item: the set's own, or its group's. + + A mixed manifest states its schemas per ``group`` (typed-decisions: per workflow) and + each item names its own group and carries the question key after the colon in its + ``item_id``, so one upstream row becomes as many items as it asks questions. The + group is IN THE ITEM rather than derived from the row, because the same question key + means different things in two groups (``action`` has four options in one workflow and + five in another) and a manifest has to be readable without a download. + """ + schemas = doc.get("question_schemas") + if not schemas: + return {"type": doc["primitive"], "instructions": doc["instructions"], + "criteria": doc.get("criteria"), "options": list(doc["options"])} + qkey = str(item["item_id"]).rsplit(":", 1)[-1] + group = item.get("group") + block = schemas.get(group) + if block is None: + raise SetError(f"{doc['id']}: item {item['item_id']} is in group {group!r}, " + f"which this manifest does not declare (declared: " + f"{', '.join(sorted(schemas))})") + if qkey not in block: + raise SetError(f"{doc['id']}: item {item['item_id']} names question {qkey!r}, " + f"which group {group!r} does not declare") + return dict(block[qkey]) + + +def criteria_for(doc: dict): + """The criteria as the manifest states them: a map for choice and noul, a list for + score. Passed through unchanged, because the wording IS the question.""" + return doc.get("criteria") + + +# ------------------------------------------------------------------ the transport + +def _get(url: str, timeout: int = HTTP_TIMEOUT, sleep=time.sleep) -> bytes: + """One GET with backoff. A public API's 429 or hiccup is not a load error yet. + + A 429 is the expected failure here, not an exception: the rows API rate limits an + unauthenticated caller partway through a multi-page fetch, so it backs off (honouring + ``Retry-After`` when the server sends one) and keeps going. Only a 4xx that says the + request itself is wrong stops early, because retrying a 404 is just slower. + """ + last = "" + for attempt in range(HTTP_RETRIES): + wait = HTTP_BACKOFF * (attempt + 1) + try: + with urllib.request.urlopen(url, timeout=timeout) as response: + return response.read() + except urllib.error.HTTPError as exc: + last = f"HTTP {exc.code}" + if exc.code in (400, 401, 403, 404): + break + if exc.code == 429: + try: + asked = float(exc.headers.get("Retry-After") or 0) + except (TypeError, ValueError): + asked = 0.0 + wait = max(wait, min(asked, HTTP_BACKOFF_MAX), HTTP_BACKOFF * 4) + except Exception as exc: # noqa: BLE001 + last = f"{type(exc).__name__}: {exc}" + if attempt + 1 < HTTP_RETRIES: + sleep(min(wait, HTTP_BACKOFF_MAX)) + raise SetError(f"could not fetch {url}: {last}. A 429 here is the rows API rate " + "limiting an unauthenticated caller; wait a minute and run the " + "download again, it starts from the beginning of that set") + + +def fetch_rows(dataset: str, config: str, split: str, up_to: int, + progress=None) -> dict: + """``{row_idx: row}`` from the rows API, paged to cover every index up to ``up_to``. + + The whole prefix rather than only the wanted indices, because the API pages by + offset and a page is one request either way: 31 requests for Banking77's 3,076 rows, + 10 for PubMedQA's 1,000. + """ + out = {} + offset = 0 + while offset <= up_to: + query = urllib.parse.urlencode({"dataset": dataset, "config": config, + "split": split, "offset": offset, + "length": ROWS_PAGE}) + page = json.loads(_get(f"{ROWS_API}?{query}").decode("utf-8")) + rows = page.get("rows") or [] + if not rows: + break + for row in rows: + out[int(row["row_idx"])] = row["row"] + offset += ROWS_PAGE + if progress: + progress(min(offset, up_to + 1), up_to + 1) + if offset <= up_to: + time.sleep(PAGE_PAUSE) + return out + + +def fetch_jsonl(dataset: str, revision: str, path: str, progress=None) -> dict: + """``{row_idx: row}`` from a JSONL (or gzipped JSONL) file at a PINNED revision. + + The row index is the line number, which is what a split's own file ordering means + and what the manifest's ``row_idx`` counts. + """ + raw = _get(RESOLVE.format(dataset=urllib.parse.quote(dataset), + revision=revision, path=path)) + if path.endswith(".gz"): + raw = gzip.decompress(raw) + out = {} + for index, line in enumerate(raw.decode("utf-8").splitlines()): + if line.strip(): + out[index] = json.loads(line) + if progress: + progress(len(out), len(out)) + return out + + +# ------------------------------------------------------------------ the three sets + +def _pubmedqa_state(row) -> dict: + """``{"question": ..., "context": [passages]}``. + + The manifest's state fields are ``question`` and ``context.contexts``, but the state + object's second key is ``context`` holding the list itself. That is not published; + it was recovered from the hashes and matches all 300. + """ + return {"question": row["question"], "context": list(row["context"]["contexts"])} + + +def _banking77_state(row) -> dict: + """``{"message": ...}``. The upstream field is ``text``; the state key is + ``message``. Also recovered from the hashes, and it matches all 300.""" + return {"message": row["text"]} + + +def _helpsteer2_state(row) -> dict: + return {"prompt": row["prompt"], "response": row["response"]} + + +def _typed_decisions_state(row): + """The state ships as a JSON STRING and is already the request body's own state. + + Parsed to the object every transport here works with; the hash is over our canonical + re-serialization of that object, so it is consistent with the other three sets. The + manifest says the hashes are ours and not upstream's, because this dataset publishes + none of its own. + """ + state = row["state"] + return json.loads(state) if isinstance(state, str) else state + + +#: Per suite: how to fetch the split, and how to build one state out of a row. +SOURCES = { + "pubmedqa": {"fetch": "rows", "state": _pubmedqa_state}, + "banking77": {"fetch": "rows", "state": _banking77_state}, + "helpsteer2": {"fetch": "jsonl", "path": "validation.jsonl.gz", + "state": _helpsteer2_state}, + "typed-decisions": {"fetch": "rows", "state": _typed_decisions_state}, +} + + +def build_questions(doc: dict, rows: dict) -> dict: + """The question file for one suite: the Jev shape plus the label and the hash. + + Strict, the way the rest of this bench loads data: a missing row, a state whose hash + does not match the published one, or a label that does not line up is an error + naming the item, never a skipped item. A set that quietly ran 297 of 300 would + publish an accuracy against a count nobody chose, and a set whose states drifted + would publish a Decision Score that is not comparable to the board it is next to. + """ + suite_id = doc["id"] + build_state = SOURCES[suite_id]["state"] + mixed = bool(doc.get("question_schemas")) + questions = [] + labels = {} + states = {} + for item in doc["items"]: + index = int(item["row_idx"]) + row = rows.get(index) + if row is None: + raise SetError(f"{suite_id}: row {index} for item {item['item_id']} is " + "not in the download") + if index not in states: + try: + states[index] = build_state(row) + except (KeyError, TypeError, ValueError) as exc: + raise SetError( + f"{suite_id}: row {index} has no {exc} field; the upstream split is " + "not the one this manifest was built from") from exc + state = states[index] + digest = state_sha256(state) + if digest != item["state_sha256"]: + raise SetError( + f"{suite_id}: item {item['item_id']} (row {index}) hashes to " + f"{digest[:16]} and the manifest says {item['state_sha256'][:16]}. The " + "upstream row has changed, or this is the wrong split file") + spec = manifest_spec(doc, item) + options = list(spec["options"]) + entry = {"id": item["item_id"], "row_idx": index, "state": state, + "state_sha256": digest} + if mixed: + # A mixed set's questions differ per entry, so each one carries its own + # typed question. A single-primitive set says it once at the top instead. + entry.update({"type": spec["type"], "instructions": spec["instructions"], + "criteria": spec.get("criteria"), "options": options}) + # Which ANSWER SPACE this question belongs to, which is the unit the + # Decision Score is defined over: its label prior is the base rates of the + # labels in THIS option list. `action` is a four-option question in one + # workflow and a five-option one in another, so the group has to be in the + # name or two different spaces would be pooled into one meaningless prior. + qkey = str(item["item_id"]).rsplit(":", 1)[-1] + entry["space"] = f"{item.get('group', doc['id'])}/{qkey}" + if isinstance(item.get("gold"), dict) and item["gold"]: + # A gold DISTRIBUTION, not a second label: it is what the answer's spread is + # compared against, and it is never sent on the wire (see wire_leaks). + entry["gold"] = {option: float(item["gold"].get(option, 0.0)) + for option in options} + questions.append(entry) + labels[item["item_id"]] = options[int(item["target"])] + built = { + "id": suite_id, + "suite_version": doc["version"], + "set": suite_id, + "recipe_of_record": doc.get("recipe_of_record") + or RECIPE_OF_RECORD.get(suite_id), + "type": doc["primitive"], + "instructions": doc.get("instructions"), + "criteria": criteria_for(doc), + "options": list(doc["options"]) if doc.get("options") else None, + "seed": doc.get("seed"), + "source": {"dataset": doc["dataset"], "config": doc["config"], + "split": doc["split"], "hf_revision": doc["hf_revision"], + "license": doc["license"], "url": doc.get("source_url", ""), + "state_fields": doc["state_fields"], + "length_cap": doc.get("length_cap"), + "state_hash_source": doc.get("state_hash_source", "jevals")}, + "attribution": doc.get("attribution") or ( + "Jevals (jevals.com), suite " + str(doc["version"]) + + ". Suite files CC-BY-4.0. Item text is not committed; it is rebuilt from " + "the upstream dataset and hash-checked."), + "contamination": CONTAMINATION.get(suite_id, []), + "notes": doc.get("notes", []), + "questions": questions, + "labels": labels, + } + if mixed: + built["instructions"] = (f"per question; {len(doc['question_schemas'])} question " + "schemas in the manifest") + built["options"] = None + return built + + +def download(suite_ids=None, out_dir=None, progress=None) -> list: + """Fetch, verify and write one question file per suite. The only networked call. + + Returns the paths written. Nothing is written for a suite whose states did not all + verify, because half a set on disk is a set somebody runs by accident. + """ + wanted = list(suite_ids or SUITES) + unknown = [s for s in wanted if s not in SUITES] + if unknown: + raise SetError(f"unknown suite(s) {', '.join(unknown)}; pick from " + f"{', '.join(SUITES)}") + directory = pathlib.Path(out_dir) if out_dir else cache_dir() + directory.mkdir(parents=True, exist_ok=True) + written = [] + for suite_id in wanted: + doc = load_manifest(suite_id) + source = SOURCES[suite_id] + if progress: + progress(f"{suite_id}: {doc['dataset']} {doc['config']}/{doc['split']}") + if source["fetch"] == "jsonl": + rows = fetch_jsonl(doc["dataset"], doc["hf_revision"], source["path"]) + else: + up_to = max(int(i["row_idx"]) for i in doc["items"]) + rows = fetch_rows(doc["dataset"], doc["config"], doc["split"], up_to) + built = build_questions(doc, rows) + path = directory / f"{suite_id}.questions.json" + path.write_text(json.dumps(built, ensure_ascii=False, indent=1) + "\n") + written.append(path) + if progress: + progress(f"{suite_id}: {len(built['questions'])} questions verified " + f"against their published hashes -> {path}") + return written + + +# ------------------------------------------------------------------ question files + +#: Keys an answer key could hide behind. A question object carrying one of these would +#: put the answer next to the state on the wire, so the label lives in the file's own +#: separate ``labels`` map instead and a question that names one of these is a load +#: error. :func:`ainode.bench.decide.suite.wire_leaks` holds the other half of the rule. +ANSWER_KEYS = ("label", "labels", "target", "expected", "expected_answer", + "answer", "answer_key", "answerkey", "passinganswer", + "passing_answer", "correct", "correct_answer", "ground_truth", + "groundtruth", "gold", "gold_label", "solution", "truth") + + +def answer_key_leaks(obj, skip=()) -> list: + """Every answer-key-shaped key anywhere in ``obj``, by path. Empty means clean. + + Walks dicts and lists, so a key nested three levels down inside a question block is + found. ``skip`` names top-level keys not to descend into, which is how the state is + left alone: the state is the caller's own whitelisted data and a private set is + allowed a field called whatever its dataset calls it. + """ + found = [] + + def walk(node, path): + if isinstance(node, dict): + for key, value in node.items(): + here = f"{path}.{key}" if path else str(key) + if str(key).lower() in ANSWER_KEYS: + found.append(here) + walk(value, here) + elif isinstance(node, list): + for index, value in enumerate(node): + walk(value, f"{path}[{index}]") + + if isinstance(obj, dict): + for key, value in obj.items(): + if key in skip: + continue + here = str(key) + if str(key).lower() in ANSWER_KEYS: + found.append(here) + walk(value, here) + else: + walk(obj, "") + return found + + +def validate_questions(doc, path=None) -> dict: + """A question file, checked down to what the runner and the metrics assume. + + The same strictness the manifests get, and for the same reason. A question file is + also what a private blind set arrives as, so this is the one gate between "somebody + handed us a file" and "we published a Decision Score from it". + + **The labels live in the file's own ``labels`` map, never inside a question.** A + question object holds an id, a state and nothing that reveals the answer, so the + transport has no label to leak even by accident: it never sees one. A question + carrying an answer-key-shaped field is a load error naming the field. + """ + where = f" in {path}" if path else "" + if not isinstance(doc, dict): + raise SetError(f"the question file{where} is not a JSON object") + for key in ("id", "type", "questions", "labels"): + if not doc.get(key): + raise SetError(f"the question file{where} is missing '{key}'") + if not isinstance(doc["labels"], dict): + raise SetError(f"the question file{where} has a 'labels' that is not a map of " + "question id to label") + from ainode.bench.decide.jevals import TYPES + + mixed = doc["type"] == "mixed" + if not mixed and doc["type"] not in TYPES: + raise SetError(f"the question file{where} has type {doc['type']!r}; pick from " + f"{', '.join(TYPES)} or 'mixed' with a type per question") + if not mixed: + for key in ("instructions", "options"): + if not doc.get(key): + raise SetError(f"the question file{where} is missing '{key}'") + questions = doc["questions"] + if not isinstance(questions, list) or not questions: + raise SetError(f"the question file{where} holds no questions") + seen = set() + for index, question in enumerate(questions): + if not isinstance(question, dict): + raise SetError(f"question {index}{where} is not an object") + qid = question.get("id") + if not qid: + raise SetError(f"question {index}{where} has no id") + if qid in seen: + raise SetError(f"question id {qid} appears twice{where}") + seen.add(qid) + if question.get("state") in (None, ""): + raise SetError(f"question {qid}{where} has no state") + # `gold` is a distribution, not an answer key, and the wire guard skips it the + # way it skips the state; everything else in a question is walked. + leaks = answer_key_leaks(question, skip=("state", "gold", "space")) + if leaks: + raise SetError( + f"question {qid}{where} carries {', '.join(leaks)}, which is an answer " + "key beside the state. Labels belong in the file's own 'labels' map, " + "which the transport never reads") + spec = question_spec(doc, question) + if spec["type"] not in TYPES: + raise SetError(f"question {qid}{where} has type {spec['type']!r}; pick from " + f"{', '.join(TYPES)}") + if not spec.get("instructions"): + raise SetError(f"question {qid}{where} has no instructions") + options = spec["options"] + if not isinstance(options, list) or len(options) < 2: + raise SetError(f"question {qid}{where} declares fewer than two options") + if len(set(options)) != len(options): + raise SetError(f"question {qid}{where} repeats an option") + if doc["labels"].get(qid) not in options: + raise SetError(f"question {qid}{where} is labeled " + f"{doc['labels'].get(qid)!r} in 'labels', which is not one " + "of its options") + gold = question.get("gold") + if gold is not None: + if not isinstance(gold, dict) or not gold: + raise SetError(f"question {qid}{where} has a 'gold' that is not a " + "probability map") + unknown = [k for k in gold if k not in options] + if unknown: + raise SetError(f"question {qid}{where} has gold for " + f"{', '.join(map(str, unknown))}, which are not its " + "options") + return doc + + +def question_spec(doc: dict, question: dict) -> dict: + """The typed question for one entry: its own fields, else the file's. + + One resolution point for both question-file shapes, so the transports, the metrics + and the validator can never disagree about what was asked. A mixed file states the + type, the instructions, the criteria and the options per question; a + single-primitive file states them once at the top and every question inherits them. + """ + return { + "type": question.get("type") or doc.get("type"), + "instructions": question.get("instructions") or doc.get("instructions"), + "criteria": (question["criteria"] if "criteria" in question + else doc.get("criteria")), + "options": list(question.get("options") or doc.get("options") or []), + } + + +def load_questions(path) -> dict: + """Read and validate a question file. Hash-checks whatever carries a hash. + + A file written by ``download`` carries a ``state_sha256`` per question, so loading + it re-verifies every state rather than trusting the disk; a hand-authored private + set carries none and is loaded as it is. + """ + path = pathlib.Path(path) + if not path.is_file(): + raise SetError(f"no question file at {path}") + try: + doc = json.loads(path.read_text()) + except json.JSONDecodeError as exc: + raise SetError(f"{path} is not valid JSON: {exc}") from exc + validate_questions(doc, path) + for question in doc["questions"]: + expected = question.get("state_sha256") + if expected and state_sha256(question["state"]) != expected: + raise SetError(f"{path}: question {question['id']} does not match its own " + "state hash; delete the cache and download it again") + doc.setdefault("set", doc["id"]) + return doc + + +def load_suite_questions(suite_ids=None) -> list: + """The downloaded question files for those suites, or an error saying how to get them.""" + wanted = list(suite_ids or SUITES) + unknown = [s for s in wanted if s not in SUITES] + if unknown: + raise SetError(f"unknown suite(s) {', '.join(unknown)}; pick from " + f"{', '.join(SUITES)}") + out = [] + for suite_id in wanted: + path = questions_path(suite_id) + if not path.is_file(): + raise SetError( + f"{suite_id} has not been downloaded: no {path}. Run " + f"`python3 scripts/ainode-bench.py decide download {suite_id}` first. " + "The item text is deliberately not committed (see " + "bench/decide/JEVALS.md)") + out.append(load_questions(path)) + return out + + +def manifest_summary(suite_id: str) -> dict: + """The one-line description of a suite, for a dry run and for a record's settings.""" + doc = load_manifest(suite_id) + schemas = doc.get("question_schemas") or {} + widths = [len(spec["options"]) for block in schemas.values() + for spec in block.values()] or [len(doc.get("options") or [])] + return {"id": doc["id"], "suite_version": doc["version"], + "type": doc["primitive"], "title": doc.get("title", doc["id"]), + "items": int(doc["n_items"]), + "options": max(widths) if widths else 0, + "dataset": doc["dataset"], + "split": f"{doc['config']}/{doc['split']}", + "hf_revision": doc["hf_revision"], "license": doc["license"], + "seed": doc.get("seed"), + "recipe_of_record": doc.get("recipe_of_record") + or RECIPE_OF_RECORD.get(suite_id), + "contaminated_for": [c["system"] for c in CONTAMINATION.get(suite_id, [])], + "downloaded": questions_path(suite_id).is_file()} + + +__all__ = ["ANSWER_KEYS", "CONTAMINATION", "ENV_CACHE", "ENV_SETS", "HTTP_RETRIES", + "HTTP_TIMEOUT", "RECIPE_OF_RECORD", "RESOLVE", "ROWS_API", "ROWS_PAGE", + "SOURCES", "SUITES", "SetError", "answer_key_leaks", "build_questions", + "cache_dir", "criteria_for", "download", "fetch_jsonl", "fetch_rows", + "load_manifest", "load_questions", "load_suite_questions", "manifest_path", + "manifest_spec", "manifest_summary", "question_spec", "questions_path", + "sets_dir", "state_json", "state_sha256", "validate_questions"] diff --git a/ainode/bench/decide/suite.py b/ainode/bench/decide/suite.py new file mode 100644 index 00000000..a83ae6f8 --- /dev/null +++ b/ainode/bench/decide/suite.py @@ -0,0 +1,1051 @@ +"""The Jevals-recipe run: two transports, five repeats per question, one record. + +This is the half of the decision bench that scores an AINode-served model on the same +public question sets the independent Jevals boards use, so a number here can be read +next to Jev and its clones. The recipe itself is recorded in ``bench/decide/JEVALS.md`` +with the URL and the date it was read, the formulas are +:mod:`ainode.bench.decide.jevals`, and the question sets are +:mod:`ainode.bench.decide.sets`. Nothing here restates a formula or a licence: this +module is the loop, the two wire shapes and the record. + +**Two transports, one flag.** ``--transport decide`` posts AINode's own +``POST /v1/decide``; ``--transport systemone`` posts ``POST /v1/systemone`` in the Jev +wire format, which is what TypeSafe's hosted Jev speaks and what a local server that +implements the same interface speaks. The second one is why a Kev, a laya.cpp or a +TypeSafe endpoint is a flag and not a fork. + +**Which key goes where is decided by the HOST, not by a flag.** A request to +``api.typesafe.ai`` resolves TypeSafe's credential and a request to anything else +resolves the node's, so no ordering of flags can post a fleet key to a vendor or a +vendor's key to one of our nodes. That is the same invariant the legacy backends hold, +made structural instead of name-based. + +``request()`` and ``parse()`` are pure functions of their arguments on both transports, +so ``tests/test_bench_decide_jevals.py`` pins every request shape and every response +shape from canned payloads with no server. Only ``ask()`` touches the network. + +Stdlib only, like the rest of ``ainode/bench``. +""" +from __future__ import annotations + +import concurrent.futures as futures +import random +import time +import urllib.parse + +from ainode.bench import auth +from ainode.bench.decide import jevals, sets +from ainode.bench.decide.backends import ( + DEFAULT_API_KEY, + DEFAULT_TIMEOUT, + ERROR_CHARS, + BackendError, + Request, + jev_api_key, + node_api_key, + post_json, +) + +SOURCE = "scripts/ainode-bench.py decide" +#: The record's ``decide.mode``, so a reader can tell at a glance which of the two +#: measurements in this package produced the block. +MODE = "jevals-0.1.0" +RECIPE = {"source": "https://jevals.com/methodology", "read": "2026-09-21", + "suite": "0.1.0", "doc": "bench/decide/JEVALS.md", + "attribution": "Jevals (jevals.com), suite 0.1.0"} + +#: The two transports ``--transport`` picks from. +TRANSPORTS = ("decide", "systemone") +#: The key every request names its one question under. One question per request is the +#: recipe's batch size, and the name comes back under itself in both response shapes. +QUESTION_KEY = "decision" +#: The hosted Jev endpoint, and the host that decides a request gets TypeSafe's key. +TYPESAFE_HOST = "api.typesafe.ai" +JEV_URL = "https://api.typesafe.ai/v1/systemone" + +DEFAULT_CONCURRENCY = 4 +#: How many probabilities a row keeps. A full 77-option vector on 1,500 decisions is +#: most of a record, so a row keeps the five it was surest about plus the labeled +#: option's own probability, which is what a reader of a wrong answer needs. +ROW_TOP_K = 5 + + +# ------------------------------------------------------------------ option order + +def order_index(repeat: int) -> int: + """Which presented order a repeat uses: 0 and 1 share one, then one each. + + The recipe's shape, so repeats 0 and 1 are byte-identical requests (which is what + makes the repeat flip rate a measurement of nondeterminism) and repeats 2, 3 and 4 + are the three further orders the order flip rate compares. + """ + return 0 if int(repeat) < 2 else int(repeat) - 1 + + +def presented_options(options, question_id: str, seed, order: int, qtype: str) -> list: + """The options in the order this repeat shows them. + + ``choice`` is shuffled, seeded on the suite seed, the question id and the order + index, so the permutation is deterministic and identical for every system this bench + runs. ``noul`` and ``score`` are never reordered, which is the recipe's rule and + necessary for ``score``, whose levels are ordinal. + + The permutation is NOT Jevals' own: they publish the properties but not the + generator (see JEVALS.md, "What we had to author"). Ours satisfies every published + property except being the same permutation, so an order flip rate from this bench is + a real order flip rate and is not their number. + """ + if qtype != jevals.CHOICE: + return list(options) + shuffled = list(options) + random.Random(f"{seed}:{question_id}:{int(order)}").shuffle(shuffled) + return shuffled + + +# ------------------------------------------------------------------ criteria + +def option_criteria(spec: dict) -> dict: + """``{option: description or None}`` for any of the three primitives. + + One place turns the three shapes a question states its criteria in into the one shape + a prompt needs: a ``choice`` map keyed by option name, a ``score`` list in level + order, and a ``noul`` map keyed ``true``/``false`` whose ``false`` belongs to the + first option and whose ``true`` belongs to the second. + """ + options = list(spec["options"]) + criteria = spec.get("criteria") + if criteria is None: + return {option: None for option in options} + qtype = spec["type"] + if qtype == jevals.SCORE and isinstance(criteria, list): + return {option: (criteria[index] if index < len(criteria) else None) + for index, option in enumerate(options)} + if qtype == jevals.NOUL and isinstance(criteria, dict) and len(options) == 2: + # Accept the option names themselves as keys too: a set whose noul options are + # already spelled `no`/`yes` states its criteria under those. + if set(criteria) & {"true", "false"}: + return {options[0]: criteria.get("false"), options[1]: criteria.get("true")} + return {option: criteria.get(option) for option in options} + if isinstance(criteria, dict): + return {option: criteria.get(option) for option in options} + return {option: None for option in options} + + +def criteria_lines(spec: dict, presented) -> str: + """``name: description`` per option, in presented order, or "" when none is described. + + Empty when every description is null, which is Banking77: its 77 options carry names + and nothing else, and appending 77 bare names under a heading would add words to the + prompt that the board's prompt does not have. + """ + described = option_criteria(spec) + if not any(described.get(option) for option in presented): + return "" + lines = [] + for option in presented: + text = described.get(option) + lines.append(f"{option}: {text}" if text else str(option)) + return "\n".join(lines) + + +# ------------------------------------------------------------------ the decision row + +class Answer: + """What one request came back as, before the recipe scores it. + + ``vector`` is the raw probability map as the endpoint stated it; normalization, + malformed detection and the pick all happen in :func:`decision_for`, once, against + the recipe's rules, so the two transports cannot drift on any of them. + """ + + def __init__(self, vector=None, stated=None, confidence=None, tokens_in=None, + tokens_out=None, model="", node="", error=None, excerpt="", + one_hot=False, malformed_reason=None): + self.vector = vector + self.stated = stated + self.confidence = confidence + self.tokens_in = tokens_in + self.tokens_out = tokens_out + self.model = model + self.node = node + self.error = error + self.excerpt = excerpt + self.one_hot = one_hot + self.malformed_reason = malformed_reason + self.wall_ms = 0 + + +def spec_for(doc: dict, question: dict) -> dict: + """The typed question for one entry. One resolution point, in :mod:`sets`.""" + return sets.question_spec(doc, question) + + +def decision_for(doc: dict, question: dict, repeat: int, presented, answer: Answer + ) -> dict: + """One decision dict, the unit :mod:`ainode.bench.decide.jevals` scores. + + The three cases the recipe distinguishes, in one place: + + * a transport failure carries the ``error`` and is never scored; + * an answer whose probabilities cannot be read is ``malformed``, scored as the + uniform distribution and a wrong pick; + * an answer with no probabilities at all is ``one_hot``: it keeps its pick, is + scored as one-hot, and is out of the calibration numbers. + """ + spec = spec_for(doc, question) + options = list(spec["options"]) + qtype = spec["type"] + decision = { + "id": question["id"], "set": doc.get("set") or doc["id"], "type": qtype, + "repeat": int(repeat), "order": order_index(repeat), + "options": options, "label": (doc.get("labels") or {}).get(question["id"]), + "space": question.get("space"), + "gold": question.get("gold"), + "vector": None, "pick": None, "confidence": None, + "malformed": False, "one_hot": False, + # What the endpoint's own probabilities summed to BEFORE renormalizing, which is + # what JevBench's two schema-validity rates are computed over: its headline + # renormalizes inside a 2 percent band and its strict column uses 0.001. + "sum_before_normalize": None, + "wall_ms": answer.wall_ms, "tokens_in": answer.tokens_in, + "tokens_out": answer.tokens_out, "error": answer.error, + "excerpt": answer.excerpt, + } + if answer.error: + return decision + if answer.one_hot: + pick = answer.stated if answer.stated in options else None + if pick is None: + decision["malformed"] = True + decision["malformed_reason"] = (answer.malformed_reason + or "no probabilities and no known answer") + decision["vector"] = jevals.uniform(options) + return decision + decision["one_hot"] = True + decision["vector"] = {option: (1.0 if option == pick else 0.0) + for option in options} + decision["pick"] = pick + decision["confidence"] = 1.0 + return decision + if isinstance(answer.vector, dict): + stated = [value for value in answer.vector.values() + if isinstance(value, (int, float)) and not isinstance(value, bool)] + if stated: + decision["sum_before_normalize"] = round(float(sum(stated)), 6) + vector, why = jevals.normalize_vector(answer.vector, options) + if vector is None: + decision["malformed"] = True + decision["malformed_reason"] = answer.malformed_reason or why + decision["vector"] = jevals.uniform(options) + return decision + pick = jevals.pick_from_vector(vector, options, presented=presented, qtype=qtype, + stated=answer.stated) + decision["vector"] = vector + decision["pick"] = pick + decision["confidence"] = None if pick is None else vector.get(pick) + if qtype == jevals.NOUL and pick is None: + # Exactly 0.5 either way. A real answer with no pick, which counts as wrong and + # has no confidence to calibrate: not malformed, and the recipe says so. + decision["confidence"] = None + return decision + + +def row_for(decision: dict, top_k: int = ROW_TOP_K) -> dict: + """One record row. Carries the numbers, not the state and not the full vector. + + The state is rebuilt from the suite manifest under the same ``id`` and is verified + against its published hash, so a record that repeated it would be mostly prompt. The + full vector is dropped for the same reason on a wide set: ``top`` keeps the five + options the system was surest about, and ``p_label`` keeps the labeled option's own + probability, which together are what a reader of a wrong answer needs. + """ + vector = decision.get("vector") or {} + label = decision.get("label") + top = sorted(vector.items(), key=lambda pair: (-pair[1], pair[0]))[:top_k] + return { + "id": decision["id"], "set": decision["set"], "kind": decision["type"], + "repeat": decision["repeat"], "order": decision["order"], + "label": label, "answer": decision.get("pick"), + "correct": (None if decision.get("error") else + bool(decision.get("pick") == label)), + "p_answer": (None if decision.get("confidence") is None + else round(float(decision["confidence"]), 6)), + "p_label": (None if not vector else round(float(vector.get(label, 0.0)), 6)), + "top": {name: round(float(value), 6) for name, value in top} or None, + "malformed": bool(decision.get("malformed")), + "malformed_reason": decision.get("malformed_reason"), + "one_hot": bool(decision.get("one_hot")), + "wall_ms": decision.get("wall_ms"), + "tokens_in": decision.get("tokens_in"), + "tokens_out": decision.get("tokens_out"), + "error": decision.get("error"), + } + + +# ------------------------------------------------------------------ the transports + +def systemone_url(endpoint: str) -> str: + """``/systemone``, adding the ``/v1`` if it was left off.""" + base = (endpoint or "").rstrip("/") + if base.endswith("/systemone"): + return base + if not base.endswith("/v1"): + base += "/v1" + return base + "/systemone" + + +def decide_endpoint(endpoint: str) -> str: + """``/decide``, adding the ``/v1`` if it was left off.""" + base = (endpoint or "").rstrip("/") + if base.endswith("/decide"): + return base + if not base.endswith("/v1"): + base += "/v1" + return base + "/decide" + + +def is_typesafe(url: str) -> bool: + """Whether a URL points at the hosted Jev, which decides the credential.""" + host = (urllib.parse.urlparse(url).hostname or "").lower() + return host == TYPESAFE_HOST or host.endswith("." + TYPESAFE_HOST) + + +def wire_leaks(payload) -> list: + """Every answer-key-shaped key in a request body, by path. Empty means clean. + + The other half of the rule :func:`ainode.bench.decide.sets.answer_key_leaks` holds + over a question file: a label lives in the file's separate ``labels`` map, a + transport is handed a question that has none, and this checks the body it actually + assembled. Two ends, because "the model must not be shown the answer" is the one + mistake in a benchmark that makes every number it produces worthless while looking + like a very good result. + + The ``state`` value is not walked. It is the caller's own whitelisted data, a private + set is allowed a field named whatever its dataset names it, and this guard is about + what the bench appends beside the question rather than what the question is about. + """ + return sets.answer_key_leaks(payload, skip=("state",)) + + +class Transport: + """Build a request, parse a response, and only ``ask`` touches the network.""" + + name = "" + #: True when the endpoint is one of OUR nodes, so a 401 or a 429 from it is + #: AINode's and stops the run before anything is scored. False for a vendor's, + #: whose refusals belong in their own row and not in an AINode sentence. + local = True + #: Where the probabilities came from, in JevBench's vocabulary plus the one value it + #: has no name for. ``native`` is the model's own distribution, which is what a + #: Jev-format server returns; ``verbalized`` is a model writing probabilities out + #: under a schema; ``logprob`` is reading them off the decode, which JevBench states + #: it does not do for anyone. A row is only comparable to a row with the same value + #: here, so the record carries it rather than a footnote. + probability_source = "native" + + def __init__(self, endpoint: str, model: str = "", api_key: str = "", + key_source: str = "", timeout: float = DEFAULT_TIMEOUT, + input_usd_per_mtok: float = 0.0, output_usd_per_mtok: float = 0.0): + if not endpoint: + raise BackendError(f"the {self.name} transport needs --endpoint") + self.endpoint = endpoint + self.model = model + self.api_key = api_key + self.key_source = key_source + self.timeout = timeout + self.input_usd_per_mtok = float(input_usd_per_mtok) + self.output_usd_per_mtok = float(output_usd_per_mtok) + self.requests = 0 + self.reported_model = "" + self.reported_node = "" + + # -- pure halves ------------------------------------------------------------ + def question(self, doc: dict, presented) -> dict: + raise NotImplementedError + + def request(self, doc: dict, question: dict, presented) -> Request: + raise NotImplementedError + + def parse(self, doc: dict, question: dict, presented, data: dict) -> Answer: + raise NotImplementedError + + # -- the one networked call ------------------------------------------------- + def ask(self, doc: dict, question: dict, repeat: int) -> dict: + """One (question, repeat) end to end, as a decision dict. + + A refusal from one of our own nodes RAISES rather than becoming 4,500 wrong + answers with a Decision Score computed over them, which is the rule the rest of + the bench runs under. A vendor's refusal is left alone. + """ + spec = spec_for(doc, question) + presented = presented_options(spec["options"], question["id"], doc.get("seed"), + order_index(repeat), spec["type"]) + request = self.request(doc, question, presented) + leaks = wire_leaks(request.payload) + if leaks: + raise BackendError( + f"the {self.name} request for {question['id']} carries " + f"{', '.join(leaks)}: that is the answer on the wire. Refusing to ask, " + "because a score measured against a prompt holding its own label is not " + "a score") + self.requests += 1 + data, wall, error = post_json(request, self.timeout) + answer = Answer() + answer.wall_ms = round(wall * 1000) + if error: + if self.local: + auth.check_error(error) + answer.error = error + else: + try: + answer = self.parse(doc, question, presented, data) + except Exception as exc: # noqa: BLE001 + answer = Answer(error=None, one_hot=False) + answer.vector = None + answer.malformed_reason = (f"unreadable response: " + f"{type(exc).__name__}: " + f"{str(exc)[:ERROR_CHARS]}") + answer.wall_ms = round(wall * 1000) + if answer.model: + self.reported_model = answer.model + if answer.node: + self.reported_node = answer.node + return decision_for(doc, question, repeat, presented, answer) + + def protocol(self) -> dict: + return {"transport": self.name, "endpoint": self.endpoint, + "timeout_s": self.timeout, + "probability_source": self.probability_source, + "cost_basis": self.cost_basis(), + "input_usd_per_mtok": self.input_usd_per_mtok, + "output_usd_per_mtok": self.output_usd_per_mtok} + + def cost_basis(self) -> str: + """Why the cost column says what it says, in JevBench's vocabulary. + + A priced endpoint is ``derived_usage_times_tariff``: measured tokens times a rate + somebody posted. An unpriced one reads $0, and this says WHY it is 0 rather than + letting a reader take it for free: nobody bills per token for a GPU we own, the + electricity is real, and an invented figure would be an estimate in a file of + measurements. + """ + if self.input_usd_per_mtok or self.output_usd_per_mtok: + return "derived_usage_times_tariff" + return "no_billable_account_no_price_given" + + +class DecideTransport(Transport): + """AINode's ``POST /v1/decide``. + + Every primitive goes out with an EXPLICIT ``options`` list rather than the + endpoint's ``boolean`` or ``score`` sugar, so the distribution comes back keyed by + this set's own option strings for all three and one parser fits them. The sugar + would be the same request (``type: boolean`` is ``["yes", "no"]`` and ``type: + score`` is ``[str(v) for v in range(min, max + 1)]``) with a different chance of a + key mismatch: PubMedQA's options are ``["no", "yes"]``, and the order matters to the + presented prompt. + + The state goes out as the CANONICAL JSON STRING rather than as an object, because + the endpoint serializes an object it is handed with sorted keys and the bytes the + recipe hashes are in state-field order. A string is passed through verbatim, so the + model sees exactly the bytes the published ``state_sha256`` covers. + + The criteria are appended to the question text because ``/v1/decide``'s choice + question takes bare option names with no room for a per-option rubric. For Banking77 + that appends nothing, since its criteria are all null. + """ + + name = "decide" + #: ``/v1/decide`` softmaxes the first generated token's logprobs, which is neither of + #: JevBench's two categories: it states it uses token-level logprobs for nobody, and + #: Jevals 0.1.0 states that logprob-based rows are not in that version. + probability_source = "logprob" + + def question(self, spec: dict, presented) -> dict: + text = spec["instructions"] + lines = criteria_lines(spec, presented) + if lines: + text = f"{text}\n{lines}" + return {"question": text, "options": list(presented)} + + def request(self, doc: dict, question: dict, presented) -> Request: + payload = {"state": sets.state_json(question["state"]), + "questions": {QUESTION_KEY: self.question( + spec_for(doc, question), presented)}} + if self.model: + payload["model"] = self.model + return Request(self.endpoint, payload, auth.bearer(self.api_key)) + + def parse(self, doc: dict, question: dict, presented, data: dict) -> Answer: + block = (data.get("decisions") or {})[QUESTION_KEY] + usage = data.get("usage") or {} + answer = Answer( + vector=block.get("distribution"), + stated=block.get("answer"), + confidence=block.get("confidence"), + tokens_in=usage.get("prompt_tokens"), + tokens_out=usage.get("completion_tokens"), + model=data.get("model") or self.model, + node=data.get("node") or "", + excerpt=str(block.get("answer"))[:ERROR_CHARS]) + if answer.vector is None: + # The engine answered under the grammar but gave no logprobs, so there is a + # pick and no spread. Scored as one-hot, which is the recipe's rule, and out + # of the calibration numbers rather than credited with a confidence of 1. + answer.one_hot = True + return answer + + +class SystemOneTransport(Transport): + """``POST /v1/systemone`` in the Jev wire format: TypeSafe's, or any server's. + + One typed question per request, the state as a JSON object (which is how Jevals + sends it), and the criteria in the shape the question file states them: a map for + ``choice``, a ``true``/``false`` map for ``noul``, a list in level order for + ``score``. Choice options are presented in this repeat's order, which is what the + order flip rate measures. + + The ``score`` response shape is the one authored piece here: jevals.com documents + the primitive but not the field a Jev-format server answers it in, so the parser + accepts ``score``, ``level`` or ``choice`` for the pick and ``probabilities`` or + ``distribution`` for the vector, and a response matching none of them is one + malformed row rather than a crash. Recorded in JEVALS.md. + """ + + name = "systemone" + #: The model's own distribution, off the wire, which is what JevBench's ``typesafe`` + #: adapter reads from this same endpoint shape. + probability_source = "native" + + def question(self, spec: dict, presented) -> dict: + qtype = spec["type"] + options = list(spec["options"]) + out = {"type": qtype, "instructions": spec["instructions"]} + described = option_criteria(spec) + if qtype == jevals.SCORE: + out["criteria"] = [described.get(option) for option in options] + elif qtype == jevals.NOUL: + out["criteria"] = {"false": described.get(options[0]), + "true": described.get(options[1])} + else: + out["criteria"] = {option: described.get(option) for option in presented} + if qtype == jevals.NOUL and not any(out["criteria"].values()): + # Two of typed-decisions' twenty questions carry no criteria at all, and a + # yes/no question needs none: its answer space is implied by the type. Only + # this case may drop the field. For `choice` the criteria MAP KEYS ARE the + # answer space, and for `score` its length is K, so dropping either would + # ask a different question (Banking77's 77 criteria are all null and still + # have to travel, because they are the option list). + out.pop("criteria") + return out + + def request(self, doc: dict, question: dict, presented) -> Request: + payload = {"state": question["state"], + "questions": {QUESTION_KEY: self.question( + spec_for(doc, question), presented)}} + if self.model: + payload["model"] = self.model + return Request(self.endpoint, payload, auth.bearer(self.api_key)) + + def parse(self, doc: dict, question: dict, presented, data: dict) -> Answer: + block = (data.get("answers") or {})[QUESTION_KEY] + usage = data.get("usage") or {} + spec = spec_for(doc, question) + options = list(spec["options"]) + vector = None + stated = None + if spec["type"] == jevals.NOUL and "noul" in block: + probability = float(block["noul"]) + # P(yes) on the wire. The second option is the positive one, which is how + # the manifests are built (PubMedQA: ["no", "yes"]). + vector = {options[0]: 1.0 - probability, options[1]: probability} + else: + raw = block.get("probabilities") + if raw is None: + raw = block.get("distribution") + if isinstance(raw, dict): + vector = {str(key): value for key, value in raw.items()} + for key in ("choice", "score", "level"): + if block.get(key) is not None: + stated = str(block[key]) + break + answer = Answer( + vector=vector, stated=stated, confidence=block.get("confidence"), + tokens_in=usage.get("input_tokens", usage.get("prompt_tokens")), + tokens_out=usage.get("output_tokens", usage.get("completion_tokens")), + model=data.get("model") or self.model, + node=data.get("node") or "", + excerpt=str(stated)[:ERROR_CHARS]) + if vector is None and stated is not None: + answer.one_hot = True + elif vector is None: + answer.malformed_reason = "no probability map and no stated answer" + return answer + + +def build_transport(name: str, endpoint: str = "", model: str = "", api_key: str = "", + timeout: float = DEFAULT_TIMEOUT, input_usd_per_mtok: float = 0.0, + output_usd_per_mtok: float = 0.0) -> Transport: + """The named transport with its key resolved from the endpoint's HOST. + + ``api.typesafe.ai`` gets TypeSafe's credential (``--api-key``, then + ``$TYPESAFE_API_KEY``, then ``~/.jev_api_key``); anything else gets the node's + (``--api-key``, then ``$AINODE_API_KEY``, then the placeholder an open node + accepts). Two separate resolvers, picked by the host, so no flag ordering can send + one party's credential to the other. + """ + if name not in TRANSPORTS: + raise BackendError(f"unknown transport {name!r}; pick from " + f"{', '.join(TRANSPORTS)}") + if name == "systemone": + url = systemone_url(endpoint or JEV_URL) + if is_typesafe(url): + key, source = jev_api_key(api_key) + if not key: + from ainode.bench.decide.backends import missing_key_message + raise BackendError(missing_key_message()) + transport = SystemOneTransport( + url, model=model, api_key=key, key_source=source, timeout=timeout, + input_usd_per_mtok=input_usd_per_mtok, + output_usd_per_mtok=output_usd_per_mtok) + transport.local = False + return transport + key, source = node_api_key(api_key) + return SystemOneTransport(url, model=model, api_key=key, key_source=source, + timeout=timeout, + input_usd_per_mtok=input_usd_per_mtok, + output_usd_per_mtok=output_usd_per_mtok) + if not endpoint: + raise BackendError("the decide transport needs --endpoint, e.g. " + "http://host:3000/v1") + key, source = node_api_key(api_key) + return DecideTransport(decide_endpoint(endpoint), model=model, api_key=key, + key_source=source, timeout=timeout, + input_usd_per_mtok=input_usd_per_mtok, + output_usd_per_mtok=output_usd_per_mtok) + + +# ------------------------------------------------------------------ the loop + +def plan(docs, repeats: int, limit: int = 0) -> list: + """``[(doc, question, repeat), ...]`` in set, question, repeat order. + + Deterministic order so two runs' rows line up one to one and a record can be read + down the file. ``limit`` takes the first N questions of each set, which is how a + smoke run proves a transport without spending a full suite on it. + """ + out = [] + for doc in docs: + questions = doc["questions"] + if limit: + questions = questions[:max(1, int(limit))] + for question in questions: + for repeat in range(max(1, int(repeats))): + out.append((doc, question, repeat)) + return out + + +def run(transport: Transport, docs, repeats: int = jevals.REPEATS, + concurrency: int = DEFAULT_CONCURRENCY, limit: int = 0, progress=None): + """Every (question, repeat) through the transport. ``(decisions, seconds)``. + + Ordered by the plan and not by completion. The wall clock covers the whole loop, + which is what ``questions_per_second`` is over, and it moves with ``concurrency`` + and with whatever else the node is serving, which is why the record carries both. + """ + work = plan(docs, repeats, limit) + decisions = [None] * len(work) + workers = max(1, int(concurrency)) + started = time.time() + with futures.ThreadPoolExecutor(max_workers=workers) as pool: + pending = {pool.submit(transport.ask, doc, question, repeat): index + for index, (doc, question, repeat) in enumerate(work)} + done = 0 + for future in futures.as_completed(pending): + index = pending[future] + decisions[index] = future.result() + done += 1 + if progress: + progress(done, len(work), decisions[index]) + return decisions, time.time() - started + + +# ------------------------------------------------------------------ the record + +def reported_model(transport: Transport) -> str: + """The model id to put in the record: what the service said it was. + + For a hosted Jev that is the version string the API reports rather than the alias + that was asked for, so the record names the thing that answered. + """ + return (getattr(transport, "reported_model", "") + or getattr(transport, "model", "") or transport.name) + + +def legacy_overall(decisions, input_usd_per_mtok: float = 0.0, + output_usd_per_mtok: float = 0.0) -> dict: + """The fields of the legacy ``overall`` block that mean the SAME thing here. + + Deliberately partial. ``brier``, ``ece``, ``bins`` and ``thresholds`` are absent + rather than filled in from the recipe's numbers: the legacy block's Brier is one + term on the labeled option over five bins and the recipe's is the multiclass sum + over ten, so putting one under the other's name would make two incomparable numbers + look like one. The README table reads "not measured" for those cells on a run like + this, which is the truth. + """ + rows = jevals.scored(decisions) + counts = jevals.tokens(rows) + block = {"n": len(decisions), "answered": len(rows), + "errors": len(jevals.failed(decisions)), + "accuracy": None if not rows else round(jevals.accuracy(decisions), 4), + "tokens": counts, + "cost_usd": round(jevals.cost_usd(counts, input_usd_per_mtok, + output_usd_per_mtok), 6)} + block.update(jevals.latency(decisions)) + return block + + +def build_jevals_block(transport: Transport, docs, decisions, seconds: float, + repeats: int, concurrency: int, limit: int = 0) -> dict: + """The ``decide.jevals`` sub-block: one metrics block per set, plus the mean.""" + kw = {"input_usd_per_mtok": transport.input_usd_per_mtok, + "output_usd_per_mtok": transport.output_usd_per_mtok, "seconds": seconds} + names = [doc.get("set") or doc["id"] for doc in docs] + per_set = jevals.summarize_sets(decisions, names, **kw) + block = { + "recipe": dict(RECIPE), + "formulas": jevals.RECIPE_JEVALS, + "probability_source": transport.probability_source, + "cost_basis": transport.cost_basis(), + "repeats": int(repeats), + "concurrency": int(concurrency), + "batch_size": 1, + "bins": jevals.BINS, + "handoff_target": jevals.HANDOFF_ACCURACY, + "handoff_min_decisions": jevals.HANDOFF_MIN_DECISIONS, + "grid": jevals.GRID, + "published_gates": {name: jevals.PUBLISHED_GATES.get(doc.get("type")) + for name, doc in zip(names, docs)}, + "recipe_of_record": {name: doc.get("recipe_of_record") + for name, doc in zip(names, docs)}, + "contamination": {name: (doc.get("contamination") or []) + for name, doc in zip(names, docs) + if doc.get("contamination")}, + "sets": per_set, + "overall": jevals.mean_decision_score(per_set), + "seconds": round(float(seconds), 1), + } + if limit: + block["limit"] = int(limit) + block["partial"] = True + return block + + +def set_summary(doc: dict, limit: int = 0) -> dict: + """What a record says about one set: which items, from where, under which licence.""" + count = len(doc["questions"]) + if limit: + count = min(count, max(1, int(limit))) + widths = [len(sets.question_spec(doc, q)["options"]) for q in doc["questions"]] + return {"id": doc["id"], "type": doc.get("type"), "questions": count, + "options": max(widths) if widths else 0, "seed": doc.get("seed"), + "suite_version": doc.get("suite_version"), + "recipe_of_record": doc.get("recipe_of_record"), + "gold_distributions": any(q.get("gold") for q in doc["questions"]), + "contamination": doc.get("contamination") or [], + "source": doc.get("source") or {}} + + +def build_decide_block(transport: Transport, docs, decisions, seconds: float, + repeats: int, concurrency: int, limit: int = 0, + model_reported: str = "") -> dict: + """The record's ``decide`` block for a Jevals-recipe run. See bench/SCHEMA.md.""" + return { + "backend": transport.name, + "endpoint": transport.endpoint, + "model_reported": model_reported or None, + "mode": MODE, + "recipe": dict(RECIPE), + "item_set": {"id": f"jevals-{RECIPE['suite']}", + "version": RECIPE["suite"], + "file": "bench/decide/sets/", + "count": sum(set_summary(d, limit)["questions"] for d in docs), + "sets": {(d.get("set") or d["id"]): + set_summary(d, limit)["questions"] for d in docs}}, + "protocol": {**transport.protocol(), "concurrency": int(concurrency), + "repeats": int(repeats), + "batch_size": 1, + "confidence": "the probability the endpoint put on its own pick; " + "for a yes/no question the larger of P(yes) and " + "P(no)", + "loss": "multiclass Brier for choice and noul, ranked probability " + "score over cumulative levels for score"}, + "sources": [set_summary(doc, limit) for doc in docs], + "overall": legacy_overall(decisions, transport.input_usd_per_mtok, + transport.output_usd_per_mtok), + "sets": {}, + "jevals": build_jevals_block(transport, docs, decisions, seconds, repeats, + concurrency, limit), + "rows": [row_for(decision) for decision in decisions], + } + + +def build_notes(transport: Transport, docs, decisions, seconds: float, repeats: int, + limit: int = 0, source: str = SOURCE) -> list: + """The notes a reader needs to know what these numbers are and are not.""" + names = ", ".join((doc.get("set") or doc["id"]) for doc in docs) + errors = [d["id"] for d in jevals.failed(decisions)] + malformed = [d["id"] for d in jevals.scored(decisions) if d.get("malformed")] + one_hot = sum(1 for d in jevals.scored(decisions) if d.get("one_hot")) + notes = [ + f"Measured by {source} in {round(seconds)}s: {len(decisions)} decisions, " + f"{repeats} repeats per question, over {names}. Nothing was loaded, unloaded " + "or restarted.", + "Scored by the Jevals recipe, suite 0.1.0, read from " + "https://jevals.com/methodology on 2026-09-21 and recorded in " + "bench/decide/JEVALS.md with every deviation named. Attribution: Jevals " + "(jevals.com), suite 0.1.0; suite files CC-BY-4.0.", + "Decision Score is 100 * (1 - L_system / L_prior): 100 is perfect, 0 is " + "answering with the label base rates, and below 0 is worse than that. Every " + "set's block carries prior_accuracy, the base-rate answer's own accuracy, so " + "the guessing floor travels with the figure.", + "Every state was rebuilt from its upstream dataset row and checked against the " + "manifest's state_sha256 before the run, so these are the same bytes the boards " + "scored. The item text is not committed; only the manifests are.", + "One question per request (batch size 1), which is the recipe's rule. A set card " + "that measured its own reference row several questions per request is reporting a " + "different latency and a different cost, and those two columns are not " + "comparable to this record's.", + "Every metrics block names the recipe its formulas follow (`recipe`), and every " + "set names the recipe its own published third-party numbers follow " + "(`recipe_of_record`). Where those two differ, only the figures under this " + "record's own recipe are comparable across rows.", + ] + contaminated = [(doc.get("set") or doc["id"], doc.get("contamination") or []) + for doc in docs] + for name, entries in contaminated: + if not entries: + continue + who = "; ".join(f"{e['system']} ({e['source']})" for e in entries) + notes.append( + f"CONTAMINATION, {name}: this set is in the published training data of " + f"{who}. A row for one of those systems on this set measures memorisation " + "and not decision quality, and it must not be read as a like-for-like " + "comparison with a system that never saw it. It says nothing about an " + "AINode-served model that did not train on it.") + if any(doc.get("type") == "mixed" for doc in docs): + notes.append( + "A set holding more than one primitive is broken down by primitive, because " + "the two losses are different arithmetic and the label prior is per answer " + "space. Its Decision Score is the plain mean of the per-primitive scores, " + "which is the recipe's rule for more than one task in a tab.") + if any(any(q.get('gold') for q in doc['questions']) for doc in docs): + notes.append( + "One of these sets ships a gold DISTRIBUTION rather than only a label, so " + "its block carries a vs_gold section: soft accuracy, total variation, KL " + "and a Brier against that distribution. Those four are AINode's own " + "definitions, stated in the block, and are not a set card's columns of the " + "same names.") + notes.append( + f"Probability source: {transport.probability_source}. JevBench labels a model's " + "own distribution `native` and a model writing probabilities out under a schema " + "`verbalized`, and states it uses token-level logprobs for nobody; Jevals 0.1.0 " + "states its LLM rows are verbalized and that logprob-based rows are not in that " + "version. A row is only comparable to a row with the same probability source.") + if transport.name == "decide": + notes.append( + "Measured through AINode's /v1/decide, which constrains the engine to one " + "option label and reads the distribution from the first token's logprobs. " + "Jevals 0.1.0 states that logprob-based rows are not in that version and " + "that its LLM rows are verbalized, so this is the same items, labels and " + "formulas through a different and generally stronger instrument than the " + "board's LLM rows.") + if any(set_summary(doc, limit)["options"] > 26 for doc in docs): + notes.append( + "On a set with more than 26 options /v1/decide letters them A..Z, " + "AA.., and a two-letter label that the tokenizer does not give its own " + "token shares its probability with the one-letter label of the same " + "first letter (see ainode/api/decide.py). Read the choice Decision " + "Score as a floor rather than a point.") + if transport.input_usd_per_mtok or transport.output_usd_per_mtok: + notes.append( + f"Cost is the posted rate given for this endpoint applied to the tokens it " + f"reported: ${transport.input_usd_per_mtok:g} per million input tokens and " + f"${transport.output_usd_per_mtok:g} per million output tokens.") + else: + notes.append( + "Cost is $0: no price was given for this endpoint, so there is nothing to " + "multiply the reported tokens by. The electricity is real and is not a " + "number this record claims to have measured.") + notes.append("No malformed-output retry, no discarded warm-up call, no bootstrap " + "interval and no rank: latency is measured at whatever --concurrency " + "this run used rather than the board's 4, and a Decision Score here is " + "a point value. Every deviation is listed in bench/decide/JEVALS.md.") + if limit: + notes.append( + f"PARTIAL RUN: --limit {limit} took the first {limit} question(s) of each " + "set, so this is a transport proof and not a suite result. A board listing " + "needs a complete run of every task in the tab at 5 repeats.") + if errors: + notes.append( + f"{len(errors)} decision(s) failed on a transport or protocol error rather " + f"than on the answer: {', '.join(errors[:10])}" + f"{', ...' if len(errors) > 10 else ''}. They are out of every figure, " + "which is the recipe's rule, and a set missing any (item, repeat) is not a " + "complete run.") + if malformed: + notes.append( + f"{len(malformed)} answer(s) could not be read as a probability vector over " + "the set's options. They are scored as the uniform distribution and a wrong " + "pick, which puts them in the Decision Score and the accuracy and keeps " + "them out of the ECE, the flip rates and the gate.") + if one_hot: + notes.append( + f"{one_hot} answer(s) carried no probabilities at all. They are scored " + "one-hot and are out of the calibration numbers, which is why a set's " + "calibrated_over can be smaller than its decisions.") + return notes + + +# ------------------------------------------------------------------ printing + +#: Wide enough for the deepest label the breakdown produces, which is four spaces of +#: indent plus a mixed set's `/` answer-space name. +COLUMNS = (("set", 38), ("type", 7), ("n", 6), ("acc", 7), ("floor", 7), ("DS", 8), + ("ECE pt", 8), ("hand-off", 10), ("gate", 12), ("flips", 7), ("swing", 7), + ("p50 ms", 8), ("p95 ms", 8), ("q/s", 7), ("bad", 5), ("cost", 9)) + + +def fmt(value, places=3): + return "-" if value is None else f"{value:.{places}f}" + + +def fmt_handoff(block): + hand = block.get("handoff_95") + if not hand: + return "-" + return f"{hand['share']:.2f}@{hand['threshold']:.2f}" + + +def fmt_gate(block): + gate = block.get("gate") or {} + if gate.get("threshold") is None: + return "none" + coverage = gate.get("coverage") + accuracy = gate.get("accuracy") + return (f"{gate['threshold']:.2f}:" + f"{'-' if coverage is None else f'{coverage:.2f}'}/" + f"{'-' if accuracy is None else f'{accuracy:.2f}'}") + + +def table_rows(block) -> list: + """``[cells, ...]``: one line per set, plus one per primitive of a mixed set. + + A mixed set's own line carries its accuracy and the mean of its primitives' Decision + Scores; the indented lines under it are the per-primitive blocks, which is where its + losses, its gate and its hand-off actually live. + """ + out = [] + for name, metrics in block["sets"].items(): + out.extend(_set_lines(name, metrics)) + for qtype, sub in (metrics.get("types") or {}).items(): + out.extend(_set_lines(f" {qtype}", sub)) + for space, sub in (metrics.get("spaces") or {}).items(): + out.extend(_set_lines(f" {space}", sub)) + return out + + +def _set_lines(name, metrics) -> list: + return [[ + name, + str(metrics.get("type") or "-"), + str(metrics.get("decisions") or 0), + fmt(metrics.get("accuracy")), + fmt(metrics.get("prior_accuracy")), + fmt(metrics.get("decision_score"), 1), + fmt(metrics.get("ece_points"), 1), + fmt_handoff(metrics), + fmt_gate(metrics), + fmt(metrics.get("pick_flip_rate"), 2), + fmt((metrics.get("confidence_swing") or {}).get("max"), 2), + "-" if metrics.get("p50_ms") is None else str(metrics["p50_ms"]), + "-" if metrics.get("p95_ms") is None else str(metrics["p95_ms"]), + fmt(metrics.get("questions_per_second"), 2), + str((metrics.get("malformed") or 0) + (metrics.get("failed") or 0)), + "$0" if not metrics.get("cost_usd") else f"${metrics['cost_usd']:.4f}", + ]] + + +def print_table(block, title: str, out=print) -> None: + """The per-set table, the reliability table under it, and what the columns are.""" + out(f"\n {title}") + out(" " + "".join(name.ljust(width) for name, width in COLUMNS)) + for cells in table_rows(block): + out(" " + "".join(cell.ljust(width) + for cell, (_h, width) in zip(cells, COLUMNS))) + mean = (block.get("overall") or {}).get("mean_decision_score") + out(f"\n mean Decision Score over {len(block['sets'])} set(s): " + f"{fmt(mean, 1)} (100 perfect, 0 the label base rates, negative worse)") + out(" floor = the base-rate answer's accuracy on the same items") + out(" hand-off = share it can take alone at 95% right @ the threshold that does it") + out(f" gate = published gate : this run's coverage / accuracy at it " + f"({jevals.PUBLISHED_GATE_SOURCE})") + out(" flips = share of questions whose pick changed at least once over the " + "repeats") + out(" swing = the largest confidence spread one question showed over its repeats") + out(" bad = malformed answers plus decisions that never came back") + out(" the board's own repeat (0 vs 1) and order (choice, 0/2/3/4) flip rates are " + "in the record") + for name, metrics in block["sets"].items(): + out(f"\n reliability, {jevals.BINS} bins on the pick's own probability, {name}") + out(" bin count accuracy mean conf") + for bucket in metrics["bins"]: + if not bucket["count"]: + continue + out(f" {bucket['lo']:.1f}-{bucket['hi']:.1f} {bucket['count']:5d} " + f"{fmt(bucket['accuracy']):>8} {fmt(bucket['confidence']):>9}") + if metrics.get("calibrated_over") == 0: + out(" nothing could be calibrated: no answer carried probabilities") + gold = metrics.get("vs_gold") + if gold: + out(f" against the gold distribution ({gold['over']} decisions): " + f"soft acc {fmt(gold.get('soft_accuracy'))} " + f"TV {fmt(gold.get('total_variation'))} " + f"KL {fmt(gold.get('kl'))} " + f"Brier vs gold {fmt(gold.get('brier_vs_gold'))}") + for name, entries in (block.get("contamination") or {}).items(): + who = ", ".join(entry["system"] for entry in entries) + out(f"\n CONTAMINATION, {name}: in the published training data of {who}. A row " + "for one of those systems here measures memorisation.") + + +def print_wrong(decisions, out=print, limit: int = 20) -> None: + """The confidently wrong list, which is this bench's useful output.""" + wrong = [d for d in jevals.scored(decisions) if not jevals.is_correct(d)] + if not wrong: + out("\n no wrong picks") + return + out(f"\n wrong picks ({len(wrong)} of {len(jevals.scored(decisions))}), " + "highest confidence first") + wrong.sort(key=lambda d: d.get("confidence") or 0.0, reverse=True) + for decision in wrong[:limit]: + vector = decision.get("vector") or {} + tag = "malformed" if decision.get("malformed") else "" + out(f" {decision['id']:<16} r{decision['repeat']} " + f"p={fmt(decision.get('confidence'), 2):<5} " + f"label {str(decision['label'])[:20]:<22} " + f"picked {str(decision.get('pick'))[:20]:<22} " + f"p(label)={fmt(vector.get(decision['label']), 2)} {tag}") + if len(wrong) > limit: + out(f" ... and {len(wrong) - limit} more, all of them in the record") + + +__all__ = ["COLUMNS", "DEFAULT_CONCURRENCY", "JEV_URL", "MODE", "QUESTION_KEY", + "RECIPE", "ROW_TOP_K", "SOURCE", "TRANSPORTS", "TYPESAFE_HOST", "Answer", + "DecideTransport", "SystemOneTransport", "Transport", "build_decide_block", + "build_jevals_block", "build_notes", "build_transport", "criteria_lines", + "decide_endpoint", "decision_for", "fmt", "fmt_gate", "fmt_handoff", + "is_typesafe", "legacy_overall", "option_criteria", "order_index", "plan", + "presented_options", "print_table", "print_wrong", "reported_model", + "row_for", "run", "set_summary", "systemone_url", "table_rows", + "wire_leaks", "DEFAULT_API_KEY"] diff --git a/bench/SCHEMA.md b/bench/SCHEMA.md index 533d7488..8d35b7ed 100644 --- a/bench/SCHEMA.md +++ b/bench/SCHEMA.md @@ -330,6 +330,183 @@ Rules specific to this block, all load-bearing: - No API key is ever in the record. Not in `settings`, not in `protocol`, not in a note. +### The `decide.jevals` sub-block + +A run of the **Jevals recipe** (`scripts/ainode-bench.py decide --suite ... --transport +...`, the recipe recorded in `bench/decide/JEVALS.md` with the URL and the date it was +read) writes the same `decide` block with three additions and two deliberate absences. +It exists so an AINode-served model can be read next to Jev and its clones on the same +public question sets with the same formulas. **Old records stay valid**: everything above +still describes a record with no `jevals` key. + +The additions: `decide.mode` is `"jevals-0.1.0"`, `decide.recipe` names the source page +and the date it was read, `decide.sources` describes each question set (its dataset, +split, pinned revision, licence, seed and whether its state hashes are upstream's or +ours), and `decide.jevals` holds the measurement. + +The absences, both load-bearing: `decide.sets` is `{}` and `decide.overall` carries **no +`brier`, `ece`, `bins` or `thresholds`**. Those four names mean the legacy definitions +above (a one-term Brier on the labeled option, five bins, gates at 0.8 and 0.9), and the +recipe's own arithmetic is a multiclass Brier over ten bins with a per-primitive gate. +Putting one under the other's name would make two incomparable numbers look like one, so +`scripts/render-bench-table.py` renders those cells as "not measured" for such a record +and reads the recipe's figures from `jevals` instead. `decide.overall` keeps only what +means the same thing either way: `n`, `answered`, `errors`, `accuracy`, `tokens`, +`cost_usd`, `p50_ms`, `p95_ms`. + +```json +"decide": { + "backend": "decide", "mode": "jevals-0.1.0", + "endpoint": "https://spark-1-dgx...:3443/v1/decide", + "recipe": { "source": "https://jevals.com/methodology", "read": "2026-09-21", + "suite": "0.1.0", "doc": "bench/decide/JEVALS.md", + "attribution": "Jevals (jevals.com), suite 0.1.0" }, + "item_set": { "id": "jevals-0.1.0", "version": "0.1.0", + "file": "bench/decide/sets/", "count": 900, + "sets": {"pubmedqa": 300, "banking77": 300, "helpsteer2": 300} }, + "protocol": { "transport": "decide", "concurrency": 4, "repeats": 5, + "batch_size": 1, "timeout_s": 120, + "input_usd_per_mtok": 0.0, "output_usd_per_mtok": 0.0, + "confidence": "...", "loss": "..." }, + "sources": [ { "id": "pubmedqa", "type": "noul", "questions": 300, "options": 2, + "seed": 20260918, "recipe_of_record": "jevals-0.1.0", + "gold_distributions": false, + "contamination": [ { "system": "...", "evidence": "...", + "source": "https://..." } ], + "source": { "dataset": "qiaojin/PubMedQA", "split": "train", + "hf_revision": "9001f285...", "license": "MIT", + "state_hash_source": "jevals" } } ], + "overall": { "n": 4500, "answered": 4500, "errors": 0, "accuracy": 0.712, + "tokens": {"in": 1, "out": 1}, "cost_usd": 0.0, + "p50_ms": 1420, "p95_ms": 7028 }, + "sets": {}, + "jevals": { + "recipe": { "...": "the same block as decide.recipe" }, + "formulas": "jevals-0.1.0", + "repeats": 5, "concurrency": 4, "batch_size": 1, "bins": 10, + "handoff_target": 0.95, "handoff_min_decisions": 100, "grid": 0.01, + "published_gates": { "pubmedqa": 0.91, "banking77": 0.96, "helpsteer2": null }, + "recipe_of_record": { "pubmedqa": "jevals-0.1.0" }, + "contamination": { "banking77": [ { "system": "...", "source": "https://..." } ] }, + "seconds": 812.4, + "sets": { + "pubmedqa": { + "recipe": "jevals-0.1.0", "type": "noul", + "items": 300, "repeats": 5, "decisions": 1500, + "failed": 0, "malformed": 0, "one_hot": 0, "calibrated_over": 1483, + "accuracy": 0.712, "prior_accuracy": 0.62, + "decision_score": 41.2, "loss": 0.188, "loss_prior": 0.32, + "ece_points": 5.8, + "bins": [ { "lo": 0.9, "hi": 1.0, "count": 900, "accuracy": 0.96, + "confidence": 0.991 } ], + "handoff_95": { "threshold": 0.93, "share": 0.62, "n": 930, + "accuracy": 0.951 }, + "gate": { "threshold": 0.91, "source": "jevals.com/methodology, suite 0.1.0, frozen", + "coverage": 0.6, "n": 900, "accuracy": 0.833 }, + "gate_local": 0.94, + "pick_flip_rate": 0.04, "pick_flip_over": 300, + "confidence_swing": { "max": 0.44, "mean": 0.02, + "question": "pubmedqa-37", "over": 300 }, + "repeat_flip_rate": 0.01, "repeat_flip_over": 300, + "order_flip_rate": null, "order_flip_over": 0, + "questions_per_second": 1.85, + "tokens": {"in": 748500, "out": 3000}, "cost_usd": 0.0, + "usd_per_1k_decisions": 0.0, + "p50_ms": 1420, "p95_ms": 7028 + } + }, + "overall": { "sets": ["pubmedqa"], "scored": ["pubmedqa"], + "mean_decision_score": 41.2 } + }, + "rows": [ + { "id": "pubmedqa-0", "set": "pubmedqa", "kind": "noul", "repeat": 0, "order": 0, + "label": "yes", "answer": "yes", "correct": true, + "p_answer": 0.999769, "p_label": 0.999769, + "top": {"yes": 0.999769, "no": 0.000231}, + "malformed": false, "malformed_reason": null, "one_hot": false, + "wall_ms": 7028, "tokens_in": 499, "tokens_out": 2, "error": null } + ] +} +``` + +Rules specific to this sub-block: + +- **The unit is a decision, not an item.** `decisions` is items times repeats, and + `accuracy` is over all of them. `items` and `repeats` say which is which, and + `decide.rows` is one row per decision with its `repeat` and its `order`. +- **`decision_score` is `100 * (1 - loss / loss_prior)`**, with both losses in the same + block so a reader can recompute it. 100 is perfect, 0 is answering with the label base + rates, and **a negative score is written as it is, never clamped**. `null` when the + prior's loss is 0, which happens only on a set where every item carries one label. +- **`prior_accuracy` is the guessing floor and travels with every block.** It is the + base-rate answer's own accuracy on the same items, so an accuracy of 0.62 next to a + floor of 0.62 reads as what it is. A block never states an accuracy without it. +- `loss` is the mean over items of the mean over that item's repeats: a multiclass Brier + for `choice` and `noul`, a ranked probability score over cumulative levels for `score`. + One item answered five times weighs the same as one answered once. +- **`ece_points` is in POINTS** (5.8 means 0.058), on the top label, over ten equal-width + bins by `min(9, floor(round(100*c)/10))`, and `bins` is the reliability table behind it: + ten rows always, `count: 0` and nulls for an empty one, which is the per-set + confidence-versus-accuracy data a model card draws a calibration curve from. `null` + when nothing could be calibrated, which is the board's dash. +- **`calibrated_over` can be smaller than `decisions`.** Malformed answers and answers + that carried no probabilities at all (`one_hot`) are in the accuracy and the Decision + Score and out of the ECE, the flip rates and the gate, which is the recipe's rule. +- `handoff_95` is the system's own threshold: the lowest confidence on the 0.01 grid + where at least 100 decisions clear it and at least 95 percent of those are right. + `share` is those decisions over **all** decisions, malformed included. `null` when no + threshold qualifies. +- `gate` is the **published frozen** gate for that primitive with this run's coverage and + accuracy at it, and `source` says where the threshold came from. `threshold: null` for + `score`, which has no gate in suite 0.1.0. `gate_local` is the same rule computed over + this one run and is **not** a board number. +- **Three flip figures, because they answer three questions.** `pick_flip_rate` is the + share of questions whose pick changed at least once across the repeats, which is the + one a caller who has to trust a single answer wants; `repeat_flip_rate` compares only + repeats 0 and 1 (byte-identical requests, so nondeterminism) and `order_flip_rate` + only the four distinct option orders of a `choice` set. Each carries its own `*_over` + count, and a rate nothing could be compared for is `null` with `over: 0`, never 0.0. +- `confidence_swing` is the largest spread one question's confidence showed across its + repeats, with the question named, plus the mean over questions. +- **A set holding more than one ANSWER SPACE is broken down by answer space, not by + primitive.** An answer space is one `(type, options)` pair, and it is the unit the + Decision Score is defined over: the label prior is the base rates of the labels in that + option list, so pooling two questions with different option lists builds a baseline over + an answer space neither of them has, and the thing that is meant to define 0 stops + defining it. Such a block carries `spaces` (one full metrics block each, with its + `options` beside it), a `types` roll-up per primitive the way a board shows one, a + `decision_score` that is the plain mean over the spaces, and `loss` / `loss_prior` of + `null` because a loss over two answer spaces is not a number. Its accuracy, its latency + and its ECE are over everything, because those do carry across. A space is named by the + question file (`/` for a mixed manifest) and `type#n` in first-seen + order otherwise. +- `vs_gold` is present only for a set that ships gold DISTRIBUTIONS rather than only + labels: `soft_accuracy` (the gold probability of the pick), `total_variation`, `kl` + (with its `kl_floor`) and `brier_vs_gold`. Its `definition` string states the + arithmetic, because these are **AINode's definitions and not a set card's columns of + the same names**. +- `within_one_level` is present only on a `score` block: the share of picks within one + level of the labeled one. +- **Every block names its recipe** (`recipe`), and every source names the recipe its own + published third-party numbers follow (`recipe_of_record`). Where those differ, only the + figures under this record's recipe are comparable across rows. +- **`contamination` is a finding, not a footnote.** A set in a listed system's published + training data carries that system, the evidence and the primary source URL, and the + record's notes repeat it in words. It says nothing about an AINode-served model that + did not train on the set; it says a row for THAT system on THAT set measures + memorisation. +- `batch_size` is always 1: one question per request, which is the recipe's rule and + removes the cross-question order effect. A set card whose own reference row was + measured several questions per request is reporting a different latency and a different + cost, and those columns are not comparable to this record's. +- **A row carries no state and no full vector.** The state is rebuilt from + `bench/decide/sets/.json` under the same `id` and verified against its + `state_sha256`; `top` keeps the five options the system was surest about and `p_label` + the labeled option's own probability, which is what a reader of a wrong answer needs. +- `partial: true` with a `limit` means `--limit` took only the first N questions of each + set. It is a transport proof, not a suite result, and the notes say so. A board listing + needs a complete run of every task in the tab at 5 repeats. + ## The `embed` block An embedding-bench run (`scripts/ainode-bench.py embed`) writes the same record with a diff --git a/bench/decide/JEVALS.md b/bench/decide/JEVALS.md new file mode 100644 index 00000000..0f54738e --- /dev/null +++ b/bench/decide/JEVALS.md @@ -0,0 +1,540 @@ +# The Jevals recipe, suite 0.1.0 + +Read from on **2026-09-21**, plus + (listing rules) and (the boards) the +same day. The three suite files were downloaded the same day from +`https://jevals.com/data/suites/0.1.0/.json` and are committed verbatim under +`bench/decide/sets/`. + +Two further sections below record, from their own primary sources read the same day, the +maintained multi-system board this field actually ranks on (**JevBench**, whose metric +names this bench adopts) and the **fourth question set** (`LocalLLaMA/typed-decisions`, +the one already in the `/v1/systemone` wire shape). A record says which recipe every +figure follows, because two boards measuring the same word differently is how a +comparison becomes a lie. + +This file exists so the bench can be read against the recipe it claims to follow +rather than against somebody's memory of it. Everything under "The recipe" is a +statement Jevals publishes. Everything under "What we had to author" is ours, and a +number this bench produces is only comparable to a board number to the extent those +authored parts do not matter. Both lists are meant to be short and complete. + +Attribution, per `https://jevals.com/policy/#license`: board data, run logs and suite +files are CC-BY-4.0, cited as "Jevals (jevals.com), release ". Jevals is an +independent project and is not affiliated with TypeSafe AI, and neither is AINode. + +## The recipe + +### What a decision is + +A system reads a **state** (any text or JSON) and answers one **typed question** with a +probability distribution over the allowed answers. Three question types, the three +primitives of Jev's interface: + +| Type | What it is | What comes back | +|------|-----------|-----------------| +| `noul` | Yes or no (short for Bernoulli) | P(yes) | +| `choice` | Pick one of up to 255 options | a probability per option | +| `score` | Place the state on an ordered rubric of 2 to 10 levels | a probability per level | + +The system's **pick** is its most likely answer. **Criteria** are the descriptions of +the options or levels that come with a question. **Confidence** is the probability of +the pick. Each primitive has its own board and its own ranking, and there is no overall +index across primitives. Every label is ground truth from a public human-labelled +dataset; no model grades another model. + +### The three tasks + +One task per primitive. Each is a fixed sample of **300 items** drawn by proportional +allocation (largest remainder) over the split's natural label distribution, after +dropping items whose state is longer than **6,000 Unicode code points**, with a fixed +seed (`20260918` in all three suite files). **Every item is answered 5 times.** + +| Board | Dataset | Config / split | Revision | Licence | Items | K | State fields | Question | +|-------|---------|----------------|----------|---------|-------|---|--------------|----------| +| `noul` | `qiaojin/PubMedQA` | `pqa_labeled` / `train` | `9001f285` | MIT | 300 | 2 | `question`, `context.contexts` | Given the context passages from a biomedical abstract, is the answer to the research question yes? | +| `choice` | `mteb/banking77` | `default` / `test` | `18072d26` | CC-BY-4.0 (Banking77, PolyAI; mirror tagged MIT) | 300 | 77 | `text` | Which intent does this banking customer's message express? | +| `score` | `nvidia/HelpSteer2` | `default` / `validation` | `990b2711` | CC-BY-4.0 | 300 | 5 | `prompt`, `response` | How helpful is the response to the prompt? | + +The state is a JSON object built **only** from those whitelisted fields, so no field +that reveals the label reaches any system. Item text is not republished by Jevals; each +item links to its upstream row. `state_sha256` is the SHA-256 of the UTF-8 bytes of +`JSON.stringify(state)` and is checked before every paid run. + +### The question wording, verbatim from the suite files + +`noul`, PubMedQA. `instructions` is the question in the table above. `criteria`: + +```json +{"true": "Yes: the passages support answering the question yes.", + "false": "No: the passages support answering the question no."} +``` + +`choice`, Banking77. `instructions` is the question in the table above. `criteria` is a +map of all 77 intent names to `null`: **the options carry no descriptions**, only their +names (`activate_my_card`, `age_limit`, ... `wrong_exchange_rate_for_cash_withdrawal`), +in Banking77's own label-id order, which is not alphabetical (`Refund_not_showing_up` +and `reverted_card_payment?` sit where their label ids put them). + +`score`, HelpSteer2 helpfulness. `instructions` is the question in the table above. +`criteria` is a list in level order, levels `"0"` through `"4"`: + +``` +0 Not helpful at all: the response misses the essence of what the user wanted. +1 Borderline unhelpful: mostly misses what the user wanted, but is useful in a small way. +2 Partially helpful: misses the overall goal of the request in some way. +3 Mostly helpful: aligned with the request, with some room for improvement. +4 Extremely helpful: completely aligned with what the prompt asked for. +``` + +So a `score` question's levels are defined by that ordered list of strings, one per +level, and the level names are the stringified indices `"0"`..`"4"`. The label is +HelpSteer2's own `helpfulness` column, which is already 0..4. + +### Probability vectors + +Every answer becomes a probability vector over the task's options before scoring. + +- Jev's values are rounded to 2 decimals on the wire; vectors that sum to 0.99 are + renormalized. +- An answer is **malformed** if it is not a JSON object with a probability map, names an + unknown or duplicate option (names must match exactly, including case and + punctuation), has a value that is not a finite number in [0, 1], or sums to 0. +- If the listed values sum to more than 1 they are divided by their sum. If they sum to + less than 1, the remainder is spread evenly over unlisted options (top-5 mode) or the + vector is divided by its sum (full mode). +- The pick is the most probable option. For `choice`, ties go to the option the system + listed first (for Jev, its own `choice` field); for `score`, ties go to the lower + level. A yes/no answer of exactly 0.5 has **no pick and counts as wrong**. + +### Decision Score + +``` +Decision Score = 100 * (1 - L_system / L_prior) +``` + +`L` is the mean per-item loss. Each item's loss is the average over its 5 repeats of + +- the multiclass Brier score, `sum_k (p_k - y_k)^2`, for `choice` and `noul`; +- the ranked probability score over cumulative level probabilities, + `sum_{k= `t` is at most +5%, with **at least 100** such decisions. If no such `t` exists the gate is empty. It is +computed once per suite version from the first release and then **frozen**, pooled +across every listed system, and each row shows its coverage (share of its decisions at +or above `t`) and its accuracy on those decisions. Rows without probabilities (one-hot +answers) have no coverage. + +**Current frozen gates, suite 0.1.0: `choice` 0.96 · `score` none · `noul` 0.91.** + +**Hand-off at 95 percent** gives each system its own threshold instead: the **lowest** +confidence `t` on the 0.01 grid at which its decisions with confidence >= `t` (at least +100 of them) are **at least 95% correct**. Its hand-off share is those decisions over +**all** its decisions, refused and malformed ones included. The threshold is chosen on +the same items it is measured on, so the share is optimistic in the same way for every +system. A row whose accuracy never reaches 95% at any threshold shows a dash. + +### Flip rates + +**Repeat flip rate**: share of items whose pick differs between repeats 0 and 1, which +are identical requests. It measures nondeterminism. **Order flip rate** (`choice` only): +share of items whose pick is not the same across the four distinct option orders +(repeats 0, 2, 3, 4). Refused and malformed answers are excluded from both. + +### Option order across the repeats + +Every system gets one question per request (batch size 1), zero-shot, with identical +instructions and criteria. **Choice options are presented in a seeded random order that +is identical for every system: repeats 0 and 1 share one order, repeats 2, 3 and 4 each +get a new one. Score levels and yes/no are never reordered.** + +### Cost, latency, failures + +- **Cost**: `$ per 1k decisions` is the total cost of all calls, malformed-output + retries and reasoning tokens included, divided by the number of decisions, times + 1,000. Always logged token usage times the list price snapshot in the run header. The + one discarded warm-up call at the start of a run is charged to it. Jev's posted rate + is $0.042 per million input tokens, output free. +- **Latency**: p95 is the 95th percentile, **nearest-rank**, of end-to-end time from + sending the request to having a parsed, validated answer, over all answered decisions. + It includes malformed-output retries but not transport-error backoff. Requests run at + **concurrency 4** after one discarded warm-up call, all from one machine on a + residential connection. +- Malformed output is retried up to **2 times**, and the retries count in cost and + latency. A refusal (a provider refusal field or a content-filter stop) is not retried. + After retries, a refused or malformed answer is **scored as the uniform distribution + and a wrong pick**: it counts in the Decision Score and accuracy and is excluded from + ECE, flip rates and the gate. +- **Transport failures** (HTTP errors, provider errors, a 60 s timeout, a truncated + response) are retried with backoff and **never scored**. A decision that still fails + is not written and the run resumes it later; a board cannot be built while any + (item, repeat) is missing. + +### 95 percent ranges and ranks + +Intervals are 95% percentile intervals from an item-cluster bootstrap: 2,000 seeded +resamples of items with replacement, all repeats of an item moving together, the prior's +loss recomputed on each resample, the same resamples shared by every row in a tab so +differences are paired. `rank = 1 + the number of rows that are significantly better`, +where row j beats row i when the 95% interval of `DS_j - DS_i` over the shared resamples +lies wholly above 0. Rows that cannot be told apart share a rank, and the pairwise tests +are not adjusted for multiple comparisons. + +**Board order**: a model wins a column when it is in that column's top two, ties +included (the two highest Decision Scores, with none when the label prior ties for +first; the two highest accuracies; the two lowest calibration gaps; the two lowest +prices; the two lowest p95 times). Boards list models by number of wins, then by +Decision Score, and models with as many wins share a place. The label prior has no wins +and no place. + +### Versions + +Every board is a release with a permanent URL under `/r//`. A patch changes no +score. A minor version regrades stored logs with no new calls. Adding, removing or +re-sampling a task is a **new suite version whose scores are not comparable with the +previous one**. Launch suite is 0.1.0; 1.0.0 is reserved for the full suite. + +### The LLM adapter (not what this bench uses) + +Jevals scores LLMs through one adapter for every model family: prompt-only JSON (no +structured-output mode), one pinned host per model with fallbacks disabled, provider +default temperature, the lowest reasoning setting the host allows, and a **verbalized** +probability distribution. Prompt hash `0383a0e3e592`; the published template is + +``` +You are answering one typed decision question about a state. + +STATE (JSON): +{state} + +QUESTION: {instructions} + +{options_heading} +{options} + +Give a probability for {what} +Reply with only this JSON object and nothing else: +{"probabilities": {"