Score decisions on the public Jevals sets, with the boards' own formulas

[webdevtodayjason]

The decision bench could only score AINode's own 110-item set, so an AINode-served model had no number anybody outside this repo could read. This adds the independent recipe the public boards use as a second measurement under the same decide subcommand, on the same public question sets, with the same formulas, so an AINode row can sit next to Jev and its clones.

The recipe is recorded in bench/decide/JEVALS.md against the pages it came from and the date they were read, with every deviation named. That file, not this PR body, is authoritative for what a number here means.

The recipe as implemented

Read from https://jevals.com/methodology (suite 0.1.0) and https://jevals.com/policy/ on 2026-09-21, plus https://github.com/fstandhartinger/jevbench's README (JevBench v1.2.10, the maintained multi-system board) and the LocalLLaMA/typed-decisions dataset card the same day.

• Decision Score 100 * (1 - L / L_prior). 100 perfect, 0 no better than answering with the label base rates, negative shown and never clamped. L is the mean per-item loss, itself the mean over that item's repeats: the multiclass Brier sum_k (p_k - y_k)^2 for choice and noul, the ranked probability score over cumulative levels sum_{k<K}(P_k - Y_k)^2/(K-1) for score. L_prior is the same loss for the label prior on the same items.
• Accuracy over items times repeats, with refused and malformed answers counted wrong. Every block carries its guessing floor as prior_accuracy and again as JevBench's majority_class_accuracy.
• ECE in points on the top label, ten equal-width bins by min(9, floor(round(100*c)/10)) (JavaScript rounding, so round_half_up, not Python's tie-to-even), with the full reliability table kept in the record so a model card can draw the calibration curve.
• Hand-off at 95 percent: the lowest confidence on the 0.01 grid where at least 100 decisions clear it and at least 95 percent of those are right; the share is those decisions over all decisions, malformed included. A dash when accuracy never reaches 95 percent.
• The gate: the published frozen thresholds (choice 0.96, noul 0.91, score none) with this run's coverage and accuracy at them, plus a gate_local computed by the same rule over this one run and labelled as not a board number.
• 5 repeats per question (--repeats), one question per request, choice options presented in a seeded order that repeats 0 and 1 share and repeats 2, 3 and 4 each get their own; score levels and yes/no never reordered.
• Flips as first-class metrics: pick_flip_rate (share of questions whose pick changed at least once across the repeats) and confidence_swing (the largest spread one question's confidence showed, with the question named), beside the board's own repeat_flip_rate (repeats 0 and 1) and order_flip_rate (the four option orders).
• Malformed detection and normalization exactly as published, with JevBench's schema_validity and schema_validity_strict under its 2 percent and 0.001 sum tolerances. A yes/no answer of exactly 0.5 has no pick and counts as wrong. A transport failure is never scored.
• Cost when a price is given, as usd_per_1k_decisions beside the total, with JevBench's cost_basis saying which it is, so a $0 is not read as free.

The sets

Committed as manifests under bench/decide/sets/, with no item text. The three Jevals suite files are verbatim from https://jevals.com/data/suites/0.1.0/<id>.json (CC-BY-4.0).

Set	Primitive	Questions	Upstream	Licence
pubmedqa	noul	300	qiaojin/PubMedQA pqa_labeled/train @ 9001f285	MIT
banking77	choice, K=77	300	mteb/banking77 default/test @ 18072d26	CC-BY-4.0
helpsteer2	score, K=5	300	nvidia/HelpSteer2 default/validation @ 990b2711	CC-BY-4.0
typed-decisions	mixed, 5 per case	2000 (400 cases)	LocalLLaMA/typed-decisions all/test @ ea930645	Apache-2.0

decide download fetches the item text into the gitignored bench/decide/cache/ and verifies every state against the manifest's state_sha256 before writing. All 900 Jevals hashes reproduce, which is what makes a number here comparable to a board number, and it is how the state construction was recovered in the first place: Banking77's state key is message, not the upstream field name text, and PubMedQA's is context holding the list of passages, not context.contexts. Neither is published. One trap worth the line: mteb/banking77 ships both a test.jsonl (3,080 lines) and the parquet behind default/test (3,076 rows), and the suite's row_idx is the parquet index, so loading from the jsonl puts 258 of the 300 states on the wrong row. The hash check catches it.

typed-decisions is the fourth set because it is the one public set already in the /v1/systemone request shape, with a train/test split and gold distributions; its hashes are ours, computed at the pinned revision, and the manifest says so.

Deviations from jevals.com, called out

1. No bootstrap intervals and no ranks. A rank is a property of a board with several rows on shared resamples. Two close AINode scores must not be read as one beating the other.
2. /v1/decide is logprob-based, which suite 0.1.0 excludes. It constrains the engine to one option label and reads the distribution off the first token's logprobs. Jevals states logprob rows are not in that version and its LLM rows are verbalized; JevBench states it uses token-level logprobs for nobody. Every record carries probability_source of logprob, native or verbalized in JevBench's own vocabulary, because a row is only comparable to a row that got its probabilities the same way. The systemone transport is native.
3. /v1/decide letters its options, and past 26 a two-letter label can share its mass with a one-letter one (the limitation already documented in ainode/api/decide.py). Banking77 has 77 options, so its Decision Score through that transport reads as a floor, not a point. The run's notes say so.
4. No malformed retries. Jevals retries twice. This scores the first answer and reports the malformed count, so that count is about the model and not about a retry policy.
5. Concurrency is a flag, not 4, and no warm-up call is discarded. Latency is measured at whatever the run used and the record says which, so a p95 here is not a board p95.
6. The adapter prompt is not implemented. Its published template leaves {options_heading}, {options} and {what} unfilled and the hash is over the filled version, so it cannot be reconstructed. Both transports here take a typed question natively, so there is no prompt of theirs to match.
7. The seeded option order is ours. Jevals publishes the properties, not the generator. Ours satisfies every published property except being the same permutation, so an order flip rate here is a real one and is not theirs.
8. The score response shape for /v1/systemone is authored: jevals.com documents the primitive but not the field a Jev-format server answers it in, so the parser accepts score, level or choice and a response matching none is one malformed row.
9. vs_gold is ours. soft_accuracy, total_variation, kl and brier_vs_gold on the one set with gold distributions are AINode's definitions, stated in the block, because the card prints columns of those names without publishing the arithmetic.
10. Our label prior is fitted on the evaluated items (the Jevals rule). The typed-decisions card's Prior row fits on the train split, so the two are close and not equal, and a record must not be read as quoting the card's.

Contamination, in the record

banking77 is in Kev's published training data (the kev-9b model card front matter lists legacy-datasets/banking77 under datasets:) and in Laya's training mix; pubmedqa sits inside decider's development loop. Each finding travels with its evidence and its primary source URL in sets.CONTAMINATION, in the record and in the run's notes. A row for one of those systems on that set measures memorisation. It says nothing about an AINode-served model that did not train on them.

One real bug found and fixed while building this

Pooling a mixed set's questions into one label prior is wrong: typed-decisions asks action with four options in one workflow and five in another, and a single pooled prior is a baseline over an answer space neither question has. The prior scored -297 on the choice board before the fix. A set is now broken down by answer space (one (type, options) pair, named <workflow>/<question>) with a per-primitive roll-up, and tests/test_bench_decide_jevals.py pins that the label prior scores exactly 0 on all four committed sets and every one of their spaces.

No answer key ever reaches the model

Labels live in a question file's separate labels map, never inside a question. A question carrying anything answer-key-shaped is a load error, and the assembled request body is checked again before it is sent: a leak refuses the request rather than producing a very good score. A gold distribution is allowed in a file and never on the wire. The round-trip tests assert that every body a fake endpoint received was clean.

Commands

# once: fetch the item text. Not committed; verified against the manifests' hashes
python3 scripts/ainode-bench.py decide download

# an AINode-served model through its own decision endpoint
python3 scripts/ainode-bench.py decide --suite all --transport decide \
    --endpoint http://100.122.26.9:3000/v1 --ainode http://100.122.26.9:3000 \
    --model ornith-ai/Ornith-1.5-35B-A3B-NVFP4 --label "Ornith on Spark-1, Jevals 0.1.0"

# anything speaking the Jev wire format: TypeSafe's hosted Jev, Kev, laya.cpp
python3 scripts/ainode-bench.py decide --suite all --transport systemone \
    --endpoint https://api.typesafe.ai/v1 --price-in 0.042 --label "jev-latest, Jevals 0.1.0"

# a private blind set in the same shape, never committed
python3 scripts/ainode-bench.py decide --questions /path/to/blind.json \
    --transport systemone --endpoint http://kev-host:8080/v1 --label blind-1

# the plan, every set, one example request per primitive, nothing sent
python3 scripts/ainode-bench.py decide --suite all --transport decide \
    --endpoint http://node:3000/v1 --dry-run --limit 1

# gates
python3 -m pytest tests/test_bench_decide_jevals.py tests/test_bench_decide.py -q   # 221 pass
python3 -m pytest tests/ -q                                                         # 2892 pass
ruff check ainode/ scripts/ tests/
python3 scripts/render-bench-table.py --check

Which key goes where is decided by the endpoint's HOST, not by a flag. api.typesafe.ai resolves TypeSafe's credential; anything else resolves the node's. No ordering of flags can post a fleet key to a vendor. Nothing is ever printed but the source it came from.

What was run, and what was not

Only a 10-question transport smoke against spark-1-dgx:3443/v1/decide with the fleet key, to prove the wire: 10 of 10 answered, accuracy 0.800 against a floor of 0.800, Decision Score 7.8, ECE 19.5 points, coverage 0.60 at the published 0.91 gate, p50 1420 ms, no malformed answers, no key in the record. That record is deliberately not committed: a 10-question single-repeat run is a transport proof, not a suite result, and it has no business in the published table. No full set was run from this worker.

The README's "Decision runs" table gains Decision Score and Hand-off at 95%, and its ECE cell now carries its unit (5.8 pt for the ten-bin recipe figure, 0.059 for the legacy five-bin ratio) so two definitions never share a column silently. The two existing legacy rows read "not measured" in the new columns, which is the truth: they did not take those measurements.

Changelog text for the release PR

Added

• The decision bench scores models on the public question sets the independent Jevals boards use, with the boards' own formulas, so an AINode-served model can be compared with Jev and its clones on one recipe. ainode-bench decide --suite all --transport decide|systemone runs PubMedQA (noul, 300), Banking77 (choice, 300, K=77), HelpSteer2 helpfulness (score, 300) and LocalLLaMA/typed-decisions (mixed, 2000 decisions over 400 cases) at 5 repeats per question, against either AINode's own POST /v1/decide or any endpoint speaking the Jev POST /v1/systemone wire format. Per set it reports the Decision Score against the label prior (100 perfect, 0 the base rates, negative worse), accuracy with its guessing floor, the calibration gap in points over ten bins with the reliability table behind it, the hand-off share at 95 percent, the published gate with this run's coverage, pick flips and confidence swing across the repeats, p50 and p95 latency, questions per second, malformed answers, and cost when a price is given. --questions <file> scores any question file in the same shape, so a private blind set never has to be committed. The recipe, the date each source page was read and every deviation from it are in bench/decide/JEVALS.md; the sets are bench/decide/sets/ and ainode-bench decide download fetches the item text, which is not committed, and verifies every state against its published hash.

🤖 Generated with Claude Code