diff --git a/AGENTS.md b/AGENTS.md index 6883622e..0c906e59 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -26,8 +26,9 @@ State / architecture / decisions / "why": Obsidian Vault, `AINode` (cluster ops: - **The agentic rubric (`ainode/bench/agentic/`, `scripts/ainode-bench.py agentic`) scores on mechanical verdicts only.** No judge model and no pass read by eye: a probe is decided by a regex, a parsed tool call, an executed subprocess or a compared call trace. Every checker is a module-level function of plain values (the reply text, the calls, the delivered tool results) so `tests/test_bench_agentic.py` can canned-response all of them with no network, and a probe that raises is one recorded failure rather than a dead run. **Group C executes model-written code on the machine driving the bench** (temporary directory, 60-second timeout, `sys.executable`), the same trade the harness bench makes when it runs an agent's edit. A run writes an `agentic` block and no `results` block, and a group `--groups` or `--quick` left out is absent from the score, never a zero. New rubric numbers go in that block: the hand-typed `rubric` key in three older records is a historical claim and must not be added to a new record. Probes and flags: `bench/agentic/README.md`. - **The decision bench (`ainode/bench/decide/`, `scripts/ainode-bench.py decide`) scores typed decisions against labels, and its confidence numbers are the product.** Accuracy is the weakest number in the block: a wrong answer at 0.95 is the failure mode, so every block carries the Brier score on the labeled option, an expected calibration error with the five-bin reliability table behind it, and the count of wrong answers surviving a 0.8 and a 0.9 gate. **A probability nobody reported is absent, never assumed** (such a row is in the accuracy, out of the calibration, and counted in `no_confidence`), **an item that failed is one row with an `error`** and never a wrong answer, and **cost is a posted vendor rate over reported tokens or `0`** for a local backend, never an estimate. `bench/decide/items.json` is repo data versioned next to its results: loading is strict, a malformed item is a load error rather than a skipped item, and a set's items must share one kind, question and option set because a set is one measurement. Backends are split `request()` / `parse()` as pure functions so `tests/test_bench_decide.py` pins every request shape and response shape with canned payloads and no network. **The TypeSafe key is never printed, never written into a record and never put in a note**: a run reports only which of `--api-key`, `$TYPESAFE_API_KEY` or `~/.jev_api_key` it came from. Sets, metrics and flags: `bench/decide/README.md`. - **The speech bench (`ainode/bench/speech/`, `scripts/ainode-bench.py speech`) scores against committed audio, and both halves of that are load-bearing.** A word error rate is only comparable over the same bytes, so the ten clips live in `bench/speech/clips/` as repo data (1.4 MB) rather than being synthesised per run, and **`clips.CLIPS_VERSION` is bumped on any edit to a text, a voice or a WAV**; `--generate-clips` rebuilds the set with macOS `say` plus `afconvert` and is a maintenance step a run never takes. The reference is the exact string handed to `say`, fixed before the run, and **nothing adjusts a reference after a transcript is seen**: a reference edited to match what a model said makes the rate a statement about the editor. The normaliser is part of the measurement, so it is versioned (`metrics.NORMALIZER_VERSION`) and the record carries BOTH rates, `wer` with number words folded to digits and `wer_orthographic` with case and punctuation only, because a transcript that heard every word and wrote "9" for "nine" is not a hearing error and one number alone hides which kind it was. Nothing is folded that changes a word: no stopword list, no stemming, no synonym map, no per-clip exception. `wer` is pooled over words, never a mean of per-clip rates. **A clip that failed is one row with an `error` and nulls for every number**, counted out of every rate, percentile and factor, never folded in as a 100 percent error rate: a transport failure inside a figure a reader takes as the model's is the one mistake this section can make. It is the one bench whose request body is not JSON (`client.py` assembles the multipart itself), so a run through a node's `:3000/v1` exercises the fleet's own audio path; the block is `bench/SCHEMA.md`. -- **One proxy handler serves every forwarded inference path**: `proxy_to_vllm` is registered for `POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`, `/v1/messages/count_tokens`, `/v1/responses`, `/v1/rerank`, `/v1/score`, `/v1/audio/transcriptions`, `/v1/audio/translations`, `/tokenize` and `/detokenize` (`GET /v1/models` is the federated union and does not forward; `POST /v1/embeddings` keeps its own handler because it validates the body first, and `/v1/decide` composes its own completions). Add a path by registering it on that handler, never by writing a second proxy: routing on the body's `model`, transport failover, the multimodal ordering below, SSE passthrough and header passthrough are all protocol-agnostic and already there. Two of the paths are NOT under `/v1` because vLLM does not serve them there (`/tokenize`, `/detokenize`), so the table is the authority on the path and not a prefix rule. **The two audio paths are the ones whose body is NOT JSON**: OpenAI's speech-to-text API is a `multipart/form-data` upload with the model id as a form field, so the handler reads it with `api/multipart.py::form_fields` over the body it already buffered (never `request.multipart()`, which consumes the stream the proxy still has to forward) and forwards the bytes UNCHANGED under the caller's own `Content-Type`: a multipart body is only parseable against the boundary in its own header, so re-encoding the parts hands the engine a body the forwarded header no longer describes. A multipart body that names no `model` is a 400 naming the field, never a fallback to this node's own model, which would send someone's audio to a chat engine. They are also the first pair whose existence is MODEL-CONDITIONAL: vLLM attaches its speech-to-text router only when the served model reports the `transcription` task, so no chat or pooling engine's `/openapi.json` lists them and the check below cannot be run on one. For a path like that, the evidence is the router's own declaration in the engine image the recipe pins (`entrypoints/openai/speech_to_text/api_router.py`), and the `openapi.json` check still applies the first time such a model actually serves. Forward `request.path_qs`, not `request.path`: Claude Code posts to `/v1/messages?beta=true`. This is a route table and not a catch-all: an unregistered path stays a 404, which is why a new path is added only after `curl http://:/openapi.json` on a real engine says the engine answers it. -- **`/v1/decide`'s response shape is a contract, not an implementation detail** (`api/decide.py`). Its bench is written against the exact shape (`model`, `node`, `latency_ms`, `decisions[key] = {answer, confidence, distribution, latency_ms}`, `usage = {prompt_tokens, completion_tokens, calls}`), so a `200` always carries every question asked: a bad request is a `400` and an engine that cannot answer is a `503`, never a partial `decisions` block. Probabilities are keyed by the caller's OPTION strings, never by the letters used to constrain the engine. It is the one `/v1` path deliberately NOT on `proxy_to_vllm`, because it composes N grammar-constrained chat completions of its own from one request and has no caller body to forward; it still routes through the proxy's own `_routing_candidates` and the shared `app["client_session"]`, so never give it its own routing rule or HTTP stack. The constraint field is vLLM 0.27.1's `structured_outputs: {"choice": [...]}`: the legacy `guided_choice` is accepted by that image and then silently ignored, so sending it instead would produce free prose with no error. +- **One proxy handler serves every forwarded inference path**: `proxy_to_vllm` is registered for `POST /v1/chat/completions`, `/v1/completions`, `/v1/messages`, `/v1/messages/count_tokens`, `/v1/responses`, `/v1/rerank`, `/v1/score`, `/v1/audio/transcriptions`, `/v1/audio/translations`, `/tokenize` and `/detokenize` (`GET /v1/models` is the federated union and does not forward; `POST /v1/embeddings` keeps its own handler because it validates the body first, and `/v1/decide` plus `/v1/systemone` compose their own completions). Add a path by registering it on that handler, never by writing a second proxy: routing on the body's `model`, transport failover, the multimodal ordering below, SSE passthrough and header passthrough are all protocol-agnostic and already there. Two of the paths are NOT under `/v1` because vLLM does not serve them there (`/tokenize`, `/detokenize`), so the table is the authority on the path and not a prefix rule. **The two audio paths are the ones whose body is NOT JSON**: OpenAI's speech-to-text API is a `multipart/form-data` upload with the model id as a form field, so the handler reads it with `api/multipart.py::form_fields` over the body it already buffered (never `request.multipart()`, which consumes the stream the proxy still has to forward) and forwards the bytes UNCHANGED under the caller's own `Content-Type`: a multipart body is only parseable against the boundary in its own header, so re-encoding the parts hands the engine a body the forwarded header no longer describes. A multipart body that names no `model` is a 400 naming the field, never a fallback to this node's own model, which would send someone's audio to a chat engine. They are also the first pair whose existence is MODEL-CONDITIONAL: vLLM attaches its speech-to-text router only when the served model reports the `transcription` task, so no chat or pooling engine's `/openapi.json` lists them and the check below cannot be run on one. For a path like that, the evidence is the router's own declaration in the engine image the recipe pins (`entrypoints/openai/speech_to_text/api_router.py`), and the `openapi.json` check still applies the first time such a model actually serves. Forward `request.path_qs`, not `request.path`: Claude Code posts to `/v1/messages?beta=true`. This is a route table and not a catch-all: an unregistered path stays a 404, which is why a new path is added only after `curl http://:/openapi.json` on a real engine says the engine answers it. +- **`/v1/decide`'s response shape is a contract, not an implementation detail** (`api/decide.py`). Its bench is written against the exact shape (`model`, `node`, `latency_ms`, `decisions[key] = {answer, confidence, distribution, latency_ms}`, `usage = {prompt_tokens, completion_tokens, calls}`), so a `200` always carries every question asked: a bad request is a `400` and an engine that cannot answer is a `503`, never a partial `decisions` block. Probabilities are keyed by the caller's OPTION strings, never by the letters used to constrain the engine. It is the one `/v1` path deliberately NOT on `proxy_to_vllm`, because it composes N grammar-constrained chat completions of its own from one request and has no caller body to forward; it still routes through the proxy's own `_routing_candidates` and the shared `app["client_session"]`, so never give it its own routing rule or HTTP stack. The constraint field is vLLM 0.27.1's `structured_outputs: {"choice": [...]}`: the legacy `guided_choice` is accepted by that image and then silently ignored, so sending it instead would produce free prose with no error. The engine-facing half of a decision request is `decide.py::run_questions`, shared with `/v1/systemone`: one path to the engines and one way the probabilities are read, so a route decides the status and the shape and nothing else. +- **`/v1/systemone` is TypeSafe's wire format and nothing else** (`api/systemone.py`). It exists so a client written for the hosted System One endpoint (Titanium's JDE `jevJudge({endpoint, model})`, browser-use's jev-ultrafast, the TypeSafe SDK, the playground) answers off a model on this fleet with its endpoint changed and nothing else, which makes the request and response shapes THEIRS and not ours to tidy: `{state, model, questions}` in, `{model, answers, usage: {input_tokens, output_tokens}, latency_ms}` out, a `choice` answering with the caller's own criteria key, a `noul` answering P(true), a `score` answering the expected level with a `legend` from position to level name, a `422` naming the field on a malformed request and a `503` when no node serves the model. JDE's `parseAnswers` discards the WHOLE answer set over one answer it cannot read, so every probability is finite and inside [0, 1] (`systemone.py::probability`) and a question the engine left unanswered is a `503`, never a 200 carrying an invented option. **`confidence` is NOT the picked option's probability**: the hosted service reports it chance corrected, `(n * p_max - 1) / (n - 1)`, which is INFERRED from every example in TypeSafe's published docs and SDK types (`systemone.py::normalized_confidence`) because no document states it, and a caller's bands are tuned against those numbers; the raw distribution goes out untouched beside it, so the formula is revisable against evidence and the numbers it came from are never lost. **A question may carry at most `TOP_LOGPROBS` criteria**, which is the engine request's ceiling and not the format's 255: only the top 20 labels come back with a probability, so a wider option set is a `422` naming the cap rather than a distribution missing its tail, and raising it means a second pass over the remaining labels, which is a measurement and not a constant. Translate in and out around `run_questions`: a gap here is never answered with a second engine path, a second routing rule or a second reading of the logprobs. **Calibration is the model's**, so nothing on this path rescales, tempers or corrects what the engine reported, and the way to find out what a local model's confidence is worth stays `scripts/ainode-bench.py decide`. `tests/test_systemone.py` carries JDE's own reader rule for rule and replays a real case from its blind set, read where it lives (`JDE_COMPLETION_CASES`, default `/Users/sem/code/jde/cases/`) and never copied in: it is someone else's measurement data, and the test skips where it is absent. - **A multimodal chat request consults the capability cache before routing** (`api/server.py::proxy_to_vllm`). A body carrying an `image_url` / `input_audio` / `video_url` / `file` part, or the Anthropic Messages spelling (an `image` / `document` block, including one nested in a `tool_result`), is never routed on the model id alone: order candidates accepting-first (cached `vision: true`), then never-probed, and drop instances cached `vision: false`; vLLM's `may be provided in one prompt` 400 is a routing miss, so record `vision: false` and fail over, while every other 4xx goes back to the caller untouched. A request with no media keeps the plain order (local hop first, then peers). There is ONE capability cache: `app["chat_caps_cache"]`, filled by `/api/models/caps` in `api/chat_routes.py`, which probes remote instances directly on their engine port. Never add a second. - **Every node-to-node request AINode makes carries the FLEET KEY, and there is one helper that puts it there** (`ainode/auth/fleet.py`: `fleet_headers(app)` off a running app, `fleet_key_headers(secret)` off a config). The key is `HMAC-SHA256(cluster_secret, "ainode-fleet-key-v1")`, so every node holding the secret computes the same one, the join flow already distributes it, rotation follows the secret, and NOTHING new is written to disk. The middleware accepts it as the caller id `fleet` (`auth/middleware.py::identify_caller`), reading the secret LIVE per request, so a node whose secret differs refuses its would-be peers exactly as it drops their datagrams. Auth was enabled-able and unusable on a cluster before this: a node with `auth.enabled` answered 401 to its own fan-outs, so the fleet ran open. Add a peer call and you add the header: an ENGINE port is not a peer call in this sense (the inference proxy, the capability probes and the embeddings route talk to a vLLM container, which never sees this middleware), and `POST /api/cluster/join` is keyless by construction. `tests/test_fleet_auth.py` WALKS THE SOURCE for peer URLs and fails on one whose function does not name the helper, with an exempt list that has to state a reason. - **A fresh install requires a key, and an update never changes an installed node's access control** (`scripts/install.sh`). The installer mints one key on a node with no `config.json`, stores the SHA-256 the way `AuthConfig` does, prints the plaintext ONCE in a box, and the summary reads "API protected, one key"; `AINODE_AUTH=off` keeps the old open behaviour and prints what that choice means. Both gates matter: minting on an existing home would lock out every client the operator already pointed at the node, with a key they never saw. Because of this, `ainode update` verifies the running release on `/api/health` and not `/api/status` (health is the one keyless route, which is why it carries `version`): reading a keyed route there made every update on a protected node pull, pin, restart and then report that it had not applied. diff --git a/ainode/api/decide.py b/ainode/api/decide.py index d3941795..aeea8c96 100644 --- a/ainode/api/decide.py +++ b/ainode/api/decide.py @@ -5,6 +5,13 @@ questions; every question is asked at once and each answer comes back with the probability the served model put on it. +The pieces below are also the core ``POST /v1/systemone`` runs on +(``api/systemone.py``, TypeSafe's Jev wire format over a local model): resolving +the model, building the prompts, asking every question at once and reading the +answers off the logprobs all happen here once, and that route translates into and +out of this shape around them. So keep them importable, and keep the pure ones +pure. + Why this is not a proxy path: the forwarded inference routes all hand ONE upstream request the caller's own body (see ``server.py::proxy_to_vllm`` and the invariant in ``AGENTS.md``). ``/v1/decide`` composes N chat completions of its @@ -36,7 +43,7 @@ import json import math import time -from typing import Any, Optional +from typing import Any, NamedTuple, Optional import aiohttp from aiohttp import web @@ -53,6 +60,13 @@ # the tail is measurable. MAX_OPTIONS = 255 +# How many alternatives the engine is asked for on the answer token. It is the +# ceiling on how many options can carry a probability back from one call: a label +# outside the top 20 is reported at 0 whatever the model thought of it, which is +# why `/v1/systemone` refuses a question with more criteria than this rather than +# answering one with a truncated distribution. +TOP_LOGPROBS = 20 + # Room for the longest label plus the end-of-turn token the template emits. LABEL_TOKEN_HEADROOM = 1 @@ -70,7 +84,12 @@ class DecideError(Exception): - """A bad request shape. Carries the message the caller gets in a 400.""" + """A bad request shape. Carries the message the caller gets in the 4xx. + + ``/v1/decide`` answers it as a 400 and ``/v1/systemone`` as the 422 the Jev + format specifies, so the message says what is wrong and never which status + somebody is about to put it in. + """ # --------------------------------------------------------------------- labels @@ -100,7 +119,7 @@ def option_labels(count: int) -> list[str]: # ----------------------------------------------------------------- validation -def _serialize_state(state: Any) -> str: +def serialize_state(state: Any) -> str: """The state as the model sees it: a string verbatim, anything else compact JSON.""" if state is None: return "" @@ -222,7 +241,7 @@ def build_chat_body(model: str, messages: list[dict], labels: list[str]) -> dict "temperature": 0, "stream": False, "logprobs": True, - "top_logprobs": 20, + "top_logprobs": TOP_LOGPROBS, # Both switch names, the way ``bench/measure.py`` does it: Qwen-family # templates read enable_thinking, DeepSeek V4 reads thinking, and a # template ignores the one it does not use. A decision function must not @@ -465,6 +484,68 @@ def node_name_for(request: web.Request, model: str, cand) -> Optional[str]: return entry.get("node_name") or None +# ------------------------------------------------------------------------ run + + +class DecideRun(NamedTuple): + """What one set of questions came back as, before anybody shapes a response. + + ``decisions`` holds one entry per question the engine answered, keyed the way + the caller keyed the question and in the caller's order; ``payloads`` are the + raw engine responses, for the usage block; ``landed`` is the candidate the + first answer came from, for naming the node; ``failures`` is one line per + question that got no answer. + """ + + decisions: dict + payloads: list + landed: Optional[tuple] + failures: list + + +async def run_questions(request: web.Request, model: str, questions: dict[str, dict], + state: str, instructions: Optional[str], + candidates: list) -> DecideRun: + """Ask every question at once against `candidates`, and read the answers. + + The whole engine-facing half of a decision request, shared by ``/v1/decide`` + and ``/v1/systemone`` so there is one path to the engines and one way the + probabilities are read. It reports what happened and decides nothing about + the response: the status, the shape and what a failure means belong to the + route. + + All questions are in flight together. They share a byte-identical prompt + prefix, so once one of them has prefilled it the rest read the shared state + out of the engine's prefix cache instead of paying for it again. + """ + session: aiohttp.ClientSession = request.app["client_session"] + keys = list(questions) + bodies = [] + for key in keys: + spec = questions[key] + labels = option_labels(len(spec["options"])) + messages = build_messages(state, instructions, spec["question"], + spec["options"]) + bodies.append(build_chat_body(model, messages, labels)) + + results = await asyncio.gather( + *(ask_one(session, candidates, b) for b in bodies)) + + decisions: dict = {} + payloads: list[dict] = [] + failures: list[str] = [] + landed = None + for key, (payload, cand, extra) in zip(keys, results): + if payload is None: + failures.append(f"{key}: {extra}") + continue + payloads.append(payload) + landed = landed or cand + decisions[key] = decision_from_payload(payload, questions[key]["options"], + float(extra)) + return DecideRun(decisions, payloads, landed, failures) + + # -------------------------------------------------------------------- handler @@ -474,7 +555,7 @@ def _bad_request(message: str) -> web.Response: status=400) -def _unavailable(message: str) -> web.Response: +def unavailable(message: str) -> web.Response: return web.json_response({"error": {"message": message, "type": "service_unavailable"}}, status=503) @@ -494,7 +575,7 @@ async def handle_decide(request: web.Request) -> web.Response: try: model = resolve_model(request, body.get("model")) questions = normalize_questions(body.get("questions")) - state = _serialize_state(body.get("state")) + state = serialize_state(body.get("state")) instructions = body.get("instructions") if instructions is not None and not isinstance(instructions, str): raise DecideError("'instructions' must be a string when given") @@ -509,53 +590,27 @@ async def handle_decide(request: web.Request) -> web.Response: candidates = candidates_for(request, model) if not candidates: - return _unavailable(f"no node is serving '{model}'") + return unavailable(f"no node is serving '{model}'") - session: aiohttp.ClientSession = request.app["client_session"] - keys = list(questions) - bodies = [] - for key in keys: - spec = questions[key] - labels = option_labels(len(spec["options"])) - messages = build_messages(state, instructions, spec["question"], - spec["options"]) - bodies.append(build_chat_body(model, messages, labels)) - - # All questions in flight at once. They share a byte-identical prompt prefix, - # so once one of them has prefilled it the rest read the shared state out of - # the engine's prefix cache instead of paying for it again. - results = await asyncio.gather( - *(ask_one(session, candidates, b) for b in bodies)) - - decisions: dict = {} - payloads: list[dict] = [] - landed = None - failures: list[str] = [] - for key, (payload, cand, extra) in zip(keys, results): - if payload is None: - failures.append(f"{key}: {extra}") - continue - payloads.append(payload) - landed = landed or cand - decisions[key] = decision_from_payload(payload, questions[key]["options"], - float(extra)) + run = await run_questions(request, model, questions, state, instructions, + candidates) collector = request.app.get("metrics_collector") total_ms = (time.monotonic() - started) * 1000 - if failures: + if run.failures: # A 200 always carries every question. A partial answer set would read # like a decision the model declined to make, and the bench treats this # response shape as fixed. if collector is not None: collector.record_request(model, total_ms, error=True) - return _unavailable(f"engine calls failed for '{model}': " - + "; ".join(failures[:5])) + return unavailable(f"engine calls failed for '{model}': " + + "; ".join(run.failures[:5])) if collector is not None: collector.record_request(model, total_ms, error=False) return web.json_response({ "model": model, - "node": node_name_for(request, model, landed), + "node": node_name_for(request, model, run.landed), "latency_ms": round(total_ms, 1), - "decisions": decisions, - "usage": merge_usage(payloads), + "decisions": run.decisions, + "usage": merge_usage(run.payloads), }) diff --git a/ainode/api/server.py b/ainode/api/server.py index 736e7601..246f6416 100644 --- a/ainode/api/server.py +++ b/ainode/api/server.py @@ -83,6 +83,7 @@ ) from ainode.api.cluster_join import register_cluster_join_routes from ainode.api.decide import handle_decide +from ainode.api.systemone import handle_systemone from ainode.api.multipart import form_fields, is_multipart from ainode.bench.api_routes import register_bench_routes @@ -289,6 +290,11 @@ def create_app( # with the proxy's own `_routing_candidates` and the shared client session, # so a head still reaches the node serving the requested model. app.router.add_post("/v1/decide", handle_decide) + # The same decision core behind TypeSafe's System One wire format, so a client + # written for the hosted Jev endpoint (Titanium's JDE, jev-ultrafast, the + # TypeSafe SDK) answers off a model on this fleet with its endpoint changed and + # nothing else. A /v1 path, so the key and the rate limiter cover it. + app.router.add_post("/v1/systemone", handle_systemone) # Chat view: the per-instance model card + the capability probe. Registered # BEFORE the model routes because aiohttp resolves in registration order and diff --git a/ainode/api/server_routes.py b/ainode/api/server_routes.py index c062e38f..d41692fd 100644 --- a/ainode/api/server_routes.py +++ b/ainode/api/server_routes.py @@ -191,6 +191,7 @@ async def _probe_loaded_models( ], "ainode": [ {"method": "POST", "path": "/v1/decide", "description": "Typed questions in, calibrated probabilities out: every question answered at once by the node serving the model"}, + {"method": "POST", "path": "/v1/systemone", "description": "The same decisions in TypeSafe's System One (Jev) wire format, so a client written for the hosted endpoint answers off a model on this fleet"}, ], } diff --git a/ainode/api/systemone.py b/ainode/api/systemone.py new file mode 100644 index 00000000..de20553e --- /dev/null +++ b/ainode/api/systemone.py @@ -0,0 +1,506 @@ +"""``POST /v1/systemone``: TypeSafe's Jev wire format, answered by a local model. + +Why this route exists: a growing set of clients is written against TypeSafe's +hosted System One endpoint, not against ours. Titanium's JDE asks through +``jevJudge({endpoint, model})``, browser-use's jev-ultrafast, TypeSafe's own +Python SDK and the playground all post the same body to one path. This is that +path, on a node, so pointing any of them at a model on this fleet is one string +and no fork of the client. ONE adapter, so none of them needs an AINode-shaped +branch. + +What it does NOT promise: calibration. The hosted service's numbers are a +property of the model TypeSafe trained and of how they fit it; these are the +probabilities the served model put on the option labels, read off its logprobs +and renormalized, and nothing here corrects, tempers or rescales them. A local +model's confidence is worth what that model's confidence is worth, and the way +to find out is ``scripts/ainode-bench.py decide``, which scores exactly this +against labels. So: same wire, same client, same typed answers, and no claim at +all about how well those answers are spread. + +Two numbers on the way out are inferences rather than readings, and each says so +where it is computed: ``confidence`` is the hosted service's chance-corrected +formula (``normalized_confidence``), reproduced from its published examples +because no document states it, and the usage block falls back to the bench's own +estimate when an engine reports no usage at all (``usage_block``). The raw +distribution goes out beside them untouched, so a caller who disagrees with +either has the numbers they came from. + +One engine call answers one question, which is also what bounds a question: only +the top ``MAX_CRITERIA`` labels come back with a probability, so a wider option +set is refused with the cap named rather than answered with a distribution that +lost its tail. + +It is a translation layer and not a second decision endpoint. Resolving the +model, composing the grammar-constrained completions, the failover and reading +the distributions are ``api/decide.py``'s, called here through +``run_questions``; this module owns the shape on the wire in both directions and +nothing else. A question type gains a meaning by being translated here, never by +a second engine path. + +Latency, because a Jev client arrives with a hosted service's deadline in hand: +JDE's production ``timeout_ms`` is 750, which no chat model on this fleet will +meet. Every question is a full prefill of the state plus one constrained token, +so hundreds of milliseconds per question is the good case and a cold engine is +worse. A JDE user pointing at a node raises ``timeout_ms`` to the measured p95 +of THAT node rather than trusting the hosted default, and the questions of one +ask run concurrently, so the ask costs about one question plus the spread. + +Unknown fields on a question are ignored, never echoed: JDE strips its own +internal fields (``passingAnswer``, which names the answer it counts as good) +before the wire on purpose, and a field that arrives anyway is code's, not +something a model should read or something this route should hand back. +""" + +from __future__ import annotations + +import json +import math +import time +from typing import Any, NamedTuple, Optional + +from aiohttp import web + +from ainode.api.decide import ( + TOP_LOGPROBS, + DecideError, + candidates_for, + merge_usage, + normalize_questions, + resolve_model, + run_questions, + serialize_state, + unavailable, +) + +# The three question types the format defines. Anything else is a 422 rather than +# a guess: a client that asked for a kind of judgement this route does not have is +# better off being told which kinds it has. +CHOICE = "choice" +NOUL = "noul" +SCORE = "score" +QUESTION_TYPES = (CHOICE, NOUL, SCORE) + +# A noul's two options, in this order, always. The answer is P(true), so `true` +# has to be an option the engine can pick whether or not the caller described it, +# and its probability has to be readable without consulting the caller's key +# order. +NOUL_OPTIONS = ("true", "false") + +# A score is an ordered rubric. Two levels is the smallest thing that is still a +# degree rather than a yes or no, and ten is where the format stops. +MIN_SCORE_LEVELS = 2 +MAX_SCORE_LEVELS = 10 + +# The most criteria a question can carry here, which is NOT the format's ceiling +# of 255. One engine call reports probabilities for the top `TOP_LOGPROBS` tokens +# only, so option 21 comes back at 0 whatever the model thought of it. A caller +# with a wider taxonomy is told the cap rather than handed a distribution that +# quietly lost its tail (raising it means a second pass over the remaining labels, +# which is a measurement, not a constant). +MAX_CRITERIA = TOP_LOGPROBS + +# The rate the bench falls back to when it cannot measure a model's tokenizer +# (``bench/measure.py::calibrate_cpt``). Used for nothing but the usage block +# below, and only when the engine reported no usage at all. +CHARS_PER_TOKEN = 4.0 + + +class Translated(NamedTuple): + """One Jev question as the decision core sees it, plus the way back out. + + ``options`` is what the model reads, one line per option. ``names`` is what + each of those options answers to on the wire, in the same order: a choice + criteria key verbatim, ``true`` / ``false``, or a score level's position. + Keeping the pair here is what lets the answer name the caller's own key + rather than the letter the engine was constrained to. + """ + + kind: str + question: str + options: list[str] + names: list[str] + + +# ----------------------------------------------------------------- translate in + + +def option_text(name: str, description: Any) -> str: + """One option line: the name the answer will carry, then what it means. + + The name comes first and VERBATIM because it is the string the caller's + client compares against, and a model that has read it beside its description + is choosing between meanings rather than between labels. The description is + flattened to one line, because the prompt renders one option per line and a + description with a newline in it would read as two options. + """ + if isinstance(description, str) and description.strip(): + return f"{name}: {' '.join(description.split())}" + return name + + +def criteria_pairs(key: str, criteria: Any, kind: str) -> list[tuple[str, Any]]: + """The ``(name, description)`` pairs of an object ``criteria``, in order. + + Insertion order is the rubric order for a score, and JSON parsing preserves + it, so nothing here sorts. A name is validated stripped and kept as written: + the answer has to carry the caller's own key back, byte for byte, because the + caller's code looks that key up. + """ + if not isinstance(criteria, dict) or not criteria: + raise DecideError( + f"question '{key}': a {kind} question needs a non-empty 'criteria' " + "object of {name: description}") + pairs: list[tuple[str, Any]] = [] + for name, description in criteria.items(): + if not isinstance(name, str) or not name.strip(): + raise DecideError(f"question '{key}': every 'criteria' name must be a " + f"non-empty string (got {name!r})") + if description is not None and not isinstance(description, str): + raise DecideError(f"question '{key}': the 'criteria' description for " + f"'{name}' must be a string") + pairs.append((name.strip(), description)) + return pairs + + +def choice_options(key: str, criteria: Any) -> tuple[list[str], list[str]]: + """A choice question's options and the criteria keys they answer to.""" + pairs = criteria_pairs(key, criteria, "choice") + if len(pairs) < 2: + raise DecideError(f"question '{key}': a choice needs at least 2 'criteria' " + f"options, got {len(pairs)}") + if len(pairs) > MAX_CRITERIA: + raise DecideError( + f"question '{key}': {len(pairs)} 'criteria' options is more than the " + f"{MAX_CRITERIA} this node can report a probability for. One engine " + f"call carries back the top {TOP_LOGPROBS} labels, so a wider option " + "set would answer with a distribution missing its tail") + return ([option_text(name, desc) for name, desc in pairs], + [name for name, _ in pairs]) + + +def noul_options(key: str, criteria: Any) -> tuple[list[str], list[str]]: + """A noul's two options, always ``true`` then ``false``. + + The criteria block is optional here, and may describe one side only: some + clients send both, some send neither, and a yes-or-no question is still + answerable from its instructions alone. What a caller may not do is rename + the sides, because the answer is P(true) and nothing else can stand in for + it. + """ + described: dict[str, Any] = {} + if criteria is not None: + if not isinstance(criteria, dict): + raise DecideError(f"question '{key}': 'criteria' must be an object of " + "{true: description, false: description}") + for name, description in criteria.items(): + flat = name.strip() if isinstance(name, str) else name + if flat not in NOUL_OPTIONS: + raise DecideError(f"question '{key}': a noul's 'criteria' names only " + f"'true' and 'false' (got {name!r})") + if description is not None and not isinstance(description, str): + raise DecideError(f"question '{key}': the 'criteria' description for " + f"'{flat}' must be a string") + described[flat] = description + return ([option_text(name, described.get(name)) for name in NOUL_OPTIONS], + list(NOUL_OPTIONS)) + + +def score_options(key: str, criteria: Any) -> tuple[list[str], list[str]]: + """A score's levels in rubric order: a list by position, an object by insertion. + + Both spellings are accepted because both are in the wild: the list form names + the levels and nothing else, the object form names them and says what each + one means. Either way position 0 is the first level the caller wrote, which + is what the legend and the expected score are counted against. + """ + if isinstance(criteria, list): + names: list[str] = [] + for level in criteria: + if not isinstance(level, str) or not level.strip(): + raise DecideError(f"question '{key}': every 'criteria' level must be " + f"a non-empty string (got {level!r})") + names.append(level.strip()) + options = list(names) + elif isinstance(criteria, dict): + pairs = criteria_pairs(key, criteria, "score") + names = [name for name, _ in pairs] + options = [option_text(name, desc) for name, desc in pairs] + else: + raise DecideError( + f"question '{key}': a score question needs 'criteria', either an ordered " + "list of levels or an object of {level: description}") + if not MIN_SCORE_LEVELS <= len(names) <= MAX_SCORE_LEVELS: + raise DecideError( + f"question '{key}': a score's 'criteria' needs {MIN_SCORE_LEVELS} to " + f"{MAX_SCORE_LEVELS} ordered levels, got {len(names)}") + if len(set(names)) != len(names): + raise DecideError(f"question '{key}': 'criteria' repeats a level name. Every " + "level must be distinct so a score names one of them") + return options, names + + +def translate_one(key: str, spec: Any) -> Translated: + """One question off the wire. Reads three fields and ignores the rest. + + ``type``, ``instructions`` and ``criteria`` are the whole question as far as + this route is concerned, which is also exactly what JDE's + ``questionsForWire`` sends. A field beyond them belongs to the caller's own + code, so it is neither read nor echoed. + """ + if not isinstance(spec, dict): + raise DecideError(f"question '{key}' must be an object") + kind = spec.get("type") + if kind not in QUESTION_TYPES: + raise DecideError(f"question '{key}': 'type' must be one of " + f"{', '.join(QUESTION_TYPES)} (got {kind!r})") + instructions = spec.get("instructions") + if not isinstance(instructions, str) or not instructions.strip(): + raise DecideError(f"question '{key}' needs a non-empty 'instructions' string") + criteria = spec.get("criteria") + if kind == NOUL: + options, names = noul_options(key, criteria) + elif kind == SCORE: + options, names = score_options(key, criteria) + else: + options, names = choice_options(key, criteria) + return Translated(kind, instructions.strip(), options, names) + + +def translate_questions(raw: Any) -> dict[str, Translated]: + """Every question in the body, in the order the caller wrote them.""" + if not isinstance(raw, dict) or not raw: + raise DecideError("'questions' must be a non-empty object of {id: question}") + out: dict[str, Translated] = {} + for key, spec in raw.items(): + if not isinstance(key, str) or not key.strip(): + raise DecideError("every question id must be a non-empty string") + out[key] = translate_one(key, spec) + return out + + +def decide_questions(translated: dict[str, Translated]) -> dict[str, dict]: + """The ``/v1/decide`` question block for a translated set. + + Goes through the core's own ``normalize_questions`` rather than around it, so + the option ceiling, the two-option floor and distinct options are checked in + one place for both routes. + """ + return normalize_questions({key: {"question": item.question, + "options": item.options} + for key, item in translated.items()}) + + +# ---------------------------------------------------------------- translate out + + +def probability(value: Any) -> float: + """A finite probability inside [0, 1], because a foreign parser demands one. + + JDE's ``parseAnswers`` reads a confidence outside [0, 1] as 0, and a ``noul`` + outside it as a malformed answer SET, discarding every other answer in the + reply with it. Rounding is the only thing here that can land a hair outside, + and a NaN from an engine that reported one is the only thing that can land + outside the reals, but a whole judgement is too much to lose to either. + """ + try: + number = float(value) + except (TypeError, ValueError): + return 0.0 + if not math.isfinite(number): + return 0.0 + return min(1.0, max(0.0, number)) + + +def normalized_confidence(top: Any, options: int) -> float: + """``(n * p_max - 1) / (n - 1)``: the hosted service's `confidence`, chance corrected. + + This is NOT the picked option's probability, and getting that wrong would make + every band a JDE user already tuned read too high. On the hosted endpoint a + two-way question at 0.6 and a ten-way question at 0.6 do not report the same + confidence: the number is how far above chance the winner is, so 1/n reports 0 + and certainty reports 1, whatever n is. + + INFERRED, not specified: it reproduces every example in TypeSafe's published + docs and SDK types for both choice and score, and Kev's playground authors + arrived at the same formula for choice, but no document states it. The raw + distribution goes out untouched in ``probabilities`` beside it, so a caller + who disagrees with the formula has the numbers it came from. + """ + spread = probability(top) + if options < 2: + return spread + return probability((options * spread - 1.0) / (options - 1)) + + +def answer_from_decision(item: Translated, entry: dict) -> Optional[dict]: + """One Jev answer from one ``/v1/decide`` decision. Pure. + + The core reports probabilities against the option strings the model read, so + the first thing here is putting them back under the names the caller's client + expects: its own criteria key, ``true`` / ``false``, or a level's position. + + None when the engine named no option this route can read. The handler turns + that into the same 503 a failed call gets, because a client reading an answer + set it cannot parse reads the whole judgement as failed anyway, and a made-up + option would be worse than either. + + An engine that reported no logprobs answered under the grammar but offered no + spread. The answer stands and ``probabilities`` is left OFF the answer, which + the format allows, rather than filled with a distribution nobody measured. + """ + picked = dict(zip(item.options, item.names)).get(entry.get("answer")) + if picked is None: + return None + dist = entry.get("distribution") or {} + by_name = {name: probability(dist.get(option)) + for option, name in zip(item.options, item.names)} + # The core reports the picked option's own probability; the format wants that + # corrected for how many ways the question split. + confidence = round(normalized_confidence(entry.get("confidence"), + len(item.names)), 6) + + if item.kind == NOUL: + # P(true) is the answer. With no spread to read it is the engine's own + # pick, which is the one thing that is known. + return {"type": NOUL, + "noul": by_name["true"] if dist else float(picked == "true")} + + if item.kind == SCORE: + legend = {str(index): name for index, name in enumerate(item.names)} + if not dist: + return {"type": SCORE, "score": float(item.names.index(picked)), + "confidence": confidence, "legend": legend} + probabilities = {str(index): by_name[name] + for index, name in enumerate(item.names)} + # The expected level, not the argmax: a rubric is ordered, so a model + # split between 3 and 4 scores 3.5 and says more than either would. + score = sum(index * probabilities[str(index)] + for index in range(len(item.names))) + return {"type": SCORE, "score": round(score, 6), "confidence": confidence, + "legend": legend, "probabilities": probabilities} + + answer = {"type": CHOICE, "choice": picked, "confidence": confidence} + if dist: + answer["probabilities"] = by_name + return answer + + +def estimate_tokens(text: str) -> int: + """Tokens in ``text`` at the bench's fallback rate. An estimate, never a count.""" + if not text: + return 0 + return math.ceil(len(text) / CHARS_PER_TOKEN) + + +def usage_block(payloads: list[dict], state: str, + translated: dict[str, Translated], answered: int) -> dict: + """``{input_tokens, output_tokens}``: the engine's own numbers when it gives them. + + Every question is its own completion carrying the whole state, so the engine + reports the state once per question and this adds those up unchanged. That is + what the engines read, prefix cache or not. + + When the engine reports NO usage at all (a stub, an older build, a proxy that + strips the block) the numbers are an ESTIMATE at the same 4 characters per + token the bench falls back to when it cannot measure a tokenizer + (``bench/measure.py::calibrate_cpt``), over the serialized state plus every + question's own text, counted the same once-per-question way. The output side + is one label token per answered question, which is the whole output the + grammar allows. Nothing here is a tokenizer and nothing here pretends to be: + a caller reading these as exact should ask an engine that reports them. + """ + merged = merge_usage(payloads) + input_tokens = merged["prompt_tokens"] + output_tokens = merged["completion_tokens"] + if not input_tokens: + per_question = estimate_tokens(state) + input_tokens = len(translated) * per_question + sum( + estimate_tokens(item.question) + + sum(estimate_tokens(option) for option in item.options) + for item in translated.values()) + if not output_tokens: + output_tokens = answered + return {"input_tokens": int(input_tokens), "output_tokens": int(output_tokens)} + + +# -------------------------------------------------------------------- handler + + +def unprocessable(message: str) -> web.Response: + """422, which is what the Jev format answers a request it cannot read. + + The body is the ``{"error": {"message", "type"}}`` shape every other ``/v1`` + path on this node uses. JDE keeps an error body's ``error`` field only when + it is a string and otherwise quotes the body truncated, so the message names + the field it refused: that name is what reaches the caller either way. + """ + return web.json_response({"error": {"message": message, + "type": "invalid_request_error"}}, + status=422) + + +async def handle_systemone(request: web.Request) -> web.Response: + """POST /v1/systemone: typed questions in the Jev shape, typed answers back. + + A 200 carries an answer for every question asked, the same rule + ``/v1/decide`` holds: a client cannot tell a half-answered judgement from a + model that declined, so a question nobody could answer is a 503 with the + reason instead. + """ + started = time.monotonic() + raw = await request.read() + try: + body = json.loads(raw or b"{}") + except ValueError as exc: + return unprocessable(f"body is not valid JSON: {exc}") + if not isinstance(body, dict): + return unprocessable("body must be a JSON object") + + try: + model = resolve_model(request, body.get("model")) + translated = translate_questions(body.get("questions")) + questions = decide_questions(translated) + state = serialize_state(body.get("state")) + except DecideError as exc: + return unprocessable(str(exc)) + + # Tag the request so the server-view log middleware attributes it correctly. + try: + request["_log_model"] = model + except Exception: + pass + + candidates = candidates_for(request, model) + if not candidates: + return unavailable(f"no node is serving '{model}'") + + # No shared instructions block: in this format a question's own instructions + # are the whole prompt for it, and the questions of one ask still never see + # each other's answers. + run = await run_questions(request, model, questions, state, None, candidates) + + answers: dict[str, dict] = {} + failures = list(run.failures) + for key, entry in run.decisions.items(): + answer = answer_from_decision(translated[key], entry) + if answer is None: + failures.append(f"{key}: the engine named no option this route can read") + continue + answers[key] = answer + + collector = request.app.get("metrics_collector") + total_ms = (time.monotonic() - started) * 1000 + if failures: + if collector is not None: + collector.record_request(model, total_ms, error=True) + return unavailable(f"engine calls failed for '{model}': " + + "; ".join(failures[:5])) + + if collector is not None: + collector.record_request(model, total_ms, error=False) + return web.json_response({ + "model": model, + "answers": answers, + "usage": usage_block(run.payloads, state, translated, len(answers)), + "latency_ms": round(total_ms, 1), + }) diff --git a/tests/test_systemone.py b/tests/test_systemone.py new file mode 100644 index 00000000..d15aea4b --- /dev/null +++ b/tests/test_systemone.py @@ -0,0 +1,776 @@ +"""POST /v1/systemone: TypeSafe's Jev wire format over this node's decision core. + +What is pinned here: + +* **The translation, both ways.** A ``choice`` answers with the caller's own + criteria key, a ``noul`` answers with P(true), and a ``score`` answers with the + expected level plus a legend from position to level name. Every probability + block is keyed by exactly the names the caller wrote, because a foreign client + looks its own keys up in it. +* **That a real client can read the answer.** ``parse_answers_like_jde`` below is + JDE's own ``parseAnswers`` (``src/judge/jev.ts``) rewritten line for line: it + discards the WHOLE answer set over one malformed answer, so "our 200 parses" is + a property worth asserting rather than assuming, and it is asserted on every + live response in this file. +* **A malformed request is a 422 that names the field**, which is what the format + specifies and what a caller needs to fix it. +* **The same engine the decide tests drive.** ``FakeEngine`` is imported from + ``tests/test_decide.py`` rather than copied: the two routes share one core, so + they are proven against one engine's behaviour, and an engine quirk fixed for + one is fixed for both. + +The engine-facing tests run a REAL AINode app whose cluster state points at that +fake on a real port, so routing, the body the engine receives and the failure +paths are exercised rather than mocked out from underneath. +""" + +import json +import math +import os +import socket +from pathlib import Path + +import pytest +import pytest_asyncio +from aiohttp.test_utils import TestClient, TestServer + +from ainode.api.decide import DecideError +from ainode.api.server import create_app +from ainode.api.systemone import ( + CHARS_PER_TOKEN, + MAX_CRITERIA, + Translated, + answer_from_decision, + decide_questions, + estimate_tokens, + normalized_confidence, + translate_questions, + usage_block, +) +from ainode.core.config import NodeConfig +from tests.test_decide import MODEL, FakeEngine, _app, _free_port + +TICKET = ("Customer writes: the invoice PDF download 500s on every browser since " + "your Tuesday release. We bill 400 clients on Friday.") + +# One of each type, in the shape JDE's `questionsForWire` puts on the wire: the +# three fields a judge reasons from, and nothing else. +QUESTIONS = { + "queue": { + "type": "choice", + "instructions": "Which queue should this ticket go to?", + "criteria": { + "billing": "an invoice, a charge or a refund", + "bug": "the product did something it should not", + "other": "none of these fit", + }, + }, + "needs_human": { + "type": "noul", + "instructions": "Does this ticket need a human today?", + "criteria": {"true": "a person has to act on it today", + "false": "it can wait or answer itself"}, + }, + "severity": { + "type": "score", + "instructions": "How severe is it?", + "criteria": {"none": "cosmetic", "some": "a workaround exists", + "bad": "money or data is at risk"}, + }, +} + + +# ============================================================================= +# JDE's own reader, so "it parses" is checked and not assumed +# ============================================================================= + +def parse_answers_like_jde(value): + """JDE's ``parseAnswers`` (``src/judge/jev.ts``), rule for rule. + + A shape check and nothing more, and ONE bad answer discards the set: that is + the behaviour our 200 has to survive, so it is reproduced here rather than + approximated. Returns None for a reply JDE would call malformed. + """ + def finite01(number): + return (isinstance(number, (int, float)) and not isinstance(number, bool) + and math.isfinite(number) and 0.0 <= number <= 1.0) + + def probabilities(block): + if not isinstance(block, dict): + return None + return {key: raw for key, raw in block.items() + if isinstance(raw, (int, float)) and not isinstance(raw, bool) + and math.isfinite(raw)} + + if not isinstance(value, dict): + return None + out = {} + for key, answer in value.items(): + if not isinstance(answer, dict): + return None + kind = answer.get("type") + if kind == "noul": + if not finite01(answer.get("noul")): + return None + out[key] = {"type": "noul", "noul": answer["noul"]} + elif kind == "choice": + if not isinstance(answer.get("choice"), str): + return None + out[key] = {"type": "choice", "choice": answer["choice"], + "confidence": (answer["confidence"] + if finite01(answer.get("confidence")) else 0), + "probabilities": probabilities(answer.get("probabilities"))} + elif kind == "score": + score = answer.get("score") + if not (isinstance(score, (int, float)) and not isinstance(score, bool) + and math.isfinite(score)): + return None + out[key] = {"type": "score", "score": score, + "confidence": (answer["confidence"] + if finite01(answer.get("confidence")) else 0), + "legend": answer.get("legend"), + "probabilities": probabilities(answer.get("probabilities"))} + else: + return None + return out + + +def test_the_jde_reader_here_rejects_what_jde_rejects(): + """The guard above is only worth something if it refuses bad answers.""" + assert parse_answers_like_jde({"k": {"type": "noul", "noul": 0.5}}) is not None + assert parse_answers_like_jde({"k": {"type": "noul", "noul": 1.5}}) is None + assert parse_answers_like_jde({"k": {"type": "noul", "noul": "0.5"}}) is None + assert parse_answers_like_jde({"k": {"type": "choice"}}) is None + assert parse_answers_like_jde({"k": {"type": "guess", "choice": "a"}}) is None + # One bad answer discards the set, which is why a partial 200 is a 503 here. + assert parse_answers_like_jde({"good": {"type": "noul", "noul": 0.5}, + "bad": {"type": "score", "score": None}}) is None + + +# ============================================================================= +# Translating a question in +# ============================================================================= + +def _err(raw): + with pytest.raises(DecideError) as exc: + translate_questions(raw) + return str(exc.value) + + +def test_a_choice_keeps_its_criteria_keys_as_the_answer_names(): + item = translate_questions(QUESTIONS)["queue"] + assert item.kind == "choice" + assert item.names == ["billing", "bug", "other"] + # The model reads the key AND what it means, in that order. + assert item.options[0] == "billing: an invoice, a charge or a refund" + assert item.question == "Which queue should this ticket go to?" + + +def test_a_noul_is_two_options_with_true_first(): + item = translate_questions(QUESTIONS)["needs_human"] + assert item.kind == "noul" + assert item.names == ["true", "false"] + assert item.options[0].startswith("true: a person has to act") + + +def test_a_noul_takes_a_missing_or_partial_criteria_block(): + """Some clients send one side, some send none. The question still answers.""" + bare = translate_questions({"k": {"type": "noul", "instructions": "Done?"}})["k"] + assert bare.options == ["true", "false"] and bare.names == ["true", "false"] + half = translate_questions({"k": {"type": "noul", "instructions": "Done?", + "criteria": {"true": "it was done"}}})["k"] + assert half.options == ["true: it was done", "false"] + + +def test_a_noul_may_not_rename_its_sides(): + msg = _err({"k": {"type": "noul", "instructions": "Done?", + "criteria": {"yes": "it was", "no": "it was not"}}}) + assert "'criteria'" in msg and "'true' and 'false'" in msg + + +def test_a_score_takes_the_object_form_in_insertion_order(): + item = translate_questions(QUESTIONS)["severity"] + assert item.kind == "score" + assert item.names == ["none", "some", "bad"] + assert item.options[2] == "bad: money or data is at risk" + + +def test_a_score_takes_the_list_form_in_position_order(): + item = translate_questions({"k": {"type": "score", "instructions": "How bad?", + "criteria": ["fine", "poor", "awful"]}})["k"] + assert item.names == ["fine", "poor", "awful"] + assert item.options == ["fine", "poor", "awful"] + + +def test_a_rubric_outside_two_to_ten_levels_is_refused(): + assert "2 to 10 ordered levels" in _err( + {"k": {"type": "score", "instructions": "q", "criteria": ["only"]}}) + assert "2 to 10 ordered levels" in _err( + {"k": {"type": "score", "instructions": "q", + "criteria": [f"level{i}" for i in range(11)]}}) + + +def test_a_repeated_level_name_is_refused(): + msg = _err({"k": {"type": "score", "instructions": "q", + "criteria": ["same", "same"]}}) + assert "'criteria'" in msg and "repeats a level" in msg + + +def test_an_unknown_type_names_the_three_that_exist(): + msg = _err({"k": {"type": "vibe", "instructions": "q", "criteria": {"a": "b"}}}) + assert "'type'" in msg and "choice, noul, score" in msg + assert "'vibe'" in msg + + +def test_a_question_with_no_instructions_is_refused(): + assert "'instructions'" in _err({"k": {"type": "noul"}}) + assert "'instructions'" in _err({"k": {"type": "noul", "instructions": " "}}) + + +def test_a_choice_with_no_criteria_is_refused(): + msg = _err({"k": {"type": "choice", "instructions": "which?"}}) + assert "'criteria'" in msg + assert "at least 2 'criteria' options" in _err( + {"k": {"type": "choice", "instructions": "which?", "criteria": {"a": "only"}}}) + + +def test_a_criteria_set_wider_than_the_engine_can_report_is_refused(): + """20 labels come back with a probability, so a 21st option is not answerable. + + The format allows 255. This node cannot report them, and a distribution that + silently lost its tail would be worse than a refusal that names the cap. + """ + at_cap = {f"opt{i}": f"option {i}" for i in range(MAX_CRITERIA)} + assert len(translate_questions( + {"k": {"type": "choice", "instructions": "which?", + "criteria": at_cap}})["k"].names) == MAX_CRITERIA + over = dict(at_cap, one_too_many="the tail") + msg = _err({"k": {"type": "choice", "instructions": "which?", "criteria": over}}) + assert f"more than the {MAX_CRITERIA}" in msg + assert "'criteria'" in msg and "distribution missing its tail" in msg + + +def test_a_score_with_no_criteria_is_refused(): + assert "'criteria'" in _err({"k": {"type": "score", "instructions": "how bad?"}}) + + +def test_an_empty_or_missing_question_set_is_refused(): + assert "non-empty object" in _err(None) + assert "non-empty object" in _err({}) + assert "non-empty object" in _err([QUESTIONS]) + + +def test_a_field_the_format_does_not_define_is_ignored_and_never_echoed(): + """JDE strips `passingAnswer` on purpose; one that arrives anyway is code's.""" + item = translate_questions({"k": {"type": "noul", "instructions": "Done?", + "criteria": {"true": "yes", "false": "no"}, + "passingAnswer": "false"}})["k"] + assert item == Translated("noul", "Done?", ["true: yes", "false: no"], + ["true", "false"]) + + +def test_the_translated_set_goes_through_the_shared_normalizer(): + """One guard for both routes: the ceiling, the floor and distinct options.""" + questions = decide_questions(translate_questions(QUESTIONS)) + assert list(questions) == ["queue", "needs_human", "severity"] + assert questions["queue"]["question"] == "Which queue should this ticket go to?" + assert questions["needs_human"]["options"][0].startswith("true: ") + + +def test_question_order_is_the_order_the_caller_wrote(): + assert list(translate_questions(QUESTIONS)) == ["queue", "needs_human", + "severity"] + + +# ============================================================================= +# Translating an answer out +# ============================================================================= + +def _entry(answer, distribution=None, confidence=None): + """A ``/v1/decide`` decision entry, the shape the core hands over.""" + return {"answer": answer, "confidence": confidence, + "distribution": distribution, "latency_ms": 12.3} + + +CHOICE_ITEM = Translated("choice", "Which queue?", + ["billing: money", "bug: broken", "other: neither"], + ["billing", "bug", "other"]) +NOUL_ITEM = Translated("noul", "Human?", ["true: yes", "false: no"], + ["true", "false"]) +SCORE_ITEM = Translated("score", "How bad?", ["none: cosmetic", "some: workaround", + "bad: money at risk"], + ["none", "some", "bad"]) + + +def test_a_choice_answers_with_the_caller_s_key_and_its_own_probabilities(): + entry = _entry("bug: broken", {"billing: money": 0.1, "bug: broken": 0.7, + "other: neither": 0.2}, 0.7) + answer = answer_from_decision(CHOICE_ITEM, entry) + # Confidence is chance corrected over three options: (3 * 0.7 - 1) / 2. + assert answer == {"type": "choice", "choice": "bug", "confidence": 0.55, + "probabilities": {"billing": 0.1, "bug": 0.7, "other": 0.2}} + # Exactly the criteria keys, nothing added and nothing left out. + assert set(answer["probabilities"]) == set(CHOICE_ITEM.names) + assert math.isclose(sum(answer["probabilities"].values()), 1.0, abs_tol=1e-6) + + +def test_a_noul_is_the_probability_of_the_true_option(): + entry = _entry("true: yes", {"true: yes": 0.82, "false: no": 0.18}, 0.82) + assert answer_from_decision(NOUL_ITEM, entry) == {"type": "noul", "noul": 0.82} + # A confident no is a low noul, not a high confidence: the number is P(true) + # whichever way the model went. + entry = _entry("false: no", {"true: yes": 0.04, "false: no": 0.96}, 0.96) + assert answer_from_decision(NOUL_ITEM, entry) == {"type": "noul", "noul": 0.04} + + +def test_a_score_is_the_expected_level_with_a_legend_by_position(): + entry = _entry("some: workaround", {"none: cosmetic": 0.0, + "some: workaround": 0.5, + "bad: money at risk": 0.5}, 0.5) + answer = answer_from_decision(SCORE_ITEM, entry) + assert answer["type"] == "score" + # Split between level 1 and level 2, so 1.5: the expected level, not the pick. + assert answer["score"] == 1.5 + # Two levels at 0.5 over three levels: (3 * 0.5 - 1) / 2, not 0.5. + assert answer["confidence"] == 0.25 + assert answer["legend"] == {"0": "none", "1": "some", "2": "bad"} + assert answer["probabilities"] == {"0": 0.0, "1": 0.5, "2": 0.5} + assert math.isclose(sum(answer["probabilities"].values()), 1.0, abs_tol=1e-6) + + +def test_a_score_on_one_certain_level_is_that_level(): + entry = _entry("bad: money at risk", {"none: cosmetic": 0.0, + "some: workaround": 0.0, + "bad: money at risk": 1.0}, 1.0) + assert answer_from_decision(SCORE_ITEM, entry)["score"] == 2.0 + + +def test_confidence_is_chance_corrected_and_not_the_picked_probability(): + """The hosted service's own formula: 1/n reports 0 and certainty reports 1. + + A two-way question at 0.6 and a ten-way question at 0.6 are not the same + judgement, and a JDE user's bands are tuned against the hosted numbers. + """ + assert normalized_confidence(1.0, 4) == 1.0 + assert normalized_confidence(0.25, 4) == 0.0 # uniform over four: no signal + assert normalized_confidence(0.7, 4) == 0.6 # (4 * 0.7 - 1) / 3 + assert normalized_confidence(0.6, 2) == pytest.approx(0.2) + assert normalized_confidence(0.6, 10) == pytest.approx(0.5555555, abs=1e-6) + # Below chance cannot happen off an argmax, and is floored rather than negative. + assert normalized_confidence(0.1, 10) == 0.0 + + +def test_an_engine_with_no_logprobs_answers_without_inventing_a_spread(): + """The core's no-distribution fallback: an answer, and no probabilities.""" + choice = answer_from_decision(CHOICE_ITEM, _entry("other: neither", + None, 1.0)) + assert choice == {"type": "choice", "choice": "other", "confidence": 1.0} + assert "probabilities" not in choice + score = answer_from_decision(SCORE_ITEM, _entry("some: workaround", None, 1.0)) + assert score["score"] == 1.0 and "probabilities" not in score + assert score["legend"] == {"0": "none", "1": "some", "2": "bad"} + assert answer_from_decision(NOUL_ITEM, _entry("true: yes", None, 1.0)) == { + "type": "noul", "noul": 1.0} + assert answer_from_decision(NOUL_ITEM, _entry("false: no", None, 1.0)) == { + "type": "noul", "noul": 0.0} + + +def test_an_answer_the_route_cannot_read_is_no_answer(): + """None, so the handler can refuse the request instead of guessing a key.""" + assert answer_from_decision(CHOICE_ITEM, _entry(None)) is None + assert answer_from_decision(NOUL_ITEM, _entry("maybe")) is None + + +def test_a_probability_outside_the_reals_cannot_poison_the_answer_set(): + """One NaN would make JDE discard every answer in the reply, not just this one.""" + entry = _entry("true: yes", {"true: yes": float("nan"), "false: no": 0.5}, 1.5) + answer = answer_from_decision(NOUL_ITEM, entry) + assert answer["noul"] == 0.0 + assert parse_answers_like_jde({"k": answer}) is not None + + +# ============================================================================= +# Usage +# ============================================================================= + +def test_usage_is_the_engine_s_own_count_when_it_reports_one(): + payloads = [{"usage": {"prompt_tokens": 120, "completion_tokens": 2}}, + {"usage": {"prompt_tokens": 118, "completion_tokens": 2}}] + assert usage_block(payloads, TICKET, translate_questions(QUESTIONS), 2) == { + "input_tokens": 238, "output_tokens": 4} + + +def test_usage_falls_back_to_the_bench_s_own_estimate(): + """No usage block at all: an estimate at 4 chars per token, documented as one.""" + translated = translate_questions(QUESTIONS) + usage = usage_block([{}, {}, {}], TICKET, translated, 3) + state_tokens = estimate_tokens(TICKET) + assert state_tokens == math.ceil(len(TICKET) / CHARS_PER_TOKEN) + # The state is read once per question, which is what the engines really do. + assert usage["input_tokens"] > len(translated) * state_tokens + # One constrained label per question is the whole output the grammar allows. + assert usage["output_tokens"] == 3 + assert isinstance(usage["input_tokens"], int) + + +# ============================================================================= +# The live route +# ============================================================================= + +@pytest.fixture +def engine_fake(): + return FakeEngine() + + +@pytest_asyncio.fixture +async def engine(engine_fake): + server = TestServer(engine_fake.app()) + await server.start_server() + try: + yield server + finally: + await server.close() + + +@pytest_asyncio.fixture +async def client(engine): + """A real AINode app that believes a peer serves MODEL on the fake's port.""" + async with TestClient(TestServer(_app(engine.port))) as c: + yield c + + +def _body(**over): + body = {"model": MODEL, "state": TICKET, "questions": QUESTIONS} + body.update(over) + return body + + +@pytest.mark.asyncio +async def test_every_question_type_round_trips(client, engine_fake): + resp = await client.post("/v1/systemone", json=_body()) + assert resp.status == 200 + data = await resp.json() + assert data["model"] == MODEL + assert data["latency_ms"] > 0 + assert set(data) == {"model", "answers", "usage", "latency_ms"} + assert list(data["answers"]) == ["queue", "needs_human", "severity"] + assert parse_answers_like_jde(data["answers"]) is not None + + # The fake always picks the first option, which is the first criteria key. + queue = data["answers"]["queue"] + assert queue == {"type": "choice", "choice": "billing", + "confidence": queue["confidence"], + "probabilities": queue["probabilities"]} + assert set(queue["probabilities"]) == {"billing", "bug", "other"} + assert queue["confidence"] > 0.9 + + needs_human = data["answers"]["needs_human"] + assert needs_human["type"] == "noul" and needs_human["noul"] > 0.9 + + severity = data["answers"]["severity"] + assert severity["legend"] == {"0": "none", "1": "some", "2": "bad"} + assert set(severity["probabilities"]) == {"0", "1", "2"} + assert 0.0 <= severity["score"] < 0.1 # nearly all the mass on level 0 + + # A noul's whole answer is P(true), so it carries no probabilities block; the + # other two spread over their own names and the spread is a distribution. + assert "probabilities" not in needs_human + for answer in (queue, severity): + block = answer["probabilities"] + assert math.isclose(sum(block.values()), 1.0, abs_tol=1e-4) + assert all(math.isfinite(p) and 0.0 <= p <= 1.0 for p in block.values()) + + +@pytest.mark.asyncio +async def test_the_answers_come_from_one_constrained_call_per_question(client, + engine_fake): + await client.post("/v1/systemone", json=_body()) + assert len(engine_fake.seen) == 3 + # In flight together, the way /v1/decide asks: one ask is about one question + # plus the spread, not three questions end to end. + assert engine_fake.max_in_flight == 3 + prefixes = set() + for body in engine_fake.seen: + assert body["model"] == MODEL + assert body["structured_outputs"]["choice"][0] == "A" + assert body["logprobs"] is True + prefixes.add(body["messages"][1]["content"].split("QUESTION:")[0]) + # Every question shares the state's prefix, so the engine prefills it once. + assert len(prefixes) == 1 + # A question's options reach the model as the caller's keys plus their meaning. + options = "\n".join(body["messages"][1]["content"] for body in engine_fake.seen) + assert "A. billing: an invoice, a charge or a refund" in options + assert "A. true: a person has to act on it today" in options + assert "C. bad: money or data is at risk" in options + + +@pytest.mark.asyncio +async def test_usage_and_a_json_state_come_back_in_the_format_s_own_keys(client, + engine_fake): + resp = await client.post("/v1/systemone", json=_body( + state={"subject": "invoice 500s", "plan": "pro"}, + questions={"k": QUESTIONS["needs_human"]})) + data = await resp.json() + assert data["usage"] == {"input_tokens": 100, "output_tokens": 2} + user = engine_fake.seen[0]["messages"][1]["content"] + assert '{"plan":"pro","subject":"invoice 500s"}' in user + + +@pytest.mark.asyncio +async def test_the_model_defaults_the_way_decide_defaults_it(client): + resp = await client.post("/v1/systemone", json={ + "state": TICKET, "questions": {"k": QUESTIONS["needs_human"]}}) + assert resp.status == 200 + assert (await resp.json())["model"] == MODEL + + +@pytest.mark.asyncio +@pytest.mark.parametrize("body,fragment", [ + ({"state": "s"}, "'questions' must be a non-empty object"), + ({"questions": {}}, "'questions' must be a non-empty object"), + ({"questions": {"k": {"instructions": "q", "criteria": {"a": "b", "c": "d"}}}}, + "'type' must be one of"), + ({"questions": {"k": {"type": "guess", "instructions": "q"}}}, + "'type' must be one of"), + ({"questions": {"k": {"type": "choice", "criteria": {"a": "b", "c": "d"}}}}, + "needs a non-empty 'instructions' string"), + ({"questions": {"k": {"type": "choice", "instructions": "q"}}}, + "'criteria'"), + ({"questions": {"k": {"type": "choice", "instructions": "q", + "criteria": {"only": "one"}}}}, + "at least 2 'criteria' options"), + ({"questions": {"k": {"type": "choice", "instructions": "q", + "criteria": {f"o{i}": "d" for i in range(21)}}}}, + "more than the 20 this node can report a probability for"), + ({"questions": {"k": {"type": "score", "instructions": "q"}}}, "'criteria'"), + ({"questions": {"k": {"type": "score", "instructions": "q", + "criteria": ["one"]}}}, + "2 to 10 ordered levels"), + ({"questions": {"k": {"type": "noul", "instructions": "q", + "criteria": {"yes": "a", "no": "b"}}}}, + "'true' and 'false'"), + ({"questions": {"k": {"type": "noul", "instructions": "q", "criteria": 7}}}, + "'criteria' must be an object"), + ({"model": "", "questions": {"k": {"type": "noul", "instructions": "q"}}}, + "'model' must be a non-empty string"), +]) +async def test_a_malformed_request_is_a_422_that_names_the_field(client, engine_fake, + body, fragment): + resp = await client.post("/v1/systemone", json=body) + assert resp.status == 422 + payload = await resp.json() + assert fragment in payload["error"]["message"] + assert payload["error"]["type"] == "invalid_request_error" + assert engine_fake.seen == [] # nothing reached the engine + + +@pytest.mark.asyncio +async def test_a_body_that_is_not_json_is_a_422(client): + resp = await client.post("/v1/systemone", data=b"{nope", + headers={"Content-Type": "application/json"}) + assert resp.status == 422 + assert "not valid JSON" in (await resp.json())["error"]["message"] + resp = await client.post("/v1/systemone", json=["a", "list"]) + assert resp.status == 422 + assert "must be a JSON object" in (await resp.json())["error"]["message"] + + +@pytest.mark.asyncio +async def test_a_model_no_node_serves_is_a_503(client, engine_fake): + """A client that left the hosted default in place is told which id failed.""" + resp = await client.post("/v1/systemone", json=_body(model="jev-latest")) + assert resp.status == 503 + body = await resp.json() + assert "no node is serving 'jev-latest'" in body["error"]["message"] + assert body["error"]["type"] == "service_unavailable" + assert engine_fake.seen == [] + + +@pytest.mark.asyncio +async def test_an_unreachable_engine_is_a_503_and_never_half_an_answer_set(): + dead = _free_port() + async with TestClient(TestServer(_app(dead))) as c: + resp = await c.post("/v1/systemone", json=_body()) + assert resp.status == 503 + msg = (await resp.json())["error"]["message"] + assert "engine calls failed" in msg and "queue" in msg + + +@pytest.mark.asyncio +async def test_an_engine_with_no_logprobs_still_answers_every_question(): + quiet = FakeEngine(with_logprobs=False) + server = TestServer(quiet.app()) + await server.start_server() + try: + async with TestClient(TestServer(_app(server.port))) as c: + resp = await c.post("/v1/systemone", json=_body()) + assert resp.status == 200 + answers = (await resp.json())["answers"] + assert parse_answers_like_jde(answers) is not None + assert answers["needs_human"] == {"type": "noul", "noul": 1.0} + assert answers["queue"]["choice"] == "billing" + assert "probabilities" not in answers["queue"] + assert answers["severity"]["score"] == 0.0 + finally: + await server.close() + + +@pytest.mark.asyncio +async def test_systemone_is_in_the_endpoint_catalog(client): + catalog = await (await client.get("/api/server/endpoints")).json() + paths = [ep["path"] for group in catalog.values() for ep in group] + assert "/v1/systemone" in paths + + +# ============================================================================= +# Auth: it is a /v1 path, so it wants a key +# ============================================================================= + +@pytest.fixture +def auth_home(tmp_path, monkeypatch): + """Keep every file this app writes out of the operator's own ~/.ainode. + + Same redirect ``tests/test_auth_gate.py`` uses: most paths resolve + AINODE_HOME at call time, but AUTH_FILE, CONFIG_FILE and SECRETS_FILE are + computed at import. + """ + monkeypatch.setenv("AINODE_HOME", str(tmp_path)) + monkeypatch.setattr("ainode.core.config.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.core.config.CONFIG_FILE", tmp_path / "config.json") + monkeypatch.setattr("ainode.auth.middleware.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.auth.middleware.AUTH_FILE", tmp_path / "auth.json") + monkeypatch.setattr("ainode.secrets.manager.AINODE_HOME", tmp_path) + monkeypatch.setattr("ainode.secrets.manager.SECRETS_FILE", + tmp_path / "secrets.json") + return tmp_path + + +@pytest_asyncio.fixture +async def keyed(auth_home): + """Auth on, plus the plaintext key, which exists only at this moment.""" + with socket.socket() as s: + s.bind(("127.0.0.1", 0)) + free_port = s.getsockname()[1] + app = create_app(config=NodeConfig(node_id="auth-node", node_name="AuthNode", + api_port=free_port, onboarded=True), + engine=None) + entry = app["auth_config"].enable() + async with TestClient(TestServer(app)) as c: + yield c, entry["key"] + + +@pytest.mark.asyncio +async def test_the_route_is_refused_without_a_key_when_auth_is_on(keyed): + client, key = keyed + body = {"state": TICKET, "questions": {"k": QUESTIONS["needs_human"]}} + resp = await client.post("/v1/systemone", json=body) + assert resp.status == 401 + assert (await resp.json())["error"]["type"] == "auth_error" + + resp = await client.post("/v1/systemone", json=body, + headers={"Authorization": "Bearer not-the-key"}) + assert resp.status == 401 + + # With the key it reaches the handler, which on this node has no engine to + # ask: past the gate is the whole assertion, and a 401 is not. + resp = await client.post("/v1/systemone", json=body, + headers={"Authorization": f"Bearer {key}"}) + assert resp.status != 401 + + +# ============================================================================= +# One real JDE case, replayed +# ============================================================================= + +# JDE's blind case set, read where it lives and never copied into this repo: the +# cases are someone else's measurement data, and a copy here would be a second +# version of it. Absent (CI, another machine) means this test skips; the shapes +# it checks are covered above without it. +JDE_CASES = Path(os.environ.get( + "JDE_COMPLETION_CASES", + "/Users/sem/code/jde/cases/completion-check-blind.json")) + +# The wording is JDE's, from src/decisions/completion-check.ts, copied because +# that file says it is the artefact: rewording a clause invalidates the +# measurement behind it. `passingAnswer` is deliberately NOT here, because +# questionsForWire strips it before the wire and a judge must never see it. +RESULT_IS_ECHO_QUESTION = { + "type": "noul", + "instructions": "Is `claimed_result` a restatement of `task` rather than a report of an outcome?", + "criteria": { + "true": "`claimed_result` repeats the task's own words or its instructions back, with no outcome of its own", + "false": "`claimed_result` reports what happened, what was produced, or what was found", + }, +} + + +def _part_question(part): + """JDE's ``partQuestion``: a reply part is judged on the claim, an action on receipts.""" + if part["kind"] == "reply": + return { + "type": "noul", + "instructions": f"Does `claimed_result` contain {part['text']}, stated as an outcome rather than a plan?", + "criteria": { + "true": "`claimed_result` carries that content itself, written as something already produced or found", + "false": "`claimed_result` does not carry it, or only says it will be produced", + }, + } + return { + "type": "noul", + "instructions": f"Do `receipts` show that this part was carried out: {part['text']}? Count a file written, a search run, a page fetched, or a tool call that produces it; do not count `claimed_result` saying so.", + "criteria": { + "true": "`receipts` carry a file, a search, a fetched page or a tool call that carries out this part", + "false": "nothing in `receipts` carries out this part, whatever `claimed_result` says about it", + }, + } + + +def _completion_questions(parts): + """JDE's ``completionQuestions``: the echo question, then the parts in order. + + A file part is code's own fact (the path was written, and not empty), so it + never becomes a question. That is why the ids below skip an index. + """ + questions = {"result_is_echo": RESULT_IS_ECHO_QUESTION} + for index, part in enumerate(parts): + if part["kind"] == "file": + continue + questions[f"part_{index}_done"] = _part_question(part) + return questions + + +@pytest.mark.asyncio +@pytest.mark.skipif(not JDE_CASES.is_file(), + reason=f"JDE's case set is not on this machine ({JDE_CASES})") +async def test_a_real_jde_completion_check_case_round_trips(client): + """The question set a JDE completion check really asks, answered by this route. + + Built the way ``completionCheck`` builds it and read the way ``parseAnswers`` + reads it, so what is proven is that an unmodified JDE pointed at a node gets + an answer set its own reader accepts. What the model SAYS is the fake + engine's, so nothing here asserts a verdict: that is the decision bench's + job, against labels. + """ + cases = json.loads(JDE_CASES.read_text()) + case = next(c for c in cases if c["id"] == "heldout5-001") + state = case["state"] + questions = _completion_questions(state["task_parts"]) + # Three parts, one of them a file: two part questions plus the echo question. + assert list(questions) == ["result_is_echo", "part_0_done", "part_2_done"] + + resp = await client.post("/v1/systemone", + json={"model": MODEL, "state": state, + "questions": questions}) + assert resp.status == 200 + data = await resp.json() + + parsed = parse_answers_like_jde(data["answers"]) + assert parsed is not None, "JDE would call this answer set malformed" + assert list(parsed) == list(questions) + for answer in parsed.values(): + # completionCheck reads every one of these as a noul and floors it at 0.7. + assert answer["type"] == "noul" + assert math.isfinite(answer["noul"]) and 0.0 <= answer["noul"] <= 1.0 + assert data["usage"]["input_tokens"] > 0 + assert isinstance(data["usage"]["output_tokens"], int) + assert data["model"] == MODEL