Apply decision adapters' temperatures and warm their grammar on bind - #281
Merged
Merged
Conversation
/v1/decide and /v1/systemone now read temperatures.json from the served
model's directory in this node's store (resolved through
registry.snapshot_dir_for) and divide each question's label logprobs by
its kind's temperature before the softmax, in run_questions so both
routes get it. Both responses carry a top-level calibration block
{applied, temperatures}; "calibration": "raw" opts out.
A model whose directory carries prompt_contract.json or
temperatures.json is sent one minimal constrained question per kind
(choice, noul, score) through ask_one as soon as its engine binds,
from _wait_for_bind, in the background. Each compile time is logged
and /api/status gains an instances list with warm per instance. The
engine-call timeout goes from 180 s to 300 s, and a connected call
that times out now reads as the grammar compiling, retry, rather than
as an unreachable node.
Closes #276
Closes #277
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two decision-model fixes, one PR, both in the shared decision core (
api/decide.py), so/v1/decideand/v1/systemoneget them together.#276: the adapter's per-type temperatures.
run_questionsnow finds the served model's directory in this node's store throughmodels/registry.py::snapshot_dir_for(a model served from an absolute path is its own directory). If that directory carriestemperatures.json, each question's label logprobs are divided by its kind's temperature before the softmax, sodistribution,confidence, systemone'sprobabilities,noul, the expectedscoreand the chance-corrected confidence all come from the tempered spread. Kinds arechoice,noulandscore. On/v1/decide,type: "boolean"maps tonoul,type: "score"toscoreand anoptionslist tochoice. Both responses carry a top-level"calibration": {"applied": bool, "temperatures": {...} | null}. A request can send"calibration": "raw"to opt out. Any other value is a 400 on decide and a 422 on systemone. A missing kind, or a value that is not a finite positive number, leaves that kind at the engine's own spread.#277: warm on load. Every launch path (boot primary, replay,
/api/modelsload, set-model, relaunch) goes throughmodels/api_routes.py::_wait_for_bind. When it reports a bind,decide.py::schedule_decision_warmupchecks the model's directory forprompt_contract.jsonortemperatures.json. If either is there, it sends one minimal constrained question per kind (a two-option choice, a noul, a two-level score) in the background, using the samebuild_chat_bodyandask_onethe routes use and a 600 s timeout. It logs the compile time for each kind. An adopted boot primary gets the same warm-up, because the flag does not survive a restart./api/statusgains aninstanceslist where each row carrieswarm(true, false while warming or after a failed warm-up, null for a model with nothing to warm) andwarm_compile_seconds. Readiness is not held back for the warm-up. The route's engine-call timeout goes from 180 s to 300 s. If a call connects and then times out, the 503 now saysthe engine is compiling the answer grammar, retryinstead ofunreachable. A connect timeout still reads as unreachable.AGENTS.md: the systemone rule "nothing on this path rescales, tempers or corrects" is updated to allow this one exception, and the warm-up hook is recorded as the single place a bound engine gets warmed. README: the
/v1/decidesection documents the calibration block, the raw opt-out and the warm flag.Closes #276
Closes #277
How it was tested
pytest tests/: 3199 passed, 2 skipped, 1 xfailed. main was 3177 passed.ruff checkon every touched Python file: clean.tests/test_decide_calibration.py(new): the temperature math against hand-computed sigmoids, the kind mapping, readingtemperatures.json(bad entries, bad JSON, a prompt contract alone), and the calibration block. Both routes are also run end to end against the fake vLLM with logprobs, with this node's store pointed at a temp directory: temperatures on,rawopt-out, no file, and a refused value.tests/test_decide_warmup.py(new): a fake engine records the three warm-up requests (constrained toA/B, logprobs on, thinking off, the noul as true/false). A bind through_wait_for_bindflipswarmfrom false to true. A failed warm-up stays cold and names the kind that failed. A non-decision model is left alone and its stale entry is cleared. The warm-up asks by--served-model-name. A slow engine produces the "compiling" message./api/statusreportswarmper instance.Changelog
Added
/v1/decideand/v1/systemoneapply a decision adapter's own temperatures (/v1/systemone and /v1/decide should apply the adapter's per-type temperatures #276). When the served model's directory carriestemperatures.json, each question's label logprobs are divided by the temperature fitted for its kind (choice,noul,score) before the softmax. Both responses reportcalibration: {applied, temperatures}, and"calibration": "raw"returns the engine's own spread./api/statusgainsinstances[]withwarmandwarm_compile_secondsper instance.Changed
🤖 Generated with Claude Code