Repository navigation
Qwen3 thinking in the agent loop; server-like Sonos eval fixture and scoring - #2
Merged
Merged
Conversation
…tep token and time accounting split_thinking and parse_output drop a leading <think> block before reading tool calls; a reply that ends inside the block is an error. Thinking::Off appends the empty block Qwen3's template adds for enable_thinking=false. set_think_cap closes a block that reached the cap and continues, on the engine's normal text-in/text-out generate. Step.timing carries prompt, thinking and answer tokens, total time and time to first piece, summed over the step's attempts. Generator gains a defaulted count_tokens.
…tep timing in the eval report, flag-driven agent_eval
…tep and Sonos sets
…ase in the report The table only carried the scorer's reason, so a failure could not be told apart as a wrong call, a correct call the rule rejects, or a text answer. CaseResult keeps the final text, and the markdown report ends with each failed case's calls and answer.
…r sets) and the calls of the nine always-failing Sonos cases
…rs get_now_playing per group FixtureCaller now returns an error result for a group or player id that the fixture household does not list, as a server would, instead of a canned success on the kitchen group. get_now_playing answers per group, matching the playbackState of each group in the household listing: kitchen paused on the Radiohead track, living room playing, bedroom idle.
…s and unquantified nudges m27 names a speaker the household does not have; like m21 it now expects no mutating call instead of a play on the kitchen group. s14 also passes on the household lookup plus an answer that the bedroom is idle, which the listing already shows. m22 also passes on no mutating call plus a question back to the user. s16, m23 and m26 ask for a volume change with no number; any volume_delta of the right sign from 1 to 20 is accepted. Scoring gains two case fields for this: arg_ranges on an expected call (a numeric argument accepted within an inclusive range) and alternatives (other expected calls, acceptance mode and answer checks that also pass the case).
…wen3-1.7B, thinking on, default, clear, clear cap 512)
…unt as a mutation In no_mutation cases, only mutating calls whose tool result was not an error count. A run that tries an unknown group or player id, gets the server's error and then answers correctly now passes. run_case records which calls were rejected and scores through the new score_run.
…s (Qwen3-1.7B, thinking on, default, clear, clear cap 512)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Qwen3 thinking support to the agent loop, and makes the Sonos eval fixture and scoring closer to a real server.
<think>blocks, adds a thinking-off switch and a thinking token cap, and records per-step token and time accounting. agent-wasm exposessetThinkingOffandsetThinkCap.exact_argsscoring and answer checks, plus a two-step tool-chaining fixture set.get_now_playingper group.no_mutationcases, a mutating call the server rejected with an error does not count as a mutation: trying an unknown id, getting the error and then answering correctly passes.docs/runs/2026-10-15-qwen3-thinking-lean.md: Qwen3-1.7B Q8_0 with thinking on and off (and 0.6B with thinking off), native WebGPU. With the final fixture and scoring, Sonos scores 23/33 (default prompt), 26/33 (clear prompt) and 28/33 (clear prompt, think cap 512).