Add Qwen TITO compatibility and verbatim Anthropic capture - #160
Merged
Conversation
The Anthropic-native lane only ever recorded OTel GenAI spans — (role, content-flattened-to-text) plus token counts. Nothing downstream could reconstruct a call from that: tool schemas were gone, `thinking` signatures were gone, a `tool_use` block was a printf'd string, and a streamed response landed as an opaque SSE blob. Worse, nothing said whether the context had been rewritten between two calls, so consumers were reduced to parsing a harness's own compaction boundary records. New `agentix.bridge.capture` defines `abridge.record.v1` and `Recorder` writes it: * Complete request body at the verbatim level — `system`, `tools` with full schemas (names alone hide a schema change), and the whole message history exactly as sent, deep-copied at entry so a handler that rewrites the body in place can't be mistaken for the agent. * Complete structured response, including `thinking` blocks whose text is empty and whose opaque `signature` is the only handle on the content. The bundled Anthropic-face clients publish the completed Message on a context var, so a streaming call records the object, not the SSE bytes. A body nobody decoded keeps its bytes and says so. * `prefix` — whether this request's message list extends the previous one. `stable: false` means the context was rewritten, which subsumes compaction detection. Modeled on the token gateway's `prefix_stable`, over per-message digests since this layer has no tokenizer. Requests are attributed to lanes keyed by the canonicalized system prompt (one key multiplexes subagents and helper calls); collisions and evictions fail toward "rewritten", never toward a bad splice. Capture is a layered, opt-in ladder — `off` / `metadata` (default: join keys, tool names, digests, block skeleton, prefix; no conversation text) / `verbatim`. `--capture-level` on `agentix-bridge-serve`; turning `--record-dir` on never turns verbatim on. Security: the tunnel carries no HTTP metadata, so no credential can reach a record by construction (asserted end to end); files are created 0600. A secret typed into a prompt is conversation content and is gated by the level, not silently redacted. Also: the native `AnthropicClient` now reuses the capture layer's request id instead of minting its own, so its upstream `x-request-id` actually matches the row that recorded the call; `_canonical_text` moves to `capture.canonical_text` and `serve.py` reuses it. Shape sharing with the token gateway's record is deliberate at the envelope (schema_version, turn_index + gap semantics, session trailer, strict JSON, log-and-serve, prefix meaning) and deliberately absent at the payload — one owns a tokenizer, the other does not.
Two gaps surfaced wiring the gateway behind Pi for Qwen3.8-27B rollouts: - Qwen3.5/Qwen3.8 share one chat-template family that raises 'No user query found in messages' when the conversation has no real user turn, so the engine's synthetic contexts (dummy system + dummy assistant) could not be rendered at all; the family also rejects mid-conversation system messages, reads reasoning from reasoning_content only, and uses the qwen3_coder tool dialect. Qwen3_5TITOTokenizer carries a dummy user turn in its synthetic base, inherits the Qwen3 <|im_end|> newline fixup, and is selected with --tito-model qwen3_5. --tito-chat-template-kwargs pins template variables (Qwen3.8 reasoning_effort) for every render. A golden test on the real Qwen/Qwen3.8-27B tokenizer asserts incremental == from-scratch across a multi-turn tool session for xhigh and low, clean mismatch audit, and that the default family indeed cannot tokenize this template's tool results. - The gateway forces stream=false upstream and used to hand a JSON body back to a client that asked for stream=true; Pi's openai-completions provider always streams and fails with 'Stream ended without finish_reason'. The completed turn is now re-framed as SSE (one delta chunk with role/content/ reasoning/tool_calls+index, one finish chunk with usage, [DONE]) when and only when the client asked for a stream; upstream errors pass through as JSON. Content is identical to the JSON body; only the framing changes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…hink>) A thinking model truncated inside its <think> block returns reasoning_content with content:null and no tool_calls; vLLM derender reports it with finish_reason=length. The harvest rejected it as malformed and the gateway answered 502, which surfaced on the first real Qwen3.8 turn through the gateway. Such a turn is a real generation with real token ids and must be recorded; only a message with none of content, tool_calls, or reasoning is malformed. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
First real Pi -> gateway -> vLLM(Qwen3.8) rollout died on turn 2 with
'rollback failed: no assistant message found in the first 2 matched
messages'. vLLM's derender stores function.arguments as json.dumps output
('{"command": "ls"}'); Pi echoes the same call back through
JSON.stringify ('{"command":"ls"}'). message_matches compared the strings
verbatim, judged the assistant turn rewritten, and tried to roll back. Both
serializations render to identical tokens (the template parses arguments
before rendering), so compare (id, name, parsed arguments) instead.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Pi (and any OpenAI-style client fed by vLLM) echoes earlier assistant turns with their thinking under 'reasoning'; Qwen3/3.5/3.8 templates only read 'reasoning_content'. The first Qwen3.8 rollout audit showed exactly that divergence: the accumulated prefix carried the real thinking tokens while the from-scratch render had empty <think> blocks for every earlier turn. Aliasing at render time makes incremental == from-scratch again. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…window) vLLM rejects `prompt + max_tokens > max_model_len` outright, so an agent profile with a generous per-turn budget turns into a hard wall on trajectory length (max_model_len - max_tokens). The gateway knows the session's exact prompt length; with the window configured, the vllm turn caps sampling_params.max_tokens at the remaining room after render and before generate. A prompt that already fills the window is left to the backend's own verdict. Unset = previous behaviour. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…vLLM's off-by-one prompt count vLLM validates prompt + max_tokens against max_model_len already in /v1/chat/completions/render, counting one token more than the session's exact prompt ids for Qwen3-family templates. Observed: prompt 160,444 (gateway) vs 160,445 (render) -> 262,145 > 262,144 -> 400 despite the post-render clamp. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ool_calls compare, reasoning alias) and --tito-context-window Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Normalize only rendered terminal-stop text while preserving native token IDs, logprobs and usage. Narrow tool-call function values for type safety. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Test plan
The user explicitly approved both the TITO and abridge feature scopes. Merge only after the required validation and review conditions are satisfied.
Generated with Devin