fix: ReAct reliability pass (structured output, preferSummarize, stale docs)

[pluginslab]

Three independent reliability fixes in the inference path. They came out of investigating why the local 7B "struggles sometimes". None of them change the model or the runtime.

Deliberately not touching the engine, the model catalog, or anything WebLLM-adjacent while #228 (the WP.org submission) is open.

---

1. Correct stale docs, delete dead config

docs/ARCHITECTURE.md and .claude/CLAUDE.md both claimed:

• a 4096-token context window
• 2000-char tool-result truncation

Both are wrong, and have been for a while. The code says:

	Docs claimed	Actually
Context, Qwen3 1.7B	4096	8192 (MODEL_CONTEXT_SIZES)
Context, Qwen2.5 7B	32768	32768 ✓
Tool result truncation	2000	3000 (maxToolResultLength)

This matters more than a typo. Any analysis of the agent built on these docs inherits the error, and the obvious "fix" it suggests, bumping the context to 16384, would actually halve the 7B's window.

Also removed MODEL_CONFIG.context_window_size = 8192 from model-loader.js. It looked authoritative, contradicted MODEL_CONTEXT_SIZES, and was never passed to the engine: its only reference in the entire codebase was its own line in the export block. It is almost certainly where the "single global 4096-ish context" impression came from.

The docs now point at MODEL_CONTEXT_SIZES as the source of truth and mention the agentic_admin_context_size localStorage override resolved by ModelLoader.getEffectiveContextSize().

2. Grammar-constrain the ReAct action envelope

The loop only ever accepts two shapes:

{"action": "tool_call",    "tool": "<tool-id>", "args": {}}
{"action": "final_answer", "content": "..."}

Until now we asked for that in the system prompt and then repaired whatever came back. parseActionFromResponse() is ~80 lines of strip-the-think-block, strip-the-code-fence, sanitize-control-characters, retry-with-quote-replacement. That is a lot of machinery for "the model produced not-quite-JSON".

response_format appeared exactly once in the whole codebase, in workflow-orchestrator.js:469, and even there as bare { type: 'json_object' } with no schema. The ReAct loop, the hot path, used none.

Now REACT_ACTION_SCHEMA is passed as response_format, so the grammar enforces the envelope during decoding. Malformed JSON becomes unrepresentable rather than something we repair after the fact. parseActionFromResponse() stays as-is, it is still the path for unconstrained turns and a safety net.

The catch, and why this is gated

A JSON grammar cannot represent Qwen 3's leading <think>...</think> block. Constraining a thinking turn would truncate the reasoning and fail. So this applies only when thinking is already off:

const useStructuredOutput = this.config.structuredOutput && suppressThinkingUi;

which covers both existing nothink paths: router-level disableThinking for clear action commands, and post-tool disableThinkingAfterTool. Thinking turns are untouched.

ExternalEngine strips the WebLLM-specific schema key from response_format before proxying. WebLLM understands it; OpenAI rejects unrecognized keys in response_format outright, so external providers keep plain JSON mode.

Escape hatch if it misbehaves in the wild: structuredOutput: false.

3. Adopt preferSummarize on the report abilities

react-agent.js:451 already has the right mechanism: if an ability sets preferSummarize, the loop skips the second LLM call and returns summarize() directly. Added in the read-file work, explicitly to dodge truncation.

It was used by 2 of 37 abilities.

Flipped it on for the four abilities that emit large, complete, self-contained reports:

Ability	Why
site-health	summarize() already branches on the user's question (PHP version, theme, memory limit, ...) and returns a complete answer. The LLM adds nothing.
security-scan	Full pass/fail report grouped by severity. Re-rendering risks dropping findings.
verify-core-checksums	Emits fenced diff blocks. The LLM mangles fenced code, and diffs blow past 3000 chars.
file-scan	Lists every plugin, theme and mu-plugin scanned. A truncated security scan is worse than none.

Selection criterion: the output is a report, not data the LLM needs to reason over or filter. I deliberately left plugin-list, post-list, user-list and friends alone, since #109 wants the LLM filtering those to match the question, which is the opposite requirement.

Workflows are unaffected. preferSummarize is only consulted by react-agent.js and chat-orchestrator.js; workflow-orchestrator.js does not look at it. The four workflows chaining these abilities (check-if-hacked, performance-check, plugin-audit, site-cleanup) read r.abilityId results directly and are untouched.

Prerequisite: broken markdown

Enabling preferSummarize meant summarize() output goes to the user verbatim, with no LLM in between. Which surfaced this:

return `Your PHP version is * * ${ result.php_version } * * .`;

* * 8.2 * * instead of **8.2**, so it renders as literal asterisks. Present throughout site-health, and also in core-site-info, core-environment-info and plugin-list. The LLM had been silently repairing it on the way out, which is exactly why nobody noticed.

Fixed in all four files. The last three do not get preferSummarize in this PR, but leaving known-broken markdown in place because it happens to be masked did not seem right.

---

Testing

• 4 new unit tests on the gating: not constrained while thinking is on, constrained when disableThinking is set, constrained on the follow-up turn once disableThinkingAfterTool fires, and structuredOutput: false honoured.
• 95 unit tests pass (was 91).
• npx wp-scripts lint-js clean on every modified file. The 4 remaining model-loader.js warnings (nested ternary, JSDoc) are pre-existing and outside this diff.
• npx wp-scripts build succeeds. Only the pre-existing bundle-size warnings.
• No PHP touched.

What is NOT verified, and why

Item 2 is not empirically validated. The tests above prove the request is shaped correctly and gated correctly; they do not prove the model behaves better, because they run against a mocked LLM.

The Ollama ability harness cannot validate it either. Verified rather than assumed:

• tests/abilities/runner.js imports nothing from src/extensions/services/. Its own comments say it mirrors react-agent.js's system prompt builder (line 168) and JSON parser (line 211). It is a reimplementation, so it never executes the response_format path.
• tests/abilities/load-abilities.js extracts only { id, label, description } by regex. This PR changes engine request params, post-tool rendering and docs, none of which the harness can see.

So the harness measures tool selection only, and this PR cannot move that number in either direction. What it can do is catch a regression in the regex loader, since I edited seven ability files. That check passed without needing Ollama: 35 abilities extracted on this branch, identical to main, with all seven edited files parsing correctly.

Genuinely validating item 2 needs WebGPU in a real browser, which means the E2E suite, not the Ollama harness.

What this does not do

It does not touch the engine, the model, or the WebLLM version, and it does not address the "7B struggles" question at the model level. If these three do not move the needle, the next honest step is the cloud fallback already on the roadmap, not a different small model.

Refs #228

[pluginslab]

Correction: the Ollama harness cannot validate this PR

The original description suggested re-running the Ollama ability harness against this branch to see whether the structured-output change moves anything. That was wrong, and I have rewritten that section.

Checked rather than assumed:

• tests/abilities/runner.js imports nothing from src/extensions/services/. Its comments say it mirrors react-agent.js's system prompt builder (line 168) and JSON parser (line 211), so it is a reimplementation and never runs the response_format path.
• tests/abilities/load-abilities.js extracts only { id, label, description } by regex. This PR changes engine request params, post-tool rendering and docs, none of which the harness observes.
• The model is not installed locally in any case: Ollama has qwen2.5vl:3b, not the qwen3:1.7b the runner defaults to.

The harness measures tool selection, and nothing here can move that number either way.

The one check it was worth doing, done without Ollama: the regex loader still extracts 35 abilities on this branch, identical to main, with all seven edited ability files parsing correctly. So the preferSummarize additions and the markdown fixes did not break tool selection.

Validating the structured-output change for real needs WebGPU in a browser, so it belongs in the E2E suite. Happy to add a case there if that is worth doing before merge.

[pluginslab]

Browser validation done — and it failed. structuredOutput is now off by default.

I tested item 2 in a real browser via Chrome DevTools (WebGPU, Service Worker mode) instead of guessing. The mocked tests passed; the real model did not.

Result:

Model	structuredOutput	Outcome
Qwen2.5-7B-Instruct-q4f16_1	ON	works — 2 iterations, 1 tool, correct answer
Qwen3-1.7B-q4f32_1	ON	broken — 0 tools, reproduced 2/2

On Qwen3 the console shows the gate firing correctly and then the generation collapsing:

[MessageRouter] Routing to ReAct (keyword + action match, no thinking)
[ReactAgent] ReAct iteration 1/10 (prompt-based mode, thinking UI: off)
[ReactAgent] Structured output: ON — action envelope is grammar-constrained
[ReactAgent] LLM response: {\n\n\n\n… (thousands of newlines, to max_tokens)
[ReactAgent] Initial JSON parse failed: Expected property name or '}' at position 1
[ReactAgent] Could not find complete JSON object
[ReactAgent] LLM output plain text instead of JSON, treating as final answer
[ChatOrchestrator] ReAct completed: 1 iterations, 0 tools used

The user sees "I had trouble understanding how to help. Could you rephrase your request?"

Cause. Qwen 3 is a thinking model and wants to open <think> before the JSON. A JSON grammar cannot represent that block. /nothink is only a soft instruction, so when the model still reaches for <think> the grammar blocks every token except whitespace and decoding degenerates.

This is precisely the risk in the original code comment, but the mitigation was wrong: gating on suppressThinkingUi tracks whether we asked for no thinking, not whether the model complied.

Change. structuredOutput now defaults to false. Since Qwen3-1.7B is DEFAULT_MODEL, shipping it on would have regressed every default install. It stays available as opt-in for non-thinking models:

new ReactAgent( modelLoader, toolRegistry, { structuredOutput: true } )

Schema, machinery and tests are kept, and the tests now assert the off-by-default behaviour explicitly. The failure and its cause are documented on REACT_ACTION_SCHEMA so nobody re-enables it without reading why.

Also found

ConnectorEngine._createCompletion() builds an explicit whitelist body (connector_id, model_id, messages) and drops everything else — including temperature, max_tokens and response_format. So this feature is a silent no-op on the WP 7.0 AI Connector path. Nothing breaks; there is simply no effect there. Worth knowing separately, since temperature and max_tokens being dropped on that path is pre-existing and probably not intended.

Net effect on this PR

Items 1 (stale docs) and 3 (preferSummarize) are unaffected and still worth merging. Item 2 ships as documented, tested, off-by-default machinery plus a real reproduction — which is more useful than the speculative "this should help" it started as.

[pluginslab]

Re-tested in the browser: fix confirmed, and the diagnosis is now observed rather than inferred

Same site, same model (Qwen3-1.7B-q4f32_1, WebGPU, Service Worker mode), same prompt, with the fixed bundle 68eecde9c1a58a5d91d3.

Before (structuredOutput on), reproduced 2/2:

list plugins → "I had trouble understanding how to help. Could you rephrase your request?" — 1 iteration, 0 tools used

After (structuredOutput off by default):

list plugins → Task completed successfully · agentic-admin/plugin-list · "There are 10 plugins installed. 7 are active and 3 are inactive." — 2 iterations, 1 tool used

The mechanism, straight from the console

[ReactAgent] ReAct iteration 2/10 (prompt-based mode, thinking UI: off)
[ReactAgent] Structured output: off — envelope unconstrained
[ReactAgent] LLM response: <think>
</think>

{"action": "final_answer", "content": "There are 10 plugins installed. 7 are active and 3 are inactive."}
[ReactAgent] LLM provided final answer
[ChatOrchestrator] ReAct completed: 2 iterations, 1 tools used

The model emitted <think></think> on a turn where the router had explicitly disabled thinking. That is the whole story, and it is no longer a hypothesis:

• Qwen 3 opens a <think> block regardless of /nothink
• unconstrained, the block is empty and harmless, and parseActionFromResponse() strips it
• grammar-constrained, that block is unrepresentable, so decoding degenerates into whitespace until max_tokens

It also confirms why gating on suppressThinkingUi could never be sufficient: the router controls what we ask for, not what the model emits.

One more fix

The debug line printed "thinking turn" whenever the constraint was absent. With the default now off, that is the wrong cause, and it actively misled me while reading these logs. It now distinguishes disabled-by-config from thinking-turn (a793075).

95 unit tests pass, lint clean, build succeeds.

[pluginslab]

Closing as stale. This pass predates all of the WordPress.org review changes that shipped in 0.11.0 (the knowledge base removal, read-file and connectors rework, and more), so it would need redoing on the current main. The branch is preserved as the tag archive/fix/inference-reliability-and-stale-docs.