Engine-agnostic agent loop on lean and lean-chat; loop fixes and tests; README and license

.cargo/config.toml

@@ -0,0 +1,5 @@
+# wasm32 SIMD128 in every wasm build of this workspace (lean's CPU forward
+# pass uses SIMD128 kernels when it is compiled in). Every browser with
+# WebGPU or a recent wasm engine supports it.
+[target.wasm32-unknown-unknown]
+rustflags = ["-C", "target-feature=+simd128"]

.github/workflows/pages.yml

@@ -3,13 +3,15 @@ name: pages
 on:
   push:
     branches: [main]
+  pull_request:
+    branches: [main]
   workflow_dispatch:
 
 permissions:
   contents: read
 
 concurrency:
-  group: pages
+  group: pages-${{ github.ref }}
   cancel-in-progress: false
 
 jobs:
@@ -30,10 +32,14 @@ jobs:
         run: ENGINE_BUILD="${{ github.sha }}" scripts/build.sh
 
       - uses: actions/upload-pages-artifact@v3
+        if: github.event_name != 'pull_request'
         with:
           path: _site
 
+  # Off until the repo has a page to publish: _site holds only pkg/, with
+  # no index.html.
   deploy:
+    if: false
     needs: build
     runs-on: ubuntu-latest
     permissions:

.gitignore

@@ -6,3 +6,4 @@
 **/pkg/
 **/pkg-mt/
 crates/agent/reference/.venv
+.claude/

Cargo.toml

@@ -13,23 +13,22 @@ edition = "2021"
 anyhow = "1.0"
 serde = { version = "1.0", features = ["derive"] }
 serde_json = { version = "1.0", features = ["preserve_order"] }
-minijinja = { version = "2", features = ["json", "preserve_order"] }
-minijinja-contrib = { version = "2.24.0", features = ["pycompat"] }
 tokenizers = { version = "0.22", default-features = false }
 clap = { version = "4.5", features = ["derive", "env"] }
+pollster = "0.3"
 
 wasm-bindgen = "0.2"
+wasm-bindgen-futures = "0.4"
 js-sys = "0.3"
-web-sys = { version = "0.3", features = ["console"] }
 console_error_panic_hook = "0.1"
-getrandom = { version = "0.3", default-features = false }
 
-# Pinned to an exact commit of origin/main so a clean clone always resolves
-# the same lean API, with no path dependency anywhere in this workspace.
-# Default features (native-bin) pull in the pollster/clap lean's own
-# native Engine::new()/CLI need; agent-wasm overrides to default-features
-# = false, features = ["web"] for the wasm32 build.
-lean = { git = "https://github.com/idle-intelligence/llm-web", rev = "b321c16117cc2aef425c6471cbcb4cd7257fe518" }
+# lean's engine (text in, text out) and its chat layer (messages and tools
+# to prompt text with the model's own template), pinned to one commit of
+# llm-web so a clean clone always resolves the same API. No default
+# features: the agent crate turns on `threads` natively and agent-wasm
+# turns on `web`.
+lean = { git = "https://github.com/idle-intelligence/llm-web", rev = "1fce23d866c23fb6f3635abb51db52c50ff87808", default-features = false }
+lean-chat = { git = "https://github.com/idle-intelligence/llm-web", rev = "1fce23d866c23fb6f3635abb51db52c50ff87808" }
 
 [profile.release]
 lto = true

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Thomas Cambau
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -1,49 +1,79 @@
 # agent-web
 
-A tool-calling agent for Qwen2.5-Instruct models, running on the [lean](https://github.com/idle-intelligence/llm-web) inference engine.
+A tool-calling agent loop for Qwen2.5-Instruct models, running on the [lean](https://github.com/idle-intelligence/llm-web) inference engine.
 
-lean is inference only: it loads weights, tokenizes, runs the model forward pass, and picks a backend (WebGPU, then CPU threads, then a single CPU thread) by capability. Everything an agent needs on top of that, the tool-calling chat template, output parsing, the step-wise loop, and an eval harness, lives here instead.
+lean runs the model: text in, text out, on a backend it picks by capability (WebGPU, then CPU threads, then a single CPU thread). Its chat layer, lean-chat, renders messages and tools into prompt text with the model's own chat template. This repo holds what an agent adds on top: parsing the model's tool calls, the step-wise loop, and an offline eval harness. The loop reaches the model through one small trait (prompt text in, streamed text out, abort, a typed context-overflow error), so any engine that completes text can drive it.
 
 ## What it does
 
-- Renders Qwen2.5-Instruct's own tools chat template (`<tools>...</tools>` in the system turn, `<tool_call>{"name":..., "arguments":...}</tool_call>` in the model's output), token-exact against `transformers.apply_chat_template(..., tools=...)`.
-- Parses the model's tool calls out of its raw output, with a retry policy for malformed or repeated calls.
-- Drives a step-wise loop: render prompt, generate, parse, hand back pending tool calls, accept results, repeat until a final text answer.
-- Scores transcripts against a fixed set of utterance cases (a Sonos speaker-control tool set) under several acceptance modes (exact call sequence, required subset, no mutation, any valid play call).
+- Renders each step's prompt with the model's own chat template and the tools, through lean-chat. For Qwen2.5-Instruct that is `<tools>...</tools>` in the system turn and `<tool_call>{"name":..., "arguments":...}</tool_call>` in the model's output. The `fixture_parity_agent_tools` test checks the prompts byte for byte against `transformers.apply_chat_template(..., tools=...)`.
+- Parses the model's tool calls out of its raw output, and regenerates when a reply is malformed, calls an unknown tool, or repeats a call.
+- Drives a step-wise loop: render the prompt, generate, parse, hand back the pending tool calls, take their results, and repeat until a final text answer.
+- Scores runs against a fixed set of utterance cases for a Sonos speaker-control tool set, under four acceptance modes: exact call sequence, required calls in order, no mutating call, any valid play call.
 
 ## Crates
 
-- `crates/agent`: the template, parser, loop, and eval library. Depends on lean through a pinned git revision.
-- `crates/agent-wasm`: wasm-bindgen bindings for the agent, running on lean's native CPU engine.
-- `examples/agent_cli.rs`: a native CLI that runs one utterance through the loop, answering tool calls from fixture JSON, and prints each step.
-- `examples/agent_eval.rs`: a native CLI that runs the full eval set and prints a markdown results table.
+- `crates/agent`: the parser, the loop and the eval harness, plus `LeanGenerator`, the loop's generator on lean's engine. Its examples are two native CLIs: `agent_cli` runs one utterance through the loop and prints each step, answering tool calls from fixture files; `agent_eval` runs the eval set and prints a markdown results table.
+- `crates/agent-wasm`: wasm-bindgen bindings for the agent on lean's engine, for a Web Worker. lean uses WebGPU when the browser grants an adapter, otherwise the CPU.
 
-## CLI examples
+lean and lean-chat come from a pinned git revision of [idle-intelligence/llm-web](https://github.com/idle-intelligence/llm-web).
+
+## Model
+
+The examples and the model tests use [Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct): the Q4_0 GGUF from [Qwen/Qwen2.5-3B-Instruct-GGUF](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF), and the tokenizer files from the model repo.
+
+```sh
+hf download Qwen/Qwen2.5-3B-Instruct-GGUF qwen2.5-3b-instruct-q4_0.gguf --local-dir models/Qwen2.5-3B-Instruct-GGUF
+hf download Qwen/Qwen2.5-3B-Instruct tokenizer.json tokenizer_config.json --local-dir models/Qwen2.5-3B-Instruct
+```
+
+## Build and test
+
+```sh
+cargo build --release
+cargo test --workspace
+```
+
+The tests that need the model files are ignored by default. With the files above:
+
+```sh
+LEAN_GGUF_QWEN25_3B=$PWD/models/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_0.gguf \
+LEAN_TOKENIZER_DIR_QWEN25_3B=$PWD/models/Qwen2.5-3B-Instruct \
+cargo test -p agent --release -- --ignored
+```
+
+The wasm package (needs [wasm-pack](https://rustwasm.github.io/wasm-pack/)) is built into `_site/pkg` by:
+
+```sh
+scripts/build.sh
+```
+
+## Run
+
+One utterance, with the tool calls answered from `crates/agent/fixtures/sonos/results/`:
 
 ```sh
 cargo run -p agent --example agent_cli --release -- \
-  --model /path/to/qwen2.5-3b-instruct-q4_0.gguf \
-  --tokenizer /path/to/Qwen2.5-3B-Instruct \
-  --tools crates/agent/fixtures/sonos/tools-12.json \
+  --model models/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_0.gguf \
+  --tokenizer models/Qwen2.5-3B-Instruct \
+  --tools crates/agent/fixtures/sonos/tools-subset.json \
   --utterance "pause the kitchen"
 ```
 
+The eval set, as a markdown table on stdout (`AGENT_EVAL_MAX_CTX` sets the context size in tokens, default 4096; `AGENT_EVAL_DATE` the date in the table's title):
+
 ```sh
-LEAN_GGUF_QWEN25_3B=/path/to/qwen2.5-3b-instruct-q4_0.gguf \
-LEAN_TOKENIZER_DIR_QWEN25_3B=/path/to/Qwen2.5-3B-Instruct \
+LEAN_GGUF_QWEN25_3B=models/Qwen2.5-3B-Instruct-GGUF/qwen2.5-3b-instruct-q4_0.gguf \
+LEAN_TOKENIZER_DIR_QWEN25_3B=models/Qwen2.5-3B-Instruct \
 cargo run -p agent --example agent_eval --release
 ```
 
-Both pick lean's GPU engine when a wgpu adapter is available and fall back to the CPU engine otherwise, the same capability-based choice lean's own CLI makes.
-
-## Demo
-
-A browser demo on the repo's GitHub Pages is planned; link to follow once it is built.
+Both pick the backend by capability: the GPU when a wgpu adapter is available, otherwise the CPU forward pass on every core. Each step reuses the KV cache of the previous one for the prompt prefix they share, so the system prompt and the tools are prefilled once per turn.
 
 ## Eval
 
-43 utterance cases, a 12-tool Sonos speaker-control set, Qwen2.5-3B-Instruct (Q4_0 GGUF), lean's CPU engine, unconstrained decoding (no grammar-constrained output): 9.1% correct, run on 2026-10-07. See `docs/runs/2026-10-07-mcp-agent-lean.md` for the full run log, including a 0.5B-model follow-up and the baseline this was compared against.
+`eval/utterances.json` holds 46 cases. With the 13-tool subset (`crates/agent/fixtures/sonos/tools-subset.json`), 33 of them are solvable and scored; the other 13 need a tool outside the subset and are skipped. Qwen2.5-3B-Instruct Q4_0, run on the CPU with unconstrained decoding (no grammar-constrained output) on 2026-10-07, got 3 of the 33 right (9.1%). The per-case results are in `docs/runs/2026-10-07-sonos-eval.md`.
 
 ## License
 
-Not yet decided; no license file is included.
+The code is under the MIT license, see `LICENSE`. The model weights are not part of this repo and come under their own license, given on each model's page.

crates/agent-wasm/Cargo.toml

@@ -11,8 +11,8 @@ agent = { path = "../agent" }
 anyhow = { workspace = true }
 serde_json = { workspace = true }
 lean = { workspace = true, features = ["web"] }
+lean-chat = { workspace = true }
 wasm-bindgen = { workspace = true }
+wasm-bindgen-futures = { workspace = true }
 js-sys = { workspace = true }
-web-sys = { workspace = true }
 console_error_panic_hook = { workspace = true }
-tokenizers = { workspace = true, features = ["unstable_wasm", "fancy-regex"] }