[BobClawblaw]

What

--drafter for the llama family: a smaller Llama on the same tokenizer proposes a stream's next tokens, and the served model verifies them in its windows, so replies keep the model's bits (builds on #1415 and #1416). It was brought up with Nanbeige/Nanbeige4.1-3B drafting for Nanbeige4.3-33B. The two share the tokenizer exactly (166,100 ids, merges, added tokens, bos and eos); only their vocab_sizes pad differently. The 3B is admitted as a drafter only: serving it on its own is refused by name.

How

zig/src/families/llama/drafter.zig is the lane host's priced copy proposer (LaneHost.proposers). A round asks it for up to 15 tokens after the stream's context. The window verifies them row by row and keeps the prefix the model's own draws agree with, as it keeps any copy draft. There is no lane-core change and no new kernel: the drafter runs the family's own forward on the served model's queue (Model.loadOn) and in its residency set.

• Caches. One cache a stream, synced to the context by their longest common prefix: an accepted draft costs no second pass, and a rejected one is cut. One budget (budget.Ledger) holds the streams' caches and the drafter's: when a stream is admitted, its drafter cache is made or grown to the stream's capacity (a growth's copy counted while it lives), and the stream waits if the two don't fit beside what is held. A freed slot keeps its tokens for the next request that shares its prefix, counted too, and gives way first when a stream needs the room.
• Drafts. Greedy steps chained on the GPU through token slots (a step's argmax is the next step's row), so one proposal is one chain of command buffers. A chain stops at an eos.
• Depth. It follows measured rates: the stream's decayed acceptance, a chained step's wall time, and a line fitted to the round's time by its window's rows. It chooses the depth with the most expected tokens a millisecond.
• Shared rounds are priced. With several streams in a round, each one drafts the depth with the most tokens a millisecond for the whole round (pricing.depthShared: every stream's 1 + p + ... + p^k tokens for a window of all their rows and every stream's chained steps, against the round undrafted); a stream drafts nothing only in a round where that costs more than it lands.
• Widths. --drafter-bits 4 (the default) serves an MLX 4-bit drafter as published, in mlx_lm's layout, read as llama: published MLX affine quants (8-bit and 4-bit) read in place, beside the bf16 default (builds on #1415) #1416 reads the served model's quants: mlx-community/Nanbeige4.1-3B-4bit @ 4d6c41d88f1416777bbda005aad92798e56e5ba9. --drafter-bits 0 serves the bf16 release @ 9c4555c37921c0982af8fffeee1e0a6e6b953c4f. The drafter's weights change no reply. Both are Apache-2.0.
• Check mode. tf-llama-run <model> <tokens> draft <drafter>,<n> decodes greedily through the drafter's windows and prints the tokens with the rounds and drafts kept, for the identity check below.

Measured (M5 Ultra, the bf16 33B)

Identity. On eight chat prompts (128 tokens, thinking on and off, English and Chinese) and after a 1,500-token prompt (256 tokens), the drafted greedy tokens equal the plain greedy tokens exactly with the published 4-bit drafter. The bf16 drafter gives the same tokens on two of the prompts. The drafted rates on those nine:

• 23.2-46.4 tok/s against 16.0-16.3 plain.
• 1.74-4.23 tokens a round.
• A chained 4-bit step costs 4.0-4.2 ms. A bf16 step costs 8.5 ms: more of its drafts land, but it is slower (22.6 and 27.2 tok/s against 26.4 and 30.3).

Served with --parallel 1 (the recipe's drafted setting). The harness's quick plan passes in full with the drafter on this branch's build (28 Oct 2026; its ruler, with no other model loaded, 113.5 / 110.7 / 109.2 / 101.8 tok/s), as it did on the first build: structured 18/18 with drafted == plain 12/12, tools 5/5, seeded sampling 9/9 equal on repeat and undrafted, turns resumed in 0.35 s, soak 60/60, concurrent 12/12, streaming, multi-turn, stop, thinking, max_tokens.

	undrafted (--parallel 2)	drafted (--parallel 1)
one stream, the harness ruler's prose / code / structured / list (256 tokens)	16.4 / 16.3 / 16.4 / 16.4 tok/s	116.8 / 110.8 / 109.3 / 99.5
its rounds	61 ms, 1 token	44.8 ms, 4.88 tokens (81% of drafts land)
bench_concurrent.py's code / chat, one stream, fresh server (128 tokens)	20.2 / 16.4	30.0-32.8 / 24.9-25.3
first token	0.23-0.26 s	0.23-0.26 s

The ruler's texts repeat, and the drafter continues them almost whole. Ordinary prompts gain 1.4-2.9x on the nine check prompts and 1.5-1.6x on bench_concurrent.py's. A sampled stream at temperature 0.7 gains less, because greedy drafts land less often against sampled draws: 19.1 and 24.5 tok/s against 16.2.

Two streams. With --parallel 2 and the drafter, every stream drafts in shared rounds. On fresh servers, eight new prompts sent two at a time (greedy, 256 tokens): 34.4-42.4 tok/s together (38.4 on average) against 26.6-30.0 (29.0) without --drafter, every reply's token ids equal. Streams joining, leaving, queued, refused (HTTP 400) and cancelled each return the ids of their plain solo run; in rounds shared from the first token to the last, 189 drafts were offered and 127 kept.

Receipt

• Environment: TensorFold at this branch (main d0c9dc9 plus these commits), Zig 0.17.0, one Apple M5 Ultra (256 GB unified, macOS 27, tensor units). Nanbeige/Nanbeige4.3-33B @ c1093c501cc6460427c0ac634db7223d0450c0ee, served with --context 131072 --parallel 2 --prompt-cache-gib 8 and LLAMA_LOAD_LIMIT_GB=150, with --drafter mlx-community/Nanbeige4.1-3B-4bit @ 4d6c41d88f1416777bbda005aad92798e56e5ba9 (--drafter-bits 4) or Nanbeige/Nanbeige4.1-3B @ 9c4555c37921c0982af8fffeee1e0a6e6b953c4f (--drafter-bits 0). An idle server of another model was loaded beside it for the first build's measurements; the 28 Oct streams and speed runs, and the harness's ruler, ran with no other model loaded.
• Exactness, tools/bench_concurrent.py --levels 1,2 --tokens 128 --temperatures 0,0.7 --alone --serial with each drafter: every reply equals its request alone and with "draft": false (alone 12/12, serial 12/12, failed 0, for each).
• Decode speed, tools/bench_openai.py --tokens 256 --reps 3, before → after with the 4-bit drafter:
  • greedy: 16.4 → 116.7 tok/s (chat) and 18.4 → 116.4 (raw prompt)
  • sampled at 0.7: 16.3 → 18.9 (chat) and 16.25 → 25.6 (raw)
  • The tool repeats each prompt, and a freed drafter cache keeps its tokens for a request sharing its prefix. So greedy repeats draft the earlier reply back whole, and these overstate ordinary text: the fresh-server figures above are the ones to read.
• Prompt speed, tools/bench_cache.py --require-cold, a 9,000-token prompt cold: 6.92 s with the drafter. The drafter reads the prompt after the model's own pass. Repeated raw-token prompts report cached_tokens 0, as in llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415: the family's prompt cache resumes chat turns, and the harness's turns check resumes 5,493 of 5,649 tokens in 0.35 s against 4.39 s cold.
• Tests run, on these commits over main 03d08f1 (the pushed heads are rebased onto d0c9dc9, which adds only a CI workflow time limit): on llama: a smaller Llama drafting for the served one (--drafter), replies unchanged (builds on #1415, #1416) #1417's head, which carries all three, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0, and on llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415's and llama: published MLX affine quants (8-bit and 4-bit) read in place, beside the bf16 default (builds on #1415) #1416's heads zig build test and test-kernel-contract exit 0; zig fmt --check is clean and the lean check finds 0 new problems (428 on main and on each head); llama_teacher_judge_test.py passes; tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 23 kernels: 23 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems (21 kernels on llama: Nanbeige4.3-33B on Metal, the bf16 release served as published, every kernel in the contract #1415's head). Rebuilt on the rebased heads, the bf16 release's 1,500-position logits are bit-identical to the judged run on all three, both quants' to theirs, and a drafted greedy chat (128 tokens) equals the plain one. The pushed heads differ from the ones CI ran on only in comments and docstrings (the lean check's one-line rule); fmt, lean, the judge's tests, the build and zig build test were rerun on them.
• Pushed head: rebased onto main d0c9dc9 (Release 1.1.8 and a CI workflow time limit), whose commits since the last push touch none of this family's files or the paths it serves through.
• Not run: the harness quality suite; CUDA, HIP, M1-M4.

Checks

zig build test covers the depth rule and its shared-round pricing, the cost fit, the acceptance history, the drafter role, the roll call, the one budget (two drafted 240,000-token requests that each fit a 100 GiB budget alone are refused together), a slot's growth bytes and a failed warm-up freeing the drafter. Without --drafter, the served path is unchanged: t1500 and t64 logits and a greedy chat are bit-identical to #1415's measured build.

[ashhart]

Drafted equals plain per the receipts, and there's no lane-core change or new kernel. That's good. Two things:

• The default --drafter-bits 4 refuses a bf16 drafter and needs a llama_convert.py --bits 4 conversion, so the default depends on llama: published MLX affine quants (8-bit and 4-bit) read in place, beside the bf16 default (builds on #1415) #1416. Please make bf16 the default, or accept MLX's layout once llama: published MLX affine quants (8-bit and 4-bit) read in place, beside the bf16 default (builds on #1415) #1416 does.
• Shared rounds. With the drafter, two-stream greedy drops from 31.0 to 27.8 tok/s (chat) and from 34.1 to 29.5 (code) on an M5 Ultra by the PR's own numbers. Yet the recipe keeps the drafter on. Either skip drafts in shared rounds when they cost more than they land, or don't recommend the drafter above --parallel 1.

[BobClawblaw]

Thanks. Pushed 5d4a3af (one commit, on #1416's new head):

--drafter-bits 4. We took the second route: with #1416 reading MLX's layout in place, the default now serves a published MLX 4-bit drafter as it is on the Hub, mlx-community/Nanbeige4.1-3B-4bit @ 4d6c41d88f1416777bbda005aad92798e56e5ba9. There is no conversion and nothing depends on llama_convert.py, which is gone. --drafter-bits 0 serves the bf16 release. With the published 4-bit drafter, drafted greedy tokens equal plain greedy tokens on all nine check prompts, at 23.2-46.4 tok/s against 16.0-16.3. The bf16 drafter gives the same tokens on two of them, at 8.5 ms a step against 4.0-4.2 ms.

Shared rounds. Both of your options, in turn:

1. While more than one stream holds a drafter cache, the drafter proposes nothing (Drafter.sharing), so shared rounds decode undrafted. On fresh servers (bench_concurrent.py, 128 greedy tokens, 8 runs each), two drafted streams give 31.2-31.3 tok/s together on chat against 31.3 without --drafter, and 31.4-31.6 on code against 34.4. On the harness ruler they give 31.9-32.5 against 28.0-31.8.
2. Code is still 8% behind, because a server without --drafter keeps the lane core's own copy drafts and shared rounds give those up. Getting them back would mean copying the lane core's copy policy into the family, so instead the recipe no longer recommends the drafter above --parallel 1. Its drafted command is now --parallel 1, and the doc says a server for several streams leaves the drafter off. The harness's quick and speed plans pass on that server (one stream 99.5-116.8 tok/s on the ruler).

One correction to the old receipt: the 87-117 tok/s single-stream figures from bench_openai.py and bench_concurrent.py --alone overstate ordinary text. Those tools send the same prompt more than once, and a freed drafter cache keeps its tokens for a request sharing its prefix, so a greedy repeat drafts the earlier reply back whole. On a fresh server one stream gives 30.0-32.8 tok/s on bench_concurrent.py's code and 24.9-25.3 on its chat, against 20.2 and 16.4. The PR body now quotes those.

Also: the 3B is admitted as a drafter only (config.zig's qualified list has a role). Serving it on its own is refused by name.

On these commits over main a1e6d34: zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security, test-pull-security all exit 0; zig fmt --check is clean; the lean check finds 0 new problems (428 on main and on the head); tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 23 kernels: 23 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems. After the rebase, the Fibonacci prompt's drafted tokens still equal the plain ones. Main then took MOSS-Image (#1457), DFlash2 and MTP fixes for other families, a GLM CUDA log message and an opt-in prompt-cache rule for CUDA families (off by default): none of it in this family's files or the paths it serves through. Rebased onto main 15a0b2a for this push, the stack's top head (#1417's, which carries these commits) reran the build, tf-kernel-contract --only llama, test-kernel-contract, zig fmt --check and the bf16, quant and drafted bit-identity checks, and all pass.

[ashhart]

Thanks @BobClawblaw. The drafter's default now reads the published 4-bit layout, or bf16 with an explicit width of 0 (zig/src/native/llama.zig:158), and the recipe pins both checkpoints.

Three asks:

1. One memory budget. Startup counts both drafter caches (zig/src/native/llama.zig:94), then hands the whole remaining budget to the target (:110). Runtime admission doesn't count the drafter's live and kept caches, its growth copies or kept prompt states. On the production formulas, with a 100 GiB cache budget, context 262,144 and two streams, startup passes, yet two admissible 240,000-token requests together exceed the budget by 25.3 GiB. Please admit the target and the drafter against one budget, and add a host test where two requests that each fit alone are refused together.
2. Shared rounds. Thanks for taking up the shared-round point. Our suggestion was to skip a shared round's drafts when they cost more than they land; the head skips them in every shared round (zig/src/families/llama/drafter.zig:141, :345), so drafting stops as soon as a second stream joins. Our recipe rules count that as a drafts-off fallback, which is a bug for an advertised drafted path (docs/recipes/SKILL.md, the concurrency rules). Please keep drafting on for every admitted stream and price each shared round, then check drafted against plain and concurrent against solo as streams join, leave, are refused and are cancelled.
3. Warm-up failure leaks the drafter. If warm-up fails (zig/src/native/llama.zig:117), the drafter made just before it is never freed: its errdefer comes after, at :121. Please register the cleanup before the warm-up, and add a test that fails it.

[BobClawblaw]

Thanks. Pushed b652456 (one commit on #1416's new head):

1. One memory budget. The backend's admission now runs on one ledger (budget.Ledger) for every cache the load holds. That covers the streams' caches, the drafter's slots (taken or kept) and a slot's growth copy while it lives. When a stream is admitted (backend.prefillFn), its drafter slot is made or grown to the stream's capacity in the same check (Drafter.slotsHook: growth, prepare, shed). If the two don't fit beside what is held, kept drafter slots give way first; if they still don't fit, the stream waits (error.LaneMemory) until another finishes. Slots no longer grow during decoding, so nothing allocates outside admission. Kept prompt states fit in what the ledger leaves. The host test (llama budget: two drafted requests that each fit alone are refused together, and pass in turn) takes the 33B's and the 3B's configs: a 100 GiB budget, a 262,144-token context and two streams pass startup. One 240,000-token request fits, the second waits (together they would pass the budget by 24.5 GiB on these formulas), and after the first ends the second fits beside the first's kept slot. growth has its own test (the new cache plus the old state's copy, nothing when the slot holds enough).

2. Shared rounds are priced, and drafting stays on. pricing.depthShared gives each stream in a shared round the depth with the most tokens a millisecond for the whole round. Each stream's drafts land 1 + p + ... + p^k tokens, for a window of all the streams' rows plus every stream's chained steps, against the round undrafted. It returns 0 only for a round where drafting costs more than it lands. Shared rounds don't feed the fitted line of a round's time by its rows. Checked on a drafted --parallel 2 server (--context 32768), greedy, token ids from return_token_ids:

• A drafts alone; B joins after A's first 40 tokens (shared rounds); C arrives while two streams hold the server and queues; D starts and its client disconnects after 20 tokens (cancelled); E sends a prompt past the context and is refused with HTTP 400. A, B and C each return the token ids of their plain solo run ("draft": false), and A alone after the cancel is unchanged.
• Two streams from the first token to the last: both equal their plain solo runs, and the shared rounds offered 189 drafts and kept 127.
• Speed on fresh servers, eight new prompts sent two at a time (greedy, 256 tokens): two drafted streams decode together at 34.4-42.4 tokens/s (38.4 on average) against 26.6-30.0 (29.0) without --drafter, all eight replies' ids equal to the undrafted ones.
• bench_concurrent.py --levels 1,2 --alone --serial on a fresh drafted server: every reply equals its request alone and undrafted (alone 48/48, serial 12/12). Its speeds aren't quoted, because its repeated prompts let a freed slot draft the earlier reply back.

3. Warm-up failure. Drafter.start owns the model from the call: if init fails it frees the model, and the drafter it makes is freed if its warm-up fails (started, with the cleanup registered before the warm-up). The host frees the drafter before the backend, whose ledger the drafter's slots give back to. The test llama drafter: a warm-up that fails frees the drafter it was made for makes the warm-up fail and checks the drafter is freed. tf-llama-run's draft mode uses Drafter.start and Drafter.prepare too.

The harness's quick plan on the drafted --parallel 1 server, this build (its ruler with no other model loaded): structured 18/18 with drafted == plain 12/12, tools 5/5, sampling 9/9, turns resumed 5,493 of 5,649 tokens (== cold), soak 60/60, drafts 12/12, concurrent 12/12, streaming 4/4, multi-turn 8/8, stop 3/3, thinking 10/10, copy 2/2, max_tokens 3/3. Its ruler gives 113.5 / 110.7 / 109.2 / 101.8 tokens/s on one stream (texts the drafter continues almost whole).

The doc's drafter section is rewritten for the budget and the priced rounds. --parallel 2 drafts both streams.

CI on these commits over main 03d08f1 (the pushed heads are rebased onto d0c9dc9, which adds only a CI workflow time limit): on #1417's head, which carries all three, zig build test, test-kernel-contract, test-golden, fuzz-cortheon, dev-tools, test-server-security and test-pull-security all exit 0, and on #1415's and #1416's heads zig build test and test-kernel-contract exit 0; zig fmt --check is clean and the lean check finds 0 new problems (428 on main and on each head); llama_teacher_judge_test.py passes; tf-kernel-contract --only llama on the M5 Ultra: SUMMARY 23 kernels: 23 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems (21 kernels on #1415's head). Rebuilt on the rebased heads, the bf16 release's 1,500-position logits are bit-identical to the judged run on all three, both quants' to theirs, and a drafted greedy chat (128 tokens) equals the plain one. The pushed heads differ from the ones CI ran on only in comments and docstrings (the lean check's one-line rule); fmt, lean, the judge's tests, the build and zig build test were rerun on them.