Repository navigation
lanes: a lone stream holds drafts only while its drafted rounds measure faster - #1382
Open
BlivionIaG wants to merge 10 commits into
Open
BlivionIaG wants to merge 10 commits into
BlivionIaG wants to merge 10 commits into
Conversation
engine.zig is 592 lines and the shared-round guard needs room in it. pipelinedRound and landInflight move unchanged to pipelined.zig as round and land; Engine.familyRound and Result become public for them. No behaviour change.
…el.shared_guard) A drafter whose hold is a pass of its own costs a shared round more than the depth rule's head steps price: the rule treats the rest of a round's time as overhead that plain rounds pay too. Facts.shared_guard (and Model.shared_guard for a backend) lets the engine compare the two modes directly. shared_guard.zig keeps, for each stream count, committed tokens over the round's wall time in plain stretches (no budgets, so no holds) and in drafted stretches. It starts plain, measures plain rounds for 16 rounds and drafted ones for 4, runs the faster mode for 512 rounds and doubles that, up to 64 times, while the same mode wins again; then it measures the other mode again. Drafted rounds must beat plain ones by 5%. A mode's first round still carries the other mode's drafts or holds and is not measured. An entry is allocated once a stream count; later rounds take no allocation. A plain stretch changes only how many drafts a window carries, never what the target draws, so replies stay the same tokens. Without the flag the engine never consults the guard. Tests: the guard's choice and re-measures, ever rarer re-measures of a losing mode, its skipped first rounds, no allocation after a stream count's first round, and replies equal to one-token decode with the guard on and off.
… stream joining one holds no drafts With the shared-round guard on, a plain round still verified the lane core's context and Recall copies, which are drafts too, priced from a prior; and a stream joining other streams held its first drafts even while the guard ran plain rounds at that stream count, a hold pass for each join. The engine now marks a shared round plain when the guard runs that stream count plain, and copyProposal proposes nothing for it, so the guard's plain mode is the drafts-off round. Open.firstDepth gives no drafts to a stream that joins live streams while the guard runs their count plus one plain. Both are behind Model.shared_guard; without it plain_round stays false and firstDepth is unchanged. Tests: a guarded plain round proposes no copies; a stream joining plain-guarded rounds holds no first drafts and, without the flag, holds them as before.
… holds follow the grants A drafter that gives each held chain draft's chance (Drafter.VTable.chances) may set Facts.chances. Drafted.facts then sets Model.draft_probabilities and Model.granted_depth, so the window allocator prices each stream's verify rows by those chances, as it does a head's node chances. With Facts.confidence_floor it instead verifies the drafts at least that likely (a fixed window when 0). With granted_depth a stream's next hold is as deep as its last grant plus a slack: one more draft when the grant took every held draft, none when it stopped short of them (Stream.grant_short), and after a grant of none a probe of draft_slack drafts at the depth rule's probe interval. Facts.slack sets one slack for all three. A shared round's granted depths are cut so the drafting streams' blocks fit one hold pass of batch_rows. When the shared-round guard starts a drafted stretch, its first round grants every stream one draft, so the stretch measures drafted rounds rather than plain ones left by grants of none. All of it is behind Facts.chances and Model.granted_depth, off by default; windows.allocate records grant_short only for granted depths. Tests: chances price the chain's rows and drafted replies equal one-token decode alone and at 1, 4 and 8 shared streams, with the scheduled verify ragged across streams and verifying fewer rows than a fixed window; the facts reach the lane config; granted depths, their slacks and the one-pass cut; holds as wide as the grants.
…py, and holds kept only while they pay The depth rule asks one head draft for a stream with a copy ahead, and a shared round's granted holds were priced only by their verify rows. A drafter whose hold runs as a pass of its own (Facts.copy_first) pays that pass for the one draft, and the copy round then drops it; Model.copy_first now asks it for none. A drafter may also give its hold pass's timed cost by blocks and depth (Facts.hold_costs, priced by hold_price.zig). With granted depths a shared round then keeps granted holds, most expected drafts a row first, only while each raises the next round's expected tokens a ms over its verify rows and the pass; a probe after a grant of none keeps its hold. A stream's last grant records the drafts it expected to keep (Stream.grant_gain). Both are off by default (copy_first false, hold_costs empty). Tests: with copy_first no hold is asked for a copy round and replies equal one-token decode at 1 and 3 streams; a hold's cost between and past its timed passes; granted holds kept while they pay, none for a dear pass, and a probe's hold kept.
…re faster With granted depths a lone stream's verify rows are priced over the shared costs and the one-stream overhead, which is measured on drafted rounds and so holds the drafter's hold time; nothing compared a lone drafted round's tokens a ms with a plain round's. lone_guard.zig measures each lone stream's rounds of held drafts in stretches of 16 against plain rounds' tokens a ms, kept from stream to stream by sampling mode and context band and taken once in a plain stretch when unknown; copy rounds are left out of both. Drafting goes on while it beats plain by 5%; else the stream runs plain rounds and measures drafting again after 16 plain rounds, doubling to 512 while plain keeps winning. A drafted stretch's first hold drafts one and its slack, and a mode's first round is not measured. It runs only with granted depths and the shared-round guard on a timed engine (Facts.chances and Facts.shared_guard); shared rounds keep their own guard. Off by default. Tests: the guard's verdicts, rarer probes and kept plain rate; through the round loop a lone stream whose drafted rounds lose runs mostly plain and one whose drafted rounds win keeps drafting, replies equal to one-token decode.
This was referenced Oct 26, 2026
BlivionIaG
marked this pull request as ready for review
October 26, 2026 12:47
ashhart
requested changes
Oct 26, 2026
ashhart
left a comment
ashhart
left a comment
Owner
There was a problem hiding this comment.
Thanks @BlivionIaG. Its lone guard correctly respects `timed`, but it depends on #1379's shared guard, which does not, and on the plain-stretch policy decision. This stack also needs alternating A/B on current main for every family the shared lane core serves.
…ule does not depend on the clock The guard planned, observed and chose with the wall clock even when timed was false. Round planning, joining a shared round and observation now skip it in that mode. A test runs the same three streams on a fast and a slow fake clock and checks equal holds and replies.
…xt stream, and probes less often A loss is counted on the stretch's own rate. After two losses in a row the plain wait grows until a probe costs at most 3% of the plain time. The band keeps its last losing verdict, so the next stream in that band starts plain. Any winning stretch clears it.
Contributor
Author
|
Thanks @ashhart. The #1379 fix is merged in, so the guards respect |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Branch
series/lanes-lone-guard, 1 commit onseries/lanes-chances:b66a7629e. Part of #1374.Update after the review
Head
6539b72be(the #1379timedfix merged in, and a change to the lone guard for long contexts).plain time. A band remembers its last losing verdict, so the next stream in that band starts plain. Any win clears it.
No flag and no environment switch.
and -9.0%, -0.7%, -0.7% after; 100,000 tokens -5.0%, -12.8%, -0.6% before and -5.1%, -4.0%, -0.8% after. Short prompts
are unchanged (one-stream code +25% and chat +21% before and after). The first request after a server start still
loses 5% to 9% at 64k and above.
equal in every cell. Other families' A/B on current main: M4 Pro, quiet machine, LFM2.5-24B-A2B 4-bit without a drafter, 4 streams, 4 alternated rounds: every head within 0.5% of main, tokens equal. Nemotron-3.5-Lightning 4-bit, 3 rounds: within 1.3% of main at 1 and 4 streams. M5 Ultra, main
8c36359d5against the lane head6539b72be, distinct prompts, greedy 256 tokens, fresh server per cell, 3 rounds alternated, tokens equal in every cell: Qwen3.8-27B with DFlash2 at 4 streams 321.2 against 319.0 tok/s (-0.7%; the sign flips between rounds) and 1 stream +0.1%; Qwen3.8-27B without a drafter -0.1% at 1 and 4 streams; Qwen3.5-2B at 4 streams +0.07%; Nemotron-3.5-Lightning at 1 stream -0.2% and at 4 streams -1.6% (rounds -3.3%, +0.1%, -0.9%). Qwen3.8-27B + DFlash2, 4 streams, 6 rotated rounds: main 322.0, a second build of the same main commit 318.8 (-1.0%), held 323.1 (+0.3%), lane 318.9 (-1.0%) tok/s (median of round medians). The two builds of identical source differ as much as main and the lane head do, so the gap is build or session variation (spread inside one cell 304 to 333). Only the LFM2 DSpark drafter setsshared_guard, so for the other families on Metal the guards never run.Status (2026-10-26 13:25): draft. The branch history is final. Host gates and failing-test receipts below are on the
final commits. Cross-family numbers: M5 Ultra, three rounds on binaries built from the final commits (complete). M4 Pro: the stack head only (Nemotron, LFM2 without drafter), 12 of 12 sessions. Tokens equal main everywhere. The M4 was shared with other builds (1-minute load 2.7 to 4.0 at session
starts); main's own spread there reaches 21%. Nemotron's medians are -1.0% to +7.6% of main's, inside those spreads. LFM2 without the drafter is 2.1% to 5.2% above main in every cell, and in code1 and d1 every head round is above every main round. With every flag off this is not expected; it is not explained and is not claimed as a gain.
Open point (M5, Qwen3.8-27B with DFlash2, 4 streams): main ran at 310.1 to 310.7 (chat4) and 331.9 to 332.8 tok/s
(d4) in all three rounds. The branches do not: chat4 sessions land either near 310 or at 295 to 300 (l1, l2, l5 one
round each, l5on all three), and d4 medians are 1.8% to 2.7% lower on l1, l3, l4 and l5, 0.2% lower on l2 and 1.5%
higher on l5on. code4 is 0.3% to 1.5% higher on every branch. L1's code is not on Qwen3.8's path (it does not use
Drafted), and L5 with the flag off adds one branch per copy plan, yet both show it. So the cause is not yet known.It could be the binaries' layout or the sessions. Tokens are equal everywhere. Not yet resolved: needs more rounds
and a rebuild of main against the same code.
What this changes
With granted depths and the shared-round guard on a timed engine, a lone stream's rounds of held drafts are measured
in stretches of 16 against plain rounds' tokens a ms (kept by sampling mode and context band, copy rounds left out).
Drafting goes on while it beats plain by 5%; else the stream runs plain rounds and measures drafting again after 16
plain rounds, doubling to 512. Off unless the drafter sets both
Facts.chancesandFacts.shared_guard. Nothing onmain does; it lands with the LFM2 DSpark drafter PR.
Receipt
c0c7bd24f. Zig 0.17.0,-Dcpu=apple_m1 -Dversion=1.1.4; on the M5 also-Dmetal-std=metal4.0(main and branch alike). M5 Ultra (64-core GPU, macOS 27.0) and M4 Pro (20-core GPU,macOS 26.5.1). Checkpoints: mlx-community/Qwen3.5-2B-MLX-4bit@93760be4f1f6,
TensorFold/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-MLX-4bit@d9d758fb8395, TensorFold/Qwen3.8-27B-MLX-4bit@22d8d538154e
with z-lab/Qwen3.8-27B-DFlash2@50307d4c4cde (
--drafter), LiquidAI/LFM2.5-24B-A2B-MLX-4bit@667408d494bd (nodrafter). Server:
serve DIR --prompt-cache-gib 0. One model process a machine, behind a machine lock.code1/code4/chat1/chat4:tools/bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4 --alone --serial(main's client), aggregate tok/s, the same prompt on every stream.d1/d4: 1 and 4 streams with adifferent prompt on each (3 reps), then each prompt alone and with
"draft": false; aggregate tok/s, median ofthe 3 reps. Three rounds, the build order alternated (forward, reversed, forward). "tokens = main" compares the
replies'
token_sha256with main's."draft": falsecheck equal.Host tests: a lone stream whose drafted rounds lose runs mostly plain, one whose rounds win keeps drafting; replies
equal one-token decode in both.
series/lanes(8d22501d8; its tree equals the gatedadff28738), on the M4 Pro, mainagainst it (all flags off), three alternated rounds:
263 pass, 1 fail, "a lone stream whosedrafted rounds lose runs plain rounds and probes now and then" (
guarded.holds * 3 < guarded.rounds). With it:264/264.
Lean: 433 problems, 433 on main, 0 new.
Not measured
receipt measures it.
Drafted) and the CUDA drafters: no cached checkpoint here.test-cuda-host,check-native-cuda-host,compile-checkpoint-consumers,hip-host-test,test-qwen35-hip-host,check-native-hip-hostwith-Dhip=true). Linux CI not run.zig build test(the full suite with Metal device checks) andkernel-contract: not run (no kernel changes).test-core-laneran once on main and on the whole series (M4 Pro): the same PASS line, 41/41 steps.bench_openai.py, sampled decoding, cold prefill (no prompt path changes), M1 to M3.Related PRs
Sits on #1381, which sits on #1379. The DSpark scheduled PR (#1392) sits on it.