[jschmied]

Part of #1156 (was 0566be8a there). Stacked on #1356; only the last commit is new here.

TF_QWEN_KV_SHARE=n: the prompt cache keeps KV as pool pages on the device instead of host copies. It is off by default. Changed in the port to current main:

• marks are rounded down to the 4096-row prompt chunk grid, so resumes stay cold cuts;
• the page size must divide 4096;
• with the share on, every stream runs on pool pages (as with TF_QWEN_KV_PAGES_ALL), and the reserved window stays idle;
• the logic sits in cuda_prefill.zig and cuda_prefix.zig.

Gates on GB10: Qwen3.6-35B-A3B MLX-4bit with --drafter (its MTP layer) and Qwen3.8-27B MLX-4bit with DFlash2, bf16 and fp8 KV, --parallel 8, 16384, TF_QWEN_KV_PAGES=256, 37 greedy outputs per arm:

gate	35B bf16	35B fp8	27B bf16	27B fp8
flags unset == main	37/37	37/37	37/37	37/37
PAGES == main PAGES (solo, resumes)	22/22	22/22	22/22	22/22
SHARE == main all-pages, no cache	37/37	37/37	37/37	37/37
SHARE resumed == fresh (8192, 8292, 8448, 4796)	12/12	12/12	12/12	12/12
drafted == plain (SHARE)	37/37	37/37	37/37	37/37
concurrent == solo 1/2/4/8 (SHARE)	15/15	15/15	15/15	15/15
lane pages at idle, first == last	yes	yes	yes	yes

On main 21db2c4c9, zig build test -Dnvcc=/usr/local/cuda/bin/nvcc exits 0, and lean shows 0 new.

Warm second-turn TTFT at 735a7f20a, 9k-token chat, fresh servers alternating off/on, 3 pairs (both resume at 8192), bf16 KV:

	35B + MTP	27B + DFlash2
turn 2, share off	0.206–0.207 s	0.736–0.740 s
turn 2, share on	0.207–0.211 s	0.740–0.744 s
cold turn 1, share off	1.75–1.79 s	7.36–7.38 s
cold turn 1, share on	1.68 s	7.16–7.18 s

Turn-2 outputs are equal off and on (same hash in all 6 runs per model).

Since main 084333955 (#1373) paged prompt chunks run on the tensor-core tiles, so the share no longer costs prefill time. Turn 2 is level with the share off (within 5 ms), and cold turn 1 is 3–6 % faster with it on. The earlier numbers (turn 2 0.69–0.78 s and 2.19–2.22 s, turn 1 4.7–5.4 s and 15.7 s) came from the scalar paged attention.

[jschmied]

Follow-up: fault injection on the share's failure paths found two holes; both are fixed in a new commit on this PR, "cuda qwen3_5: TF_QWEN_KV_SHARE's pages through its failure paths".

The new tests (cuda_prefix_fault_test.zig, zig build test-qwen35-share-faults-host) run the real Pages and Prefix on the fake CUDA device (paging_test_device.zig). That device now holds several pinned buffers, copies to the host, and can fail the nth stream wait. After every failure the tests assert that no page is leased or promised, no page is held twice, and no lane is open. At the end they assert that once the store closes, every page is free and every pinned copy is gone. A double free would already panic in the pool.

• Page adoption failure, and every other allocation in a resume: each allocation fails in turn. This found the first hole. Prefix.open had begin retain the resumed pages, but when onGrid failed (an allocation, whenever a grid cut above the resume is still to be kept) it returned before openWith took them, so every adopted page kept one reference forever. The pool's idle census does not see this, because a page the store holds twice still counts as "kept". Fixed: those pages go back right there.
• Lease failure inside openWith(): no free lane (LaneMemory), and a promise that fails even after reclaim (the store cannot evict pages the resume holds). Both settle.
• Recurrent-state restore failure in Prefix.open(): every wait of a resume fails in turn, including the adopted pages' table upload and the state put back. Both settle.
• Cache release: Pages.close after each case, then Prefix.deinit; every page comes back.
• Second hole, in keep(): Pages.swap moved the lane's references to the store's pages before uploading the table. A failed upload left the device table pointing at pages the lane had already given back, and the pass carried on (keep errors are only logged). Fixed: the table is uploaded first. If that fails, swap returns QwenLaneTable and the stream ends with the lane still holding its own pages. A test injects that failure.

Stream sync in keep(): a 9k-token turn (8981 tokens, share on, fresh server, 2 runs per model) calls keep once, at 8192, and the wait there took 0 µs on 35B-A3B + MTP and on 27B + DFlash2. The chunk's forward has already waited for its last row's logits, so the stream is idle by the time keep runs. That is 0 % of TTFT (cold turn 4.76 s / 15.8 s; warm turn 0.70 s / 2.21 s, resumed at 8192). I kept the wait as the ordering guarantee and left it in place.

Gates (35B-A3B + MTP and 27B + DFlash2, bf16 KV, --parallel 8 at 16384, TF_QWEN_KV_PAGES=256 TF_QWEN_KV_SHARE=8): share == the original stack's share and == every stream on pages without share (37/37 each, which includes the resumes at 8192, 8292, 8448 and 4796); drafted == plain (37/37); concurrent == solo (15/15); no lane page is held at idle. zig build test passes, and lean shows 0 new problems.

[ashhart]

Review is on #1355, covering the whole stack #1355–#1359.

[BobClawblaw]

Tested at f3d545559 (with the failure-path commit) on an RTX 4090 under Windows (24 GB, sm_89, CUDA 13.3): Qwen3.8-27B MLX 4-bit with DFlash2, --context 8192 --parallel 3, against base 21db2c4c9, fresh servers, 8 greedy chats plus a 6,073-token two-turn chat.

gate	bf16	fp8
flags unset == base	27/27	27/27
TF_QWEN_KV_PAGES=256 == base paging (where the base answered; it faults, see #1355)	16/16	16/16
TF_QWEN_KV_SHARE=8 == all pages, no cache	27/27	27/27
SHARE resumed turn 2 (4,096 of 6,109 cached) == the same turn uncached	yes	yes
drafted == plain (SHARE)	8/8	8/8
three at a time == alone (SHARE, and all pages)	8/8	8/8
three at a time == alone, TF_QWEN_KV_PAGES=256 (first stream contiguous)	7/8	5/8

The last row is #1344's split: a batch with a paged stream takes the partial-only attention while a lone first stream takes the tensor-core path, and which near-ties part depends on how requests share rounds (6/8 on an earlier run of this stack at a1804ab58). All-pages and SHARE batches, where every row takes one path, are 8/8. The earlier head a1804ab58 gave the same replies as f3d545559 on 215 of 216 across these arms; the one difference is in that paged batch row.

Timing, one stream: at this size the server has no room for a prompt cache on 24 GB ("no prompt cache" at startup), so SHARE is the only way this card resumes a turn. Turn 2 of the 6k-token chat: 1.34 s with SHARE (4,096 resumed) against 3.2 s with all pages and no cache (fp8: 1.40 s against 3.3 s); cold turn 1: 8.3 s against 3.2 s (fp8: 10.3 s against 3.3 s), the slower paged prompt path you describe.

Not run: the 35B with MTP, more than three streams, windows past 8,192 (none fit beside the drafter with whole windows on this card), and the share's injected failure paths on the device.

[jschmied]

Agreed to hold this until paged prompt chunks have a tensor-core path. Here is the memory receipt you asked for, plus what changed on the branch.

Changes on the branch (no change to the share's code path):

• It now sits on cuda qwen3_5: a paged prompt chunk's attention goes through most_verify rows at a time #1355 and cuda qwen3_5: an MTP stream's ring and state are part of its pool lane's run (#1156) #1356's new heads.
• The refusal of --learn-conversations with TF_QWEN_KV_SHARE moved here from cuda qwen3_5: --learn-conversations keeps the 27B's and 35B's states on disk (#840) #1358, which no longer depends on this PR. The share is the reason for the refusal. cuda qwen3_5: --learn-conversations keeps the 27B's and 35B's states on disk (#840) #1358 and this PR carry the same engine_api build lines, so they merge cleanly in either order.
• The host step test-qwen35-share-faults-host moved a few lines down in cuda_family_host.zig, so it no longer touches cuda qwen3_5: --learn-conversations keeps the 27B's and 35B's states on disk (#840) #1358's hunk.

Memory, share off vs on, --parallel 8. GB10, Qwen3.6-35B-A3B MLX-4bit + MTP, --context 16384, TF_QWEN_KV_PAGES=256; on: TF_QWEN_KV_SHARE=32. One exclusive acquire, a fresh server per arm. Load: 8 conversations at once, 3 turns each, ~5.9k-token prompts with distinct prefixes, 64 new tokens per turn. Sampled every second during the load:

• Device memory (nvidia-smi, per process): 25,837 MiB at idle in both arms; peak 25,849 MiB off and 25,853 MiB on. It does not move, because the pool's pages are allocated at startup. The share's kept rows take pool pages instead: 8 states × 16 pages of 256 rows, about 0.6 GiB of the 4.85 GiB pool. A promise reclaims them when streams need the room.
• Pinned host memory (process RSS above idle; both prompt caches keep their copies in cuMemHostAlloc buffers, and nothing else grows): peak +1,181 MiB off and +542 MiB on. Per kept 4096-token state that is about 148 MiB off (KV rows plus the recurrent state) and 68 MiB on (the recurrent state only).
• Both arms resumed every turn-2 and turn-3 request at 4096 (16/16).

So at --parallel 8 the share saves pinned host memory: the KV rows' copy, about 80 MiB per kept 4096-token state on this model, and 0.62 GiB in total for this load. It does not save device memory. On GB10 host and device are the same physical memory, so the saving is real, but it is a small fraction of a 25 GiB process.

Sizing caveat found while measuring. With TF_QWEN_KV_SHARE=8 and the same 8 concurrent conversations, nothing resumed (0/16). In a sequential run every conversation resumed. The likely cause, from reading Store.reserve (not yet confirmed with a counter): each new pass makes room for its marks up front by evicting the least recently used state, and here that is another live conversation's. So n has to cover every live conversation's kept states plus each running pass's reservation; 32 resumed everything. If this goes ahead, n should default from --parallel, or the reservation should spare states whose conversations are still live.

Your other two points:

• The synchronize in keep(): measured at 0 µs on both models on a 9k-token turn, because the chunk's forward has already waited for its last row's logits. It stays as the ordering guarantee.
• Tests for cuda_prefix.zig: cuda_prefix_fault_test.zig (zig build test-qwen35-share-faults-host) drives the real Prefix and Pages on the fake device through failed allocations, lanes, promises, uploads and copies. It found and fixed two leaks in this PR.

zig build test passes and lean shows 0 new problems.

[jschmied]

The tensor-core path this was held for is up as #1418: a one-part paged prompt chunk now runs on paged twins of the contiguous prompt tiles, bit-equal to contiguous. Measured with this PR stacked on it (27B, TTFT s): 8k 6.38 / 6.50 with TF_QWEN_KV_SHARE=8 vs 6.98 / 6.52 on main contiguous, a warm turn 3.17 / 3.17 vs 3.26 / 3.23. The 3.4× warm and 2–3× cold cost is gone; numbers for the 35B are in #1418.

[jschmied]

Rebased on main 3f144ccf8 (head 735a7f20a), stacked on #1356 as before; #1355's commits are gone. Since 084333955 (#1373) a paged prompt chunk takes the tensor-core tiles, which is the path this PR was held for, so the share-on TTFT rows in the description (measured on the scalar paged partials) are stale. I'll re-measure them on this head and update the table. Lean 0 new; zig build test passes apart from the two FSR arena tests that b30d5b9d3 fixes on main.

[jschmied]

Re-measured on this head (735a7f20a) and updated the TTFT table in the description. With paged prompt chunks on the tensor-core tiles, the share is level on a warm turn (35B 0.206 → 0.209 s, 27B 0.738 → 0.742 s) and slightly faster on a cold one (35B 1.77 → 1.68 s, 27B 7.37 → 7.17 s). Turn-2 outputs are equal with the share off and on.

[ashhart]

Thank you for the memory receipt, the fault tests and the re-measurement. With #1373 in, the reason we held this is
gone, and the two leaks the fault tests found were worth finding. On read the share is exact. Resumes sit on the
4,096 grid and pages divide it, so adopted pages are never written again, and swapped pages hold the same rows.

Two things to settle, here or in a follow-up. First, the sizing caveat you found: TF_QWEN_KV_SHARE=8 with 8 live
conversations resumed nothing. Please default n from --parallel, or warn at startup when it is below a few times
the stream count. Second, #1335 makes chat requests carry chunk starts. The share then applies only to raw
completions, because it requires stream.chunks.len == 0. Whichever of the two lands second needs to handle that.

The gate table in the description is from main 21db2c4c9. We'll rerun the exactness gates on this head with
TF_QWEN_KV_SHARE=32.

[jschmied]

Thanks. Pushed 968c177bc.

• Sizing. I went with the startup warning rather than deriving n from --parallel. TF_QWEN_KV_SHARE=n stays what it says, the most snapshots the store keeps, and nothing is sized behind it. A default from --parallel would need a second meaning for the variable (on with auto size vs. an explicit count), and the right multiple depends on how many conversations come back, which the server doesn't know. When n < 4 x --parallel, the server now logs one line at startup with the value it should be at least. With 8 streams that is 32, the value you plan to gate with. fewSnapshots has a host test (8 and 31 at 8 streams warn; 32 and off don't).
• qwen3_5 cuda: a chat's next turn resumes at the previous turn's history (message starts for the planned cache) #1335 chunk starts. Agreed. The share requires stream.chunks.len == 0, so with qwen3_5 cuda: a chat's next turn resumes at the previous turn's history (message starts for the planned cache) #1335 in, chat requests would skip it. Whichever lands second handles it, and I'll take it as a follow-up if this one is second.

Checks: lean 0 new. zig build test passes except the two FSR arena tests, which fail because this base (3f144ccf8) predates main's b30d5b9d3 fix.

[ashhart]

Thanks @jschmied. Our GB10 run on the 27B: the default path is clean (0 of 20 prompt rows differ, sweep within ±0.4%), and with TF_QWEN_KV_SHARE=32 every reply is still exact (alone 24/24 and serial 8/8 at every level, chat text equal to main's). One real problem in the opt-in mode, though: at 8 streams the aggregate fell from 87.5 to 51.0 tok/s.

The cause: with the share on, attach sets paged_only, so every stream runs on pool pages and the engine's own window sits idle, but the pool is still built with --parallel - 1 lanes and fits refuses a stream once they're all taken. So --parallel 8 serves 7 at once: in every 8-stream rep one request waited a whole reply (TTFT 20.5-23.1 s against at most 0.95 s on main), and the --parallel 2 chat cell ran its two requests one after the other. Main's TF_QWEN_KV_PAGES_ALL check mode behaves the same way, and the share reuses it.

Could you give the pool --parallel lanes when the share is on (or the engine's window back to a stream)? A host test that --parallel N admits N streams with the share would pin it. #1356 is landing now without it; #1357 follows once that's in.

[jschmied]

Thanks for catching it. Fixed and pushed as 7db39e45b, rebased onto main now that #1356 is in.

Fix. When every stream runs on pages (TF_QWEN_KV_SHARE, or main's TF_QWEN_KV_PAGES_ALL checks; both set paged_only in attach), the pool now gets --parallel lanes, not --parallel - 1: the idle window's stream gets a lane in the pool. The page count is still the plan's, and fits still admits a stream only when its pages can be promised. I didn't hand the engine's window back to a stream, because then a stream's bytes would depend on the lane it got, and a stream on the window can't adopt kept pages.

Host test. cuda_admit_test.zig, in test-qwen35-share-faults-host, attaches the pool the way the server does at --parallel 2 and 8 (parallel - 1 streams, the plan's pages) and opens streams while fits admits them. With the share on, and again with TF_QWEN_KV_PAGES_ALL=1, it admits N. Before the fix it fails with expected 2, found 1.

Served, GB10. Qwen3.8-27B MLX-4bit + DFlash2, bf16 KV, --parallel 8 --context 16384, TF_QWEN_KV_PAGES=256, share off vs TF_QWEN_KV_SHARE=32. Fresh server per arm in ABBA order, one exclusive acquire each; the binary was built from 66399260a, which differs from the pushed head only in comments. Each arm runs 8 distinct prompts alone, then 3 rounds of all 8 at once, streamed, 256 new tokens each, greedy:

prompts	arm	aggregate tok/s (3 rounds)	max TTFT
short (~15 tokens)	off	173.3–174.9	0.654–0.661 s
	on	174.0–175.8	0.659–0.664 s
1–2k tokens	off	76.0–77.2 (first, cold server), 84.8–85.0	12.2–14.1 s
	on	81.3–84.5	12.2–13.1 s

• With short prompts a round's wall time is 11.7–11.8 s, so a request waiting for a lane would show up as a TTFT near 11 s. The worst across all 24 requests is 0.664 s, level with the share off.
• With the longer prompts, TTFT rises in steps of about 1.5 s from 1.0 to 12–14 s in both arms. That is the eighth prompt waiting behind the other seven's prompt passes, not a wait for a lane, and the arms match.
• Replies exact: every concurrent reply equals its own run alone (24/24 per arm, all 8 arms), and the solo and concurrent texts are the same in every arm, off and on.

One cosmetic thing is left: the startup line still describes the plan ("the engine's own at a 16384-token window, 7 more taking 959 pages"). With the share on, all 8 run on those pages. I can change the wording here or in a follow-up, whichever you prefer.

Checks on main db3fd5aba: zig fmt --check zig, lean 0 new, zig build test -Dnvcc=/usr/local/cuda/bin/nvcc RC 0 with 0 errors.

[ashhart]

Thanks @jschmied. Giving the pool a lane for each of --parallel's streams when every stream runs on pages is the fix
we hoped for. The default path is untouched, and fits still admits a stream only when its pages can be promised. In
your runs with all eight at once there's no lane wait, and every reply equals its run alone.

One thing breaks: the new test doesn't link on Windows. zig-windows fails with lld-link: undefined symbol: setenv
and unsetenv, called at cuda_admit_test.zig:51-52 through the extern "c" declarations at the end of the file.
Main's @import("sys").env has both on every platform (core/heat.zig:198 and fabric/mcdma.zig:13 use it), so
switching to it should fix the link. The desktop failure is the known hub-endpoint.test.ts clean-up race, not this
PR.

On the startup line: since you'll push for the Windows fix anyway, please fold it in here. With the share on, it
should say all of --parallel's streams run on the pool's pages.

Then we'll rerun on a GB10: the 27B with DFlash2 and the 35B with MTP, at --parallel 8 with TF_QWEN_KV_PAGES=256.
With the flags unset we'll compare against main by token hash. With TF_QWEN_KV_SHARE=32 we'll check:

• the eight-stream aggregate against the share off;
• TTFT;
• concurrent against alone;
• resumed against fresh at the grid edges.

[jschmied]

Thanks. Both are in 65ef4ffcc, a fast-forward on 7db39e45b.

• Windows link. cuda_admit_test.zig now sets and clears TF_QWEN_KV_PAGES_ALL through @import("sys").env, and the extern "c" declarations are gone. zig build test-qwen35-share-faults-host -Dtarget=x86_64-windows-gnu builds and links here.
• Startup line. With TF_QWEN_KV_SHARE or TF_QWEN_KV_PAGES_ALL, it now reads "all N on P pages of R rows (… GiB) as they need them, the engine's own window idle, … MiB fixed each". The family reports this through a new all_on_pages field on its Loaded, read like own_prompt_cache. cuda_kv_pool.zig's startup test pins the new wording, and the default wording is unchanged.

Checks: zig fmt --check zig; lean 0 new; zig build test -Dnvcc=... RC 0. No device run on this head: the change is the test's environment calls and the startup text.

[ashhart]

Thanks @jschmied. Both are in on 65ef4ffcc. We've queued the GB10 cells with Qwen3.6-35B-A3B with MTP at --parallel 8:

• With the flags unset, main against this branch, token hashes and speed.
• With TF_QWEN_KV_PAGES=256 on both builds and TF_QWEN_KV_SHARE=32 on this branch: the eight-stream aggregate against the share off, TTFT, concurrent against alone, and resumed against fresh.

They run once the box finishes its kernel contracts, and the results will be here.

[ashhart]

Thank you @jschmied. This is in TensorFold 1.2.0, released today: https://github.com/ashhart/TensorFold/releases/tag/v1.2.0

It landed with your commits under your own name, so you're listed as a TensorFold contributor, and it's credited to you in the release notes and the changelog.