[BobClawblaw]

Follows #1344. With TF_QWEN_KV_PAGES, any batch holding a paged stream ran every row's attention through the
partial-only kernels (tf_q35_attn_partial_paged + merge), the prompt chunks included, while whole windows take the
tensor-core tiles (tf_q35_attn_partial/attn_rows, hpattn_kernel). The two paths round differently, so a stream's
reply depended on whether a page table was in play: beside other streams on pages it differed from its solo reply
(6-8 of 12 on an RTX 5090).

What changes

• qwen35_attn_tc.cuh: Bf16Tiles and Fp8Tiles take the key head and address rows by cache position; with sg.pt
  set they read through the stream's page table (TfQ35PageRun, the layout qwen35.cu/qwen35_kv8.cu write), else
  the slab as before. attn_one and attn_rows pass positions, not slab rows.
• prefill_attention_rows.cuh: the head-major mode (HM) reads pages when PT is set (PTOK rows a head, PS
  apart, in the cache's units); hpattn_kernel/hpattn8_kernel take (PT, PS, PTOK), and cuda_prompt.zig passes
  a paged stream's table with the layer's block of a page as the base (new mangled symbols pinned there).
• cuda_forward.zig: the if (shape.paged) branch is gone; paged and whole batches take the same launches.
  cuda_attention.zig keeps hasPages (the graph key) and loses the partial-only launcher and its fake-engine test.
• measurements/qwen_lanes_133.zig: the four tensor-core launches' nvcc 13.3 receipts, re-measured from five-SM
  fatbins (zig build fatbins kernel-contract-fatbins -Dsm=80,86,89,120,121, the limits runner's RECORD lines). The
  page-or-slab addressing costs registers: tf_q35_attn_rows/attn_rows8 255 on every SM (from 254/246 and 255/245),
  tf_q35_attn_partial 204 below sm_120 (168 there, unchanged), attn_partial8 218/218/228/206/206 (from
  199/199/207/197/197); cuobjdump --dump-resource-usage shows no stack or local memory for any of them on any SM.
  The 13.0 base records of these launches (cuda_lane_pending.zig, used below nvcc 13.2) are from before this change;
  no nvcc 13.0 here to re-measure them.
• contract_qwen_pages.zig also runs the tensor-core decode kernels and the prompt kernel on pages against contiguous
  (bytes equal, the contiguous results within the float64 bound, the swapped-table control red for them too).

Checked on an RTX 5090 (CUDA 13.3, driver 615), Qwen3.8-27B MLX 4-bit + DFlash2, fp8 caches, int8 prompts, main
at 31a864f plus this change, the suite's 12 snapshot prompts, replies compared by token ids:

• one stream at 32k, TF_QWEN_KV_PAGES=256 TF_QWEN_KV_PAGES_ALL=1 against the same build's whole window: 12/12
  identical (before the change, on 1.1.5: 7/12 differed; the earlier diagnosis in Qwen3.8-27B CUDA: with TF_QWEN_KV_PAGES a batch takes the partial-only attention path, so its replies differ from the same replies alone (1.1.5, RTX 5090) #1344)
• the same build's whole window against v1.1.5's: 12/12 identical (the unpaged path is untouched)
• 8 streams at 65,536 on pages (TF_QWEN_KV_PAGES=256): together == alone 12/12 (1.1.5: 6-8/12)
• 8 streams at 8,192 on whole windows: together == alone 12/12
• tf-cuda-test qwen-pages: bf16 and fp8 pass at every case (paged == contiguous for the partial, tensor-core and prompt rows; rope stores and keep; the swapped-table and planted-row controls red)
• zig build kernel-contract -Dsm=120: 340 kernels pass gates 1-4, 94 controls red as planted, 5 known findings and 161 limit problems, the same set as main's at 31a864f on this card (the awaiting launches and 13.3's live register counts against older records); the four re-measured launches read record current
• speed, through the recipe harness on the same card, the fix build in place of v1.1.5 (decode tok/s prose/code/JSON/list,
  one stream, the suite's prompts):
  • production flags (262k whole window, one stream): 278.2/537.2/352.5/337.3 against 1.1.5's 278.3/538.1/352.6/337.3
    (the unpaged path takes the same launches, the tiles now choosing slab or page addressing; the harness's structured and tool cases were not run against
    this build, which has no grammar library beside it)
  • 8 streams at 65,536 on pages (TF_QWEN_KV_PAGES=256): 277.4/536.3/351.5/336.1 against 1.1.5's 274.5/538.6/352.4/336.8;
    the 12 prompts together == alone 12/12, drafted == plain 12/12; 30,744 MiB idle

Required checks (zig build test, test-kernel-contract, test-golden, test-server-security test-pull-security, fuzz-cortheon, zig fmt --check build.zig zig, the lean check): all pass; lean 0 new

Rebased onto 40c6406 before the push (no overlap with the commits in between); the required checks and tf-cuda-test qwen-pages were run again on the rebased tree and pass. The card measurements above are from the change on 31a864f.

[BobClawblaw]

Tested on a second architecture: RTX 4080 SUPER (sm_89, 16 GB), Windows 11, CUDA 13.3, built with -Dsm=89. Compared main eea9dc9c9 against this PR at c2660b99.

#1344 reproduces on sm_89, and this PR fixes it. The test runs bench_concurrent --levels 4 --tokens 300 --reps 3 --temperatures 0 --alone --serial at --parallel 4 and counts how often four streams at once match the same prompts run alone and run serially:

model / kv	layout	main	#1373
Qwen3.5-9B 4-bit, bf16	whole windows	12/12	12/12
Qwen3.5-9B 4-bit, bf16	TF_QWEN_KV_PAGES=256	0/12 alone, 0/12 serial	12/12, 12/12
Qwen3.5-9B 4-bit, fp8	whole windows	12/12	12/12
Qwen3.5-9B 4-bit, fp8	TF_QWEN_KV_PAGES=256	9/12, 0/12	12/12, 12/12
Qwen3.8-27B oQ2, fp8 (--parallel 2)	TF_QWEN_KV_PAGES=256	0/6, 3/6	6/6, 6/6

Speed (9B, --levels 1,4 --tokens 256): whole-window numbers are the same on both builds. Paged runs slightly faster with this PR: 393/375 tok/s against 383/365 on main.

Suites:

• tf-cuda-test qwen-pages: 22/22 on both builds.
• The single-GPU synthetic suites all pass on main.
• kernel-contract on both builds: all 330 kernels pass gates 1-4, and all 90 planted controls are caught. Main reports 120 limit problems and this PR 121. In every one the launch fits the driver's limit; only the register count compiled for sm_89 differs from the recorded table (e.g. glm_mm16_16 94 vs 95, tf_experts3_widen 12 vs 34). The extra one on this PR is tf_nemo_router_tiles (194 vs 184), which this PR does not touch. So the limits table looks out of date for sm_89 / CUDA 13.3. That is separate from this PR.

[BobClawblaw]

Also checked on an RTX 4090 (24 GB, sm_89) under Windows, CUDA 13.3 (nvcc 13.3.73, MSVC host), driver 616.56, at c2660b999.

Served, Qwen3.8-27B MLX 4-bit + DFlash2, --context 8192 --parallel 3, fresh servers, 8 greedy chats (24 to 4,616 prompt tokens, thinking off, 160 tokens) sent alone, with "draft": false, and three at a time:

	bf16 KV, bf16 prompts	fp8 KV, int8 prompts
TF_QWEN_KV_PAGES=256: three at a time == alone	8/8	8/8
TF_QWEN_KV_PAGES=256 == whole windows (every reply)	24/24	24/24
drafted == plain, paged and whole	8/8	8/8

Before this change, the same harness with TF_QWEN_KV_PAGES=256 gave 6/8 and 7/8 (bf16) and 5/8 (fp8 KV) three at a time against alone on this card.

zig build kernel-contract -Dsm=89: SUMMARY 330 kernels: 330 pass gates 1-4, 0 fail, 5 known findings; 90 controls: 90 red as planted, 0 missed; 120 limit problems. The qwen35 pages cases pass on all ten layouts in bf16 and fp8 (paged == contiguous for queries, rows, stores and keep, tensor-core and prompt rows; repeats, the float64 bound and guards), with both controls red. Of the 120 limit problems, none is in a launch this change records: the live check reads tf_q35_attn_partial 204, tf_q35_attn_rows 255, tf_q35_attn_partial8 228 and tf_q35_attn_rows8 255 registers, as recorded. The rest are records this base does not yet hold for nvcc 13.3 (#1397's run on this card, with them, has 43), the pending tf_q35_attn_partial*_paged launches that sit on main as they do here, and the awaiting 1024-thread launches.

tf-cuda-test kernel-contract-limits on five-SM fatbins built here (-Dsm=80,86,89,120,121): your re-measured launches read "record current" on every SM at exactly your numbers: tf_q35_attn_rows and rows8 255 on all five, tf_q35_attn_partial 204/204/204/168/168, tf_q35_attn_partial8 218/218/228/206/206. The runner stops later with SymbolNotInImage at dsv41_x3ld, whose records name the Linux mangling (PKl, Windows has PKx); that is on main too (see #1397).

zig build test-kernel-contract test-golden passes; zig fmt --check build.zig zig passes; lean against 40c6406b3: 442 problems, 442 on the base, 0 new.

[ashhart]

Landed on main as 0843339, in 1.1.7, fixing #1344. On a GB10 with Qwen3.8-27B, prompt tokens equal main and prefill and the stream sweep are unchanged (within 0.8%). Thank you!