[jschmied]

Part of #840, the "Spill of evicted states" row (#68, @gilby; claimed in #840, comment 6399090685). It sits on main (e9b69ed70), no longer stacked on #1357.

On CUDA, --learn-conversations wrote kept states to disk for Nemotron only, and the qwen3_5 family (Qwen3.8-27B, Qwen3.6-35B-A3B) refused the flag (cuda_refusal.zig). This PR gives the family the same disk tier Nemotron has: 256-token blocks shared between states, the core's payoff rule, --learn-gib, --learn-daily-gib, the free-disk floor, and the SIGTERM flush.

Commits

1. rows: a grid family's disk states carry the grid's chunk starts (core).
   qwen3_5 is a planned family with a 4096-row grid. A request that names no chunk starts keeps its RAM states on the grid (Store.startAt). The disk tier read the empty list literally, so every state was named as one chunk from row 0 and no two states shared a block. matches also refused every read, because the request's starts did not contain the state's position. Options.grid now gives the disk tier the grid, so such a request is written and matched with the grid's starts below the state. Families without a grid, and requests that name their starts, behave as before. New test: rows_test.zig "a grid family's requests name no chunk starts ...".
2. cuda qwen3_5: --learn-conversations keeps the 27B's and 35B's states on disk.
   The family's kept states are already host copies (cuda_reuse.zig): the recurrences and conv tails, then the key and value rows of each attention layer by head, in bf16 or in fp8 with its scales.
   • families/qwen3_5/cuda_disk.zig lays the copy out as the rows store's image. It has 16 bytes of words (position, rotary shift, cache format), the recurrent state as the state's own part, and each head's rows as row parts.
   • A copy read back is a new host copy and is restored like any other. A copy whose words name another position or format is refused.
   • native/cuda_disk.zig splits its identity hash from Nemotron's probe. A family that declares kept_disk attaches the same store. Its identity covers the checkpoint, the kernels, the device, the driver, the cache format, whether a drafter set the target's taps, and the hash of a probe state at the first chunk cut. Nemotron's identity bytes do not change.
   • A state whose prompt holds images stays in RAM, because the image identities are not stored on disk. TF_QWEN_KV_SHARE keeps prompts as pool pages without the host cache, so --learn-conversations with it is refused at startup.

What a kept state holds is unchanged from the RAM cache's copy: the KV prefix (contiguous or paged, gathered page by page) plus all recurrent state (GDN state and conv window) at the mark. The DFlash2/MTP drafter context is not part of the CUDA kept state, in RAM or on disk. After any resume, the drafter's context starts at the mark (cuda_draft.zig "a gap"). That changes acceptance only. Tokens and logits are the target's, and the drafted == plain gate below covers it.

Gates (GB10, Qwen3.8-27B MLX 4-bit + DFlash2, Qwen3.6-35B-A3B MLX 4-bit + MTP)

Each config runs these arms:

• fresh: no prompt cache.
• plain: no cache, --no-drafts.
• diskA: turn 1 of two chat conversations (about 9.6k and 4.6k tokens). The writer is allowed to settle, then the server gets SIGTERM.
• diskB: a new server on the same --learn-dir. It runs turn 2 of both conversations, read back from disk, then four more turn-2 variants solo and at c=1/2/4.

Each request records its text and a hash of the last prompt row's logits. The qwen CUDA backend has no logprobs route (first_row), so the hash comes from a gate-only log line in firstMaskedFn, which is not part of this PR. TF_IMPRINT_VERIFY=1 reads back every written state and compares it byte for byte.

Every row below matched in full (tokens and the logits hash), except where the row says otherwise.

Gate	27B bf16	27B fp8	35B bf16	35B fp8
read from disk after restart == fresh (contiguous)	2/2 + 4/4	2/2 + 4/4	2/2 + 4/4	2/2 + 4/4
read from disk after restart == fresh (TF_QWEN_KV_PAGES=256, _ALL=1)	2/2 + 4/4	2/2 + 4/4	2/2 + 4/4	2/2 + 4/4
resumed at c=1/2/4 == solo	10/10 (both modes)	10/10	10/10	10/10
fresh at c=1/2/4 == solo	20/20	20/20	20/20	20/20
drafted (DFlash2 / MTP) == --no-drafts	10/10	10/10	10/10	10/10
TF_IMPRINT_VERIFY read-back of each written state	byte-equal, 0 differ	byte-equal	byte-equal	byte-equal

Turn 2 of the 9.6k-token conversation, read from disk vs fresh: 27B bf16 1.69 s vs 7.73 s (658 MiB read in 0.40 s); 35B bf16 0.49 s vs 1.82 s (221 MiB in 0.13 s).

27B only (the core paths are the same for the 35B, so its runs were limited to the exactness gates):

• cross: written without paging, read by a paged-all server. The paged server probes another identity, so nothing is read, and the output equals the paged fresh arm (2/2 + 4/4).
• corrupt: one .blk truncated and one .st byte flipped. Each read fails its check (RowsRead / RowsHash) and its states are forgotten. The turn is recomputed and equals fresh. Nothing crashes.
• --learn-gib 0.8: the disk stays at 845 MB, then 422 MB, under the cap. The state that did not fit is recomputed and equals fresh.
• --learn-daily-gib 0.5: the second state "stays in RAM only (DailyBudgetSpent)", and its turn recomputes and equals fresh.

Turn-2 TTFT of a 29,455-token two-turn chat after a restart (ttft.py, 27B + DFlash2, 2 disk + 2 cold, alternating): 1.88 s and 1.97 s read back from disk (28,672 tokens, 1,938 MiB in 1.0 to 1.1 s), against 25.4 s and 25.5 s cold. All four runs gave the same text and the same last-row logits hash. The 35B TTFT run was skipped under the GPU-time rules, because the family's code is the same.

Not shown on the device:

• The payoff rule never declines a read on this box. The smallest qwen state is at 4096 tokens: 402 MiB, read in about 0.2 s, against about 2.4 s to prefill it on the 27B. The rule is the core's (rows_select.zig) and is covered by rows_test.zig "a read that costs more than prefilling ...".
• A server started with TF_QWEN_KV_PAGES_ALL probes another identity (paged arithmetic differs from contiguous in the last bits). States written without paging are therefore not read by a paged server: they are recomputed, which is the correct behavior (row "cross" above).

Full zig build test -Dnvcc=/usr/local/cuda/bin/nvcc exits 0. lean: 0 new.

The qwen3_5 commit carries Co-authored-by: gilby <880013+gilby@users.noreply.github.com>; the grid fix is new.

Overlap: @gilby's #1335 adds a line next to ours in qwen3_5/cuda_native.zig and cuda_native_moe.zig (a textual conflict only). With #1335 the server names message starts, so its turns already carry starts; the grid fix covers requests that name none.

[jschmied]

Follow-up: two changes on this PR:

• The disk probe now has one cleanup path. The qwen3_5 probe (families/qwen3_5/cuda_disk.zig), and Nemotron's in native/cuda_disk.zig that it was copied from, released the probe stream by hand: once after the prompt pass, and once on the pass's own error. If the stream wait after the pass failed, the probe returned without releasing the stream, and it never dropped a state the pass had kept. Now the pass and its wait run in one block whose defer releases the stream, and a kept state is dropped by a defer of its own. Commit: "cuda: a disk probe releases its stream once, whatever fails after the prompt pass". A host test in cuda_disk_test.zig drives the probe with a failing pass, a failing wait, and a pass that keeps no state; each releases the stream exactly once.
• New rows_test case for the switch between requests that name chunk starts and requests that name none, in both directions, writing a state, restarting and reading it back from disk (commit "rows: a grid family's disk states between requests that name chunk starts and requests that name none"):
  • starts named on the grid → none: resumes at 1024;
  • starts named off the grid (state at 1000) → none: no resume;
  • a state at 1024 whose lower chunks were cut at 300 → none: no resume;
  • none → named starts with the grid's below 1024 and 1024 among them: resumes at 1024;
  • none → named starts without 1024, or off the grid below it: no resume;
  • plus none → none and named → the same named.
    Every resume equals a fresh pass. If matches drops its onGrid check, the 300 case goes red. This matters once qwen3_5 cuda: a chat's next turn resumes at the previous turn's history (message starts for the planned cache) #1335 makes the server send message starts.

The Windows CI fix for cuda_disk_test.zig (platform.c.getpid, shell edits skipped on Windows) is included. The new tests use no shell and no pid formatting.

Gates (35B-A3B + MTP, bf16, contiguous and TF_QWEN_KV_PAGES=256): states written, server restarted, read back. Resumed == fresh on tokens and the last row's logits hash (turn 2: 2/2; others: 4/4; concurrent resumes: 10/10). Reads at 8192 and 4096, 0 checks failed; drafted == plain 10/10; concurrent == solo. zig build test passes on this branch and on the next, and lean shows 0 new problems.

[ashhart]

Review is on #1355, covering the whole stack #1355–#1359.

[jschmied]

Decoupled: this PR now sits directly on main (e9b69ed), with #1357 out of its history, and it builds and tests alone.

The two pieces it borrowed from #1357:

• cuda_kv_pool.ownCache: only the refusal of --learn-conversations with TF_QWEN_KV_SHARE used it. That refusal belongs to the share, so it moved into cuda qwen3_5: the prompt cache keeps KV pages on the device (TF_QWEN_KV_SHARE, #1156) #1357 (where the share is introduced), and this PR no longer names it. Whichever of the two lands second, the combination stays refused.
• the engine_api import: build/cuda.zig's family() gives the qwen35 module the engine_api module the server links, and engines() and the family host build (build/cuda_family_host.zig) take that same one. cuda qwen3_5: the prompt cache keeps KV pages on the device (TF_QWEN_KV_SHARE, #1156) #1357 carries the identical lines, so the two merge cleanly in either order (checked with git merge-tree both ways).

#1411 (wojo, GLM-5.3-Flash on CUDA). That PR carries our grid-starts commit (a63afd4 here as 7aab9c2) with the same patch and message, so it drops out of whichever PR lands second. Its identityOf has the same name, signature and hash order as ours; ours is now pub like theirs, so that hunk merges cleanly. A family module that has to open the store itself (GLM's per-rank parts) can declare attach in its kept_disk. attachKept then calls it and leaves the probe and identity to the module, which still takes identityOf from here. facts may now be an array or a slice. Our rows_test addition moved to the end of the file, so it no longer collides with the grid commit's block. What is left against #1411 is four one-line conflicts on lines both PRs rewrite:

• the image line and the attach line in native/cuda.zig;
• the --learn-conversations refusal in native/cuda_refusal.zig;
• that refusal test's name.

Each resolves by keeping both (... else if (F == glm.native) &glm_disk.image else if (comptime cuda_disk.keptDisk(F)) |K| &K.image else null, both attach lines, and F != glm beside keptDisk(F) == null). I resolved it that way locally on top of #1359's split: zig build test passes on the merge. Either can land first. Once both are in, GLM can declare kept_disk with its own attach and drop its two explicit branches.

Otherwise the commits are unchanged: grid starts (byte-identical to the original), the rows transition test, the disk image, the Windows test fix and the probe's single cleanup path.

Gates (35B-A3B + MTP, bf16, fresh servers; states written, the server restarted, then read back from disk):

• Contiguous, on this branch alone: resumed == fresh on tokens and the last row's logits hash (turn 2: 2/2, others: 4/4, concurrent resumes: 3/3 and 5/5), drafted == plain 10/10, concurrent == solo at c = 1/2/4. Reads at 8192 and 4096, 0 checks failed.
• TF_QWEN_KV_PAGES=256 with every stream on pages: on this branch alone, main's cuda qwen3_5: a paged prompt chunk's attention goes through most_verify rows at a time #1355 fault ends the server in the no-disk baseline (illegal address on a paged prompt chunk past 128 rows). Merged with cuda qwen3_5: a paged prompt chunk's attention goes through most_verify rows at a time #1355, the same gate passes with the same counts. zig build test passes on this branch, and lean shows 0 new problems against main.

[ashhart]

Thanks for decoupling this from #1357; it now builds and tests on main alone. The grid-starts fix and its transition
tests read correctly. The identity work looks sound too: the facts, a probe through the served prefill, and a single
cleanup path.

One small change: rebased on current main, zig/build/cuda.zig comes to 601 lines, which is a new lean finding.
#1357 adds the same engine_api import as a single line with a trailing comment. If you use exactly that line, this
PR stays at 600, and the two no longer conflict in family() (they do on the current heads). Two optional
points. First, add prompt_int8 to facts: the probe should already catch it, but naming it costs nothing. Second,
cuda_disk_test.zig shells out (system("rm -rf ..."), truncate, dd) under a fixed /tmp path. With
std.testing.tmpDir and the file APIs it would need no shell, and it could run on Windows too.

After that we'll run the write, restart and read cell on the 27B (MLX 4-bit and NVFP4), bf16 and fp8, including a
contiguous-to-paged read now that #1373 has made paged arithmetic equal to contiguous. The description's first lines
still say the PR is stacked on #1357; please update them to say it sits on main.

[jschmied]

Thanks. Rebased on main 59044397a and pushed as 402e64fbd; the force-push is from the rebase.

• cuda.zig at 600. It now uses cuda qwen3_5: the prompt cache keeps KV pages on the device (TF_QWEN_KV_SHARE, #1156) #1357's engine_api import exactly, one line with its trailing comment, so zig/build/cuda.zig is 600 lines and family() no longer conflicts with cuda qwen3_5: the prompt cache keeps KV pages on the device (TF_QWEN_KV_SHARE, #1156) #1357. ATC overlap shows no conflict with cuda qwen3_5: the prompt cache keeps KV pages on the device (TF_QWEN_KV_SHARE, #1156) #1357. It still shows one with qwen3_5 cuda: a chat's next turn resumes at the previous turn's history (message starts for the planned cache) #1335, which is the chunk-starts interplay already noted on cuda qwen3_5: the prompt cache keeps KV pages on the device (TF_QWEN_KV_SHARE, #1156) #1357.
• prompt_int8 in facts (optional, done). qwen3_5/cuda_disk.zig facts now returns three words: the cache format, taps, and int8.on.
• Shell-free disk test (optional, done). The restart test runs in std.testing.tmpDir. It cuts a .blk to 100 bytes and overwrites a .st byte at offset 40 through setLength and writePositionalAll. There is no system(), no fixed /tmp path, and no Windows skip. I haven't run it on Windows.

Checks: lean 0 new (428 on base and head); zig build test passes on this head.

[ashhart]

Thanks: every earlier ask is met. The one-line import keeps zig/build/cuda.zig at 600 lines, prompt_int8 is in the identity facts, and the disk test uses std.testing.tmpDir with no shell and no Windows skip. The grid correction is right too: removing the earlier-cut guard turns your transition test red (305/305 to 304/305), and restoring it turns it green.

Next is our GB10 run of the disk-restore cells on the current head: a written, restarted and read conversation equal to a fresh one at grid and off-grid positions, on main and on this branch.

[ashhart]

Main moved on under this: spill writes are now staged and written by an I/O thread (afa4fa3, cbd5305, d22d704), and rows_write.queued hands off to rows_write_sync.queued, so this branch conflicts in zig/src/core/rows_write.zig. For our device run we resolved it by keeping main's rows_write.queued and calling your rows_store.startsBelow(r, r.gpa, starts, e.at) in rows_write_sync.queued in place of its inline loop. Your two grid-family tests and the rows host tests pass on that merge.

Could you rebase onto main the same way, or say if you'd rather do it differently? The GB10 run of the disk-restore cells (written, restarted and read back against fresh, with a 16,384-token and a 14,000-token conversation) is queued on that merge.

[jschmied]

Rebased onto main 5897e0bca and force-pushed as f593a527e, resolved the way you did: rows_write.queued is main's, and rows_write_sync.queued calls rows_store.startsBelow(r, r.gpa, starts, e.at) in place of its inline loop.

One test needed the I/O thread too. cuda_disk_test's writeBoth stepped the write queue on its own thread, so on main nothing was written (expected 2 states, found 0). It now drains the write worker before it checks, as your 5897e0bca does for the persistence tests. That change is folded into 8436d6dee, the commit that added the test.

Checks: lean 0 new (428 on base and head); zig build test -Dnvcc passes, including the disk test. #1359 is restacked on this head as e018f176e, with the same checks green.

[ashhart]

Merged, with your six commits as they were, and the changelog entry follows in 8c2037c. Thank you @jschmied, and @gilby for your part. It ships in 1.1.8.

Our device run on a GB10 with Qwen3.8-27B: for a 16,384-token and a 14,000-token conversation, a server with --learn-conversations wrote the state, a fresh server on the same folder read it back after the restart (12,288 tokens cached each time, 3.9 s against 12.7 s fresh at 16k), and every reply was byte-equal to main's fresh reply. The full test suite, the Linux CUDA cross-compiles and the release path check pass on the merged tree.

[ashhart]

This is in TensorFold 1.1.8, released today. Thank you, @jschmied and @gilby!