Repository navigation
spark25 cuda: TF_KV_SHARE keeps prompts as the pool's pages in the prompt_radix store (on #1653) - #1756
Merged
Conversation
jschmied
force-pushed
the
kv-share-radix-spark
branch
from
October 31, 2026 16:20
f2893a5 to
20afc28
Compare
… pool's pages, in one prompt_radix store A family on the core KV pool that declares share_pool and share_attach gets one prompt_radix store over that pool (native/cuda_prompt_pages.zig): the tree holds the prompt pages themselves, a page edge's state past them comes from the family's snap_* decls, and n caps those states. The host makes no prompt_cache store then, and the cache budget becomes pool pages (Budget.kept_ppm). core/kv_prompts.zig is what the family calls: begin (resume point, adopted pages, marks), keep at a mark, report, and Blob, a state's copied rows in pages the store holds. Pool.hold takes those pages, asking the store to reclaim first as promise does. Unset or 0: as before.
…tate's rings in store-held pages With the shared store a prompt pass begins at the store's resume point: the matched pages adopted into the stream's lease, its sliding rings restored there from the kept state. Chunks cut at the store's marks (page edges, 256 tokens); at each the store takes the pages it lacks, the stream's copies swapped for the tree's, and the state's rings go into pool pages the store holds (snap_bytes counts those). Unset: as before. tf-cuda-test spark25-pages with TF_KV_SHARE=4 resumes a kept prompt plain and drafted, equal to linear.
ashhart
force-pushed
the
kv-share-radix-spark
branch
from
October 31, 2026 22:43
20afc28 to
e2edae2
Compare
Owner
|
Thank you @jschmied. This is merged into main with your commits under your own name, so you're listed as a TensorFold contributor. It ships in the next release, credited to you in the changelog and the release notes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1657: Spark2.5 on the native-CUDA
prompt_radixadapter, off by default. On main, after #1659 and #1653 (both merged).Two commits:
native cuda: TF_KV_SHARE=n ...is the adapter, the same commit as in native cuda: TF_KV_SHARE=n keeps a paged family's prompts as its core pool's pages in one prompt_radix store (Kolibri-1 first) #1755 (Kolibri-1, which stacks on kolibri1 CUDA: the full layers' caches in pages of the core pool, every stream leasing; sliding layers keep their rings #1620). Whichever PR lands first carries the adapter. native cuda: TF_KV_SHARE=n keeps a paged family's prompts as its core pool's pages in one prompt_radix store (Kolibri-1 first) #1755 explains the adapter itself.spark25 cuda: ...is the family wiring:Unset or 0: as before.
Gates (GB10, Spark-X2.6-4B 8-bit)
tf-cuda-test spark25-pages, final binaryTF_KV_SHARE=4: PASS, resume at 1280, reply and logits same as linear. Unset: PASS. The same code before the move onto main: 4 more share PASSv.sh(lean 0 new,zig build test, CI set, diff-check)Default path (
TF_KV_SHAREunset,--prompt-cache-gib 0). Base (#1653's commits without these two) vs these two on top, fresh server per arm, ABBA over 2 rounds, aggregate tok/s, greedy token_sha equal in every cell:The Gemma 4 wiring follows #1619.