Repository navigation
prompt_radix: a lookahead in the tree and a planned grid, so a family like GLM's can share the store - #1659
Merged
Conversation
…sume and evict Seven passes under the rules HIP's families and Qwen's CUDA path use (a page of 4 tokens here, min_gap 16, the tail kept): two conversations over a shared system block, a growing and an edited turn, a third prompt that evicts, and one past the limits. Each pass's resume point and marks, and the tree's states, pages, bytes and counts after it, are the ones the store kept before it learns a lookahead and a grid.
… like GLM's can share the store A page is shared and matched only while the prompt repeats the lookahead's tokens past it: GLM's MTP rows sit on pages, and row at-1 reads token at, so a page whose next token differs holds other rows. Each node keeps those tokens past its last page; child, common, insert and split compare and carry them, and every slice the store hands the tree grows by the lookahead. A planned store cuts and resumes only on its grid, a multiple of the page: marks snap to it, deepest skips states off it and keep refuses them. With lookahead 0 and no grid every path is as before (the golden run).
Store.init refused a zero page or a planned grid that is no multiple of the page only through std.debug.assert, so a release build would take them. It now returns error.BadPromptRadixRules, with a test for each.
ashhart
force-pushed
the
pr/radix-lookahead-grid
branch
from
October 30, 2026 23:00
9f805f5 to
b100164
Compare
9 tasks done
Owner
|
Thank you @Bizuayeu. This is in TensorFold 1.2.0, released today: https://github.com/ashhart/TensorFold/releases/tag/v1.2.0 It landed with your commits under your own name, so you're listed as a TensorFold contributor, and it's credited to you in the release notes and the changelog. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No family turns these rules on here, so nothing on main changes: HIP's families and #1357's Qwen path keep their marks, pages and bytes, and a golden run pins that. This is the half of #1510's option B that
prompt_radixitself needs, and #1657's shared adapter builds on it. What it enables, from GLM's port on these rules (TF_GLM_KV_SHARE=4, the branch on #1510) against main's flat store, on GB10 at TP=3 with the 1M window:bench_openai)What this changes
Three commits in
core/prompt_radix.zig,core/prompt_radix_tree.zigand their test file. No family code.A golden run first. Seven passes under the rules HIP's families and #1357's Qwen path use (lookahead 0, no grid, the tail kept): two conversations over a shared system block, a growing and an edited turn, a third prompt that evicts, and one past the limits. Each pass's resume point and marks, and the tree's states, pages, bytes and counts after it, are pinned before the store changes, and the next two commits leave the run as it was.
A lookahead in the tree. GLM keeps its MTP rows on pages, and row at−1 reads token at, so a page followed by a different token holds different rows. A page is now matched and shared only while the prompt repeats the lookahead's tokens past it: each node keeps those tokens past its last page,
child,common,insertandsplitcompare and carry them, and every slice the store hands the tree grows by the lookahead. A resume leavesmax(1, lookahead)tokens to prefill.A planned grid. A planned store cuts and resumes only on its grid, a multiple of the page: marks snap to it,
deepestskips states off it andkeeprefuses them.Rules it cannot keep.
Store.initreturnserror.BadPromptRadixRulesfor a zero page, or for a planned grid that is zero or no multiple of the page. It was only astd.debug.assert, so a release build took them, as @jschmied noted on #1510.With lookahead 0 and no grid, every path stays as it was.
Receipt
9f805f5a0(three commits on mainf350cd3ea), host tests on Linux x86_64. The table above is GLM's port at0cc5cd348, on cuda qwen3_5: the prompt cache keeps KV pages on the device (TF_QWEN_KV_SHARE, #1156) #1357's65ef4ffccand these commits (GLM-5.3-Flash XSFP4, PARALLEL=4, KV pages on, the drafts on, three GB10 in a ring on NCCL, launched unset, on, on, unset): the comment on #1510 has the method and the TP=2 pair, where main's flat store has room and the two match.test-kv-pool28/28,engine_api's tests 411/411,zig build test441/441 steps (the first run's one failure,cuda_cost_identity's timing-dependent drafter test outside these files, passed when that step ran again),zig fmt --check, andtools/zig/lean_check.pywith nothing new (426 against main's 426). Merged with cuda qwen3_5: the prompt cache keeps KV pages on the device (TF_QWEN_KV_SHARE, #1156) #1357's65ef4ffcc(no conflicts),test-kv-pool28/28 andtest-qwen35-share-faults-host14/14.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. (Nothing on main changes, and the golden run pins the store's resumes. On GLM's port at TP=3 and TP=2, with the drafts on, a resumed prompt equals a fresh one and concurrent equals solo, on GLM CUDA: the KV page pool crowds out the prompt cache, so long chats prefill again every turn #1510. Drafted against plain was not run there.)SUMMARYline is in the receipt. (No GPU kernel is added or changed.)deepesttest does not build, asdeepesttakes no step there, and with that test left out the first lookahead test stops at the store's assert. With their property taken out here, the tree without the lookahead fails the two lookahead tests, a planned store stepping by the page fails the two planned tests and GLM's shape, anddeepestoff the step fails its own. The rules test fails on main's assert. The golden run passes before and after, as it pins what main does. None needs a device.)