Repository navigation
KV pool: host tier for the core pool (5/8) - #1501
BlivionIaG wants to merge 1 commit into
Conversation
c73827f to
d625f19
Compare
|
Thanks @BlivionIaG. This is the tier we hoped for: generic, beside the pool rather than a second pager, with the pool still the only owner of page ids. The random loop, the lockstep digest and the mutants make a good test suite. It doesn't need slices 1 to 4. The last commit cherry-picks cleanly onto main by itself, and there Before it leaves draft:
No family uses the tier yet, so we'd land it together with its first user, off by default: slice 6, as #1419 lists it. Device cells can then check tokens against the device-only pool under eviction, and speed. |
59758dc to
24693fe
Compare
A Tier beside the Pool for families whose pages are cached on the device from a host store. The pool stays the only owner of page ids and of what is promised; the tier adds a second currency (device frames a stream may pin at once), the page-to-frame mapping, an LRU order, pins for the frames a call reads, a dirty flag so a frame is dropped only once its rows reached the host store, and a generation per frame so a table written before a frame was recycled is told from a current one. A stream can also be lent frames for its life, mapped to no page, for a family that manages them itself. The pool tells the tier when a page loses its last holder. It moves no bytes and uses no device API: the family copies what a miss names, and a snapshot reads canonical rows from the host store. Tests: reservation conservation in both currencies, eviction order, pin rules, lent frames, generation safety, canonical rows, a randomised serving loop that audits the mapping and checks every row read against the last write, two ranks in lockstep (and a step or a lent count taken by one rank caught by the digest), and mutants (no generation, no pin rule, dirty frames unpinned) that the checks fail on. A failed batch unmaps the misses it mapped (their generation moves on), a page is marked dirty only under a pin of the marking lease, a stream cannot end while it holds the last pin of a dirty page, and a family can lend a run of consecutive frames (lendRun).
24693fe to
9c628c6
Compare
|
Thanks @BlivionIaG. The latest push looks like it covers both points: a failed fill now One small thing: failedFillReads takes a fault but only runs with none. A mutant As before, we'd land this together with its first user, slice 6, off by default. Could |
K/V pool: optional host-resident tier
Draft PR; validation is in progress. Pushed head:
9c628c6ac155f69fe778da909942ba59ac1b200c. Standalone core source, based on main7c0688951584497dad8b0d5894b657d432fa917f.Adds a generic optional host-resident tier to the core K/V pool: canonical host pages, device-frame residency, generation checks and pin-aware eviction. It is disabled by default and is intended to land with its first consumer, #1522.
This PR is independently based on the listed main commit and changes only three core files. Its archived standalone source passes all 48 K/V-pool tests. Family code, CUDA kernels and EXL3 serving integration remain outside this PR.
This exact pushed prefix passes 48/48 executed host tests, 5/5 build steps (macOS arm64, Zig 0.17.0). It remains a standalone three-file generic-core change; final-family device evidence is additional integration evidence.
Validation and remaining gates
Pushed family tip:
8743eaa3f17717312a25152d2e039c733b1ae590. Tested inference build source:0184770e52e48e0c68479d4f0076dad989e9d42f, whose 9,031 tracked files were verified against its Git tree. The only subsequent changes are the snapshot validation client, its negative tests, and launch-count documentation; all native inference/build-input blobs are identical. Runtime receipts retain the actual build source instead of being relabeled. Native binary SHA256246f5c3e3c171b60d2e6e6bf0e02cc99fd98acb6cbbb851028fc01a513e17b57; complete CUDA runner SHA25687886278dc5f900dc1a7b8c9f3c7717e33d5eaf2c206e81db59b91b89e17b54f.This update enables page-aware decode, verification, deep-prefill and conditional-head graph bindings. Host-tier frame ownership advances before capture/replay, and captured kernels consume the resulting device tables. Page geometry and tier windows participate in graph identity; TP rendezvous permits the legitimate rank-specific page strides. Pooled sequence graphs retire before window storage is released, and all engine graphs retire before engine buffers.
The immediate predecessor
ce9eea00da9500121cb59b07e1bab29e54518790passes all eight short-prompt comparisons: TP1/TP2, paged tier off/on, eager/graph. Two repetitions of 128-token code and prose replies match full token/output hashes, isolated replies and plain replies. Every graph case has actual decode and verification replay on every rank and conditional-head replay on rank 0. These short prompts do not qualify sparse host fetches. The tested inference source adds graph retirement ordering without changing inference arithmetic. All 36 current-build cold replies pass full token/output hashes across TP1/TP2, paged tier off/on, raw 2,048/8,192/32,700 tokens, and first/repeated/concurrent phases, with 64 outputs and zero cached prompt tokens. Both tier runs exercise sparse host fetches. The direct graph/snapshot, long-context replay and additional final functional gates below are complete; the full runner failed in its live resource stage without a final SUMMARY.Completed device checks at the tested inference source
The following results use build source
0184770e52e48e0c68479d4f0076dad989e9d42fand the binary hashes above:std.mem.eql. Two reversed-page lanes are saved at 3,075 and 2,821 tokens, continued by 8,192 prompt tokens and 16 decode steps, and restored under eager execution and three graph cycles. All byte comparisons pass on every participating rank. Each rank's graph cycles record 14/16/16 decode replays; the later cycles each record 144 prefill replays.Completed final functional audit
The independent GPU functional audit passes all ten listed checks, recording PR-build source
0184770e52e48e0c68479d4f0076dad989e9d42fand the separate current-main merge source04383964f9b5d9fdae8491b87f2f1f704e1e8267. The source-specific direct graph and benchmark results above remain attributed to their actual builds. Additional completed gates are:04383964f9b5d9fdae8491b87f2f1f704e1e8267passes three distinct long replies in each of its three geometries, plus both ranks' actual-byte snapshots and observed graph replay.This is a functional-GPU audit, not a waiver or pass for the broader host or global resource gates.
Independent numerical-phase audit and failed resource stage
The full CUDA runner at merge source
04383964f9b5d9fdae8491b87f2f1f704e1e8267, runner SHA256658152c104d30d3aab30827eaa406db81899b6825b9748a1d6f886e5035b25a7, exited 1 during the live resource audit withUnknownContractImage: fp8_experts. It emitted no final runner SUMMARY. The resource audit did not complete, and the whole-suite gate is not passed and not waived.A separate independent audit of the complete raw log (SHA256
4805bcd22d620f84c293d3d4f0043d8d4b39703d90a61712b18476d35a2ed033) covers the completed numerical phase: 441 registeredrunAllkernels, 315 planted controls red, six named KNOWN nonfinite annotations, and zero unexplained numerical failures. All supplemental numerical suites returned before execution enteredlive(). This is an independent numerical-phase audit, not a recovered runner SUMMARY or a whole-suite pass.The six KNOWN annotations remain recorded:
tf_q35_attn_partial,tf_q35_attn_partial8,tf_q35_attn_partial_legacy,tf_q35_attn_partial8_legacyandtf_q35_attn_merge. A NaN key in one chunk can leave the final row finite because the merge skips that chunk.tf_glm_scoresandtf_glm_index_scoreson pages out of order: two annotations where the ReLUfmaxfoperation turns a NaN dot product into zero while the FP64 reference retains NaN.The inherited broader host failure and stale global resource records remain unwaived. This outcome adds no performance result for the required 16,384-input/1,024-output c=1/c=8 workload, which remains pending.
The preceding pushed tip
f94bcff5d1b0f0a506637e6ff70ecf1e17448c0dfixes retained borrowed recurrent spares after actual peer-window settlement. Its positive regression passes; removing its guard gives 246 pass, one fail, no crash. At that source, nine paged TP2 first/repeated/concurrent replies at 2,048/8,192/32,700 raw tokens plus 64 outputs match the TP1 reference with no CUDA error or host allocator report. That predecessor's paged path was eager despite its graph setting; those replies are not graph qualification.02a5440b2e3e7dee1f2e8a9f3cc00f3aab29f79e3b13fedab30206a714d9e5a3, recorded independently of the inference binary. At tested inference source0184770e52e48e0c68479d4f0076dad989e9d42f, the full macOS host/contract/golden/fuzz build passes 596/596 steps, with 986 executed tests passed, four skipped and remaining work cached. The final Linux native/host build passes 276/276 steps, 1,102 executed tests passed and four skipped, using unchanged prebuilt kernels. The complete Flash Next host-test module cross-compiles for Windows. Lean remains at 425 inherited findings. All eight current pushed draft heads have nine successful CI checks each.7c0688951584497dad8b0d5894b657d432fa917findependently reports 1,237 stale-record problems. All shared launch resource measurements match this chain exactly; neither run is a whole-repository gate-5 pass. No five-SM whole-repository pass is claimed.digest cache detects same-size rewrites even when mtime is restoredfails atdigest_cache_test.zig:87on that tip and unchanged main7c0688951584497dad8b0d5894b657d432fa917f; digest/build source blobs are unchanged by this update. Immediate same-size rewrites with restored mtime retain ctime on tmpfs and ordinary storage; delayed diagnostic rewrites change ctime. The failed gate is preserved without a source or test waiver.The following receipts retain their actual preceding source identity
a4dd9f9d02b9c47897f2078817772dee9165b0ff, rather than being attributed to the new head:SUMMARY 2 kernels: 2 pass, 0 fail, 0 known; 1 controls: 1 red, 0 missed. Independent FP64 stress references, CPU trellis decoding, exact-integer and half-midpoint assertions, and FP16/BF16 simulations in sequential, pairwise and 32-lane orders cover six shapes and five-run repeatability per kernel. GU checks complete scratch bytes; down permits only two documented FP32 epilogue roundings. BF16 misses the down bound by at least 5061x; FP16 misses both kernels. These kernel and test sources are unchanged by this update.The required exact 16,384-input/1,024-output performance workload at c=1 and c=8 remains pending, including eight active distinct inputs at c=8 and all four timing/rate metrics. The full CUDA runner on merge source
04383964f9b5d9fdae8491b87f2f1f704e1e8267exited 1 during the live resource audit atUnknownContractImage: fp8_expertsand emitted no final SUMMARY. The independent numerical-phase audit above is not a whole-suite pass. The inherited broader digest-cache host failure and 1,235 stale global resource records remain failed and unwaived. Canonical reduction-order acceptance and the explicitly deferred matched #1113 performance comparison remain outstanding. These PRs remain drafts.The fresh official CUDA build of the separate current-main validation merge
04383964f9b5d9fdae8491b87f2f1f704e1e8267preserves every instruction/control encoding word in all 31 pre-existing untiered attention/batch functions against baseline7c0688951584497dad8b0d5894b657d432fa917fat sm_120/nvcc 13.4. This comparison excludes ELF metadata and the added plane/canonical EXL3 functions. Added plane functions have separate contract coverage. A matching NVFP4 checkpoint is unavailable, so no end-to-end NVFP4 serving receipt is claimed. The checkpoint manifest covers 24 files and 85,139,440,617 bytes, SHA2564e7cbfc3b3d24c2e381b134e9203147dea4f18860026c6b9c17e4c29c54e8ec5.Current observed main is
d33dcfc767ee093393c6bfa00b3949bd7b4d8e88. Since the preceding observationdb46ffd2a3ebc6c7600eb9e2cad5b1b7fa4992db, it adds an unrelated Qwen4 HC activation-contract correction and documentation, with no EXL3 runtime changes. The final pushed tip merges cleanly with this main locally. A separate local-only validation merge,04383964f9b5d9fdae8491b87f2f1f704e1e8267, has all 9,049 tracked files verified and its official build passes 577/577 steps, with 2,549 tests passed and four skipped. Its native binary SHA256 is6416b63defa3906bc2a4292201f3fefe47cfc4fd823aad15728af1e865b5e607; CUDA runner SHA256 is658152c104d30d3aab30827eaa406db81899b6825b9748a1d6f886e5035b25a7. The final functional audit passes the integration three-distinct-long-prompt cases in all three tested geometries and both ranks' byte-level snapshots with actual graph replay. Its full runner exited 1 in the live resource stage without a final SUMMARY; this merge is not a pushed PR head. PR-build runtime receipts retain source0184770e52e48e0c68479d4f0076dad989e9d42f; integration receipts retain source04383964f9b5d9fdae8491b87f2f1f704e1e8267. Neither set is relabeled as the other.Performance work is deferred on purpose
This chain lands the architecture first: a working, bit-exact base on upstream main (TP=2 equals TP=1, drafted equals plain, prompt rows equal decode rows, NVFP4 bytes unchanged). The performance optimizations of the origin draft (#1113) are not all ported here. We ported only the ones that are exact and cheap; the rest follow in separate PRs once this base is merged, each with its own byte-equality receipt, so nothing in this chain has to be rewritten.
Known gaps against #1113 on the same hardware (numbers in the tables below): deep prefill is slower than #1113's non-exact version, drafted decode and the expert-line count at 2 streams are below it, and the two-stream prose case with drafts on is slower than plain. Candidates for the follow-ups: tile tuning, overlapping the partial all-reduce with attention, trimming the deep-prefill chunk buffers, a cheaper draft probe.
Required performance acceptance workload — pending
The required workload is exactly 16,384 input tokens and 1,024 output tokens per request, measured separately at c=1 and c=8. At c=8, all eight requests must be active together and use eight distinct 16,384-token inputs. Submitting eight requests that execute sequentially does not establish this concurrency case.
Both cases must record all four metrics: TTFT seconds, prefill tokens per second, decode tokens per second and mean ITL milliseconds. Qualification requires exact observed input/output lengths, zero cached input tokens, complete replies, full token/output SHA256 evidence and the promised drafted/plain, concurrent/solo and rank-count equality checks. The graph path must have observed graph capture and actual replay evidence; enabling a graph flag or executing eagerly does not by itself establish graph execution.
These required c=1/c=8 measurements are pending. The completed default-prompt 256-output measurements, 128-output sweeps at widths 1/2/4, earlier three-stream long-prompt checks and 32,700-input/64-output memory cases retain their original scopes; none is relabeled as the required 16,384-input/1,024-output qualification. A matched #1113 performance comparison remains unmeasured. The unchanged performance-deferral note above does not turn this target workload into a completed gate.
Prior/default short-workload measurements (256 output tokens)
These completed measurements cover the prior/default short-prompt workload, not the required 16,384-input/1,024-output performance gate. Build source
0184770e52e48e0c68479d4f0076dad989e9d42f, native SHA256246f5c3e3c171b60d2e6e6bf0e02cc99fd98acb6cbbb851028fc01a513e17b57: four fresh single-stream TP1/TP2, plain/drafted configurations, 256 output tokens, three repetitions, greedy temperature 0, context 32,768, graphs and asynchronous instantiation enabled, prompt cache off. All 24 complete replies match full token/output hashes per prompt across rank counts and draft modes, with zero failed/unmeasured replies. Requested expert-cache lines are 72; observed lines differ as shown.Medians, in TTFT seconds / prefill tokens per second / decode tokens per second / mean ITL milliseconds:
TP2 prose drafted decode is 39.557 versus plain 41.258 tokens/s, with mean ITL 25.280 versus 24.238 ms. Drafted cache geometry is 79/88 lines, versus 93/91 plain: these mode differences are measurements, not an isolated draft-overhead experiment. A matched comparison against #1113 remains unmeasured; no within-10% claim is made. The final functional audit separately passes 12 concurrency-sweep cells at widths 1/2/4 with 128-token replies.
Qualified two-rank host-tier memory comparison
Separate merge source
04383964f9b5d9fdae8491b87f2f1f704e1e8267, native SHA2566416b63defa3906bc2a4292201f3fefe47cfc4fd823aad15728af1e865b5e607: TP2, 32,700 raw prompt tokens plus 64 outputs, uniform 16 expert lines per rank, prompt cache off, graphs/async enabled. The three phases submit one request at a time, including the one-request concurrency phase; two pool lanes are available. Tier-on uses 4,096 MiB host budget per rank, a 256-frame cap and slots setting 9. System-reported physical RAM is about 251.55 GiB. Both cases retain exact cold full hashes.Owned requested bytes after all requests and before shutdown:
At this 256-frame setting, tier-on increases tracked device live bytes by 254,254,944 and requested pinned live bytes by 8,257,656,992. These runs do not establish a lower-frame-cap VRAM-saving or capacity configuration.
The counters cover requested
DeviceBuffer/HostBufferbytes only; driver/context/graph/NCCL internals and allocations outside those wrappers are excluded. Retained peaks are since the runtime's last reset, not physical high-water marks; observed sampling cannot exclude unseen resets. Endpoint readings are included in the derived maxima. Stable double reads do not hold the allocator mutex and are not atomic across ranks; HTTP gauges expose the same counters rather than an independent physical measurement. Per-rank peaks are not simultaneous cluster peaks; pinned sums are process accounting rather than unique physical pinned-page counts. RSS includes mapped/shared file pages, so its sum is not a physical RAM footprint. Cleanup zero counters are not established by this receipt. NVML/OS values are one-second samples. PCIe integrals remain device-wide estimates including model load and may miss bursts; OS VmPin/VmLck do not measure CUDA-pinned bytes.Serving measurements recorded at the preceding tip
This preceding-source corpus uses 256-output default prompts and separate 64-output cold cases; it does not measure the required 16,384-input/1,024-output workload at c=1/c=8. These are independent GPU diagnostic measurements on preceding source
a4dd9f9d02b9c47897f2078817772dee9165b0ff, using native SHA256fba7887b03084d6cb60f4019e7850a86163e4b63d188c822d3dcfe6ac9103cf7; they are not relabeled as new-head runs; the inherited broad host gate remains failed. Two RTX 5060 Ti 16 GiB GPUs, driver 615.71.09, CUDA 13.4, Zig 0.17.0; context 32,768, greedy requests, graphs enabled, asynchronous graph instantiation disabled, prompt cache off. All eight primary configurations completed.Decode: 256 output tokens, three repetitions, medians. Each row gives TTFT seconds / prefill tokens per second / decode tokens per second / mean ITL milliseconds. Cache lines are observed per rank, not just the requested setting:
Cold raw-token prompts: 64 output tokens; first request at each length, same four units. Requested 40 lines, expert reserve 512 MiB, server reserve 1.5 GiB; observed lines are 40 at TP1 and 67/76 at TP2. Prefill is prompt tokens / TTFT; mean ITL is stream-arrival span / 63, including zero gaps inside a chunk.
A measured gap remains: TP2 prose drafted decode is 39.59 versus plain 42.25 tokens/s (-6.3%), with mean ITL 25.26 versus 23.67 ms (+6.7%). Their cache geometries differ: 79/88 drafted, 93/91 plain. Current performance deltas against #1113 are unmeasured; no within-10% claim is made.
This PR remains a draft while qualification and maintainer acceptance are outstanding. Fixes #1511.