[wojo]

What this changes

On a unified-memory GPU on Linux, the CUDA pool now adds to MemAvailable the free pages that sit on the kernel's
per-CPU lists beyond their resting level, at most MemTotal. Every reading that sizes a start goes through that pool:
the weights check, the room each rank reports, the streams, the KV pages and the kept prompt states. So a start right
after a bulk free no longer budgets those GiBs short. Each rank logs what it counted from 0.25 GiB. Discrete cards and
Windows are unchanged.

Fixes #1311

What is wrong on main (67cc8fb)

• On an integrated device, poolFor takes MemTotal and MemAvailable from /proc/meminfo
  (zig/src/native/cuda_family_memory.zig:22-24, cuda_memory.zig:174-176). openWith reads that pool before the load
  (native/cuda.zig:272), in the room probe that a family over hosts gathers, where rank 0 takes the smallest
  (:285, families/glm/cuda_native.zig:272), and after the load (:337). That last reading sizes the streams, pages
  and kept states (:340-342, :387) and is the start-up line's of X GiB left (:391).
• hostCacheBytes (cuda_memory.zig:84-88) and Drama's readPool (:270) read the same counts.
• Since kernel 6.7 the per-CPU lists tune their own size. After a bulk free they hold GiBs of free pages for minutes.
  The zone's free count leaves those pages out, so neither MemFree nor MemAvailable reports them. Nothing in the
  tree reads /proc/zoneinfo, where they show.
• On two GB10s (GLM-5.3-Flash, TP=2, main 674fb9e, the same budget code), each rank had ~1 GiB parked when rank 0
  printed its budget line on a warm start, and 2.61 / 1.92 GiB on a cold start right after an image and checkpoint
  load (CUDA on unified memory (GB10): the start-up budget does not count free pages parked on the kernel's per-CPU lists #1311).

The change

• zig/src/native/parked_pages.zig (new, 138 lines with its tests): bytes(text, page) sums every zone's count:
  lines under pagesets and subtracts their high_min: lines. Kernels before 6.7 print no high_min:, so there it
  subtracts a fixed 512 MiB resting level. It returns 0 when the text lists no pages or any field fails to parse or
  overflows, so a bad reading never raises a budget.
• cuda_memory.counting(m, zones) adds that to MemAvailable, capped at MemTotal. poolFor, readPool and
  hostCacheBytes read /proc/zoneinfo right after /proc/meminfo. The lists drain into MemAvailable, so a page
  that moves between the two reads is missed, not counted twice. They read it only where they already read
  /proc/meminfo.
• Pool.parked keeps the counted share (default 0). family_memory.noteParked logs it from 0.25 GiB, after the load
  (the CUDA memory budget counts the X GiB free on the per-CPU page lists beyond their resting level (MemAvailable does not)) and in the room probe ("the room this rank reports"), so each rank's process logs its own. The lines
  carry no rank number, like the other start-up lines, since GLM's processes take their tensor rank from
  TF_GLM_RANK rather than the engine option.
• GB10 fixtures, as asked on CUDA on unified memory (GB10): the start-up budget does not count free pages parked on the kernel's per-CPU lists #1311: zig/src/native/fixtures/gb10_parked/ holds /proc/meminfo and
  /proc/zoneinfo from a GB10 (kernel 6.17, 4 KiB pages) during a server start: one pair 6.6 s before the budget
  was read, with the lists still draining (1,027,231,744 bytes parked by this rule), and one pair at the read, with
  the lists at rest (0). Two tests parse them: the draining pair raises MemAvailable and stays under MemTotal, the
  pair at rest counts nothing.
• No switch: it changes only sizes, never replies. No live reading while serving goes through this pool, so nothing
  that guards memory at run time changes.

Tests

cuda_family_memory_test.zig, a Linux unified pool counts the free pages parked on the per-CPU lists, at most MemTotal: the fixture's procText now answers /proc/zoneinfo separately. With 96 GiB available of 128 GiB, a
zoneinfo holding 6 GiB beyond high_min must give a pool of 102 GiB; 40 GiB parked must stop at 128; no zoneinfo, an
empty one or a garbled one must give 96. Windows and a discrete card must not read zoneinfo at all.

On main (this test alone on 67cc8fb, zig build test-cuda-host; trimmed to the failure, paths relative):

error: 'cuda_family_memory_test.test.a Linux unified pool counts the free pages parked on the per-CPU lists, at most MemTotal' failed:
       expected 109521666048, found 103079215104
       zig/src/native/cuda_family_memory_test.zig:98:5: in test.a Linux unified pool counts the free pages parked on the per-CPU lists, at most MemTotal (test)

On this branch it passes. Also new:

• parked_pages.zig: the 6.7+ format (every CPU, zone and node; lists under their resting level count 0); the format
  before 6.7, without high_min: (512 MiB at rest); only lines under pagesets count (stray count: lines, another
  zone's field, a list ending at its zone's next field); no lists, empty, garbled, negative, binary or overflowing text
  count 0. A Linux-only test reads this host's /proc/zoneinfo and checks it parses to less than MemTotal.
• cuda_memory.zig, counting: adds, caps at MemTotal, and leaves the reading alone without zoneinfo or with a bad one.
• cuda_family_memory_test.zig, the pool names the parked share it counted: 6 GiB, 4 GiB when capped, 0 without.

Receipt

• Environment, host tests: base 67cc8fbce, branch commit b104cf77b; Zig 0.17.0; Apple M4 Mac
  (16 GiB), macOS 27.0.1. The Xcode Metal Toolchain component is not installed, so the Metal library builds in
  zig build test fail on both base and branch.

• Environment, served run: base 67cc8fbce against branch b104cf77b, both tensorfold-native built alike (Zig
  0.17.0, aarch64-linux-gnu, -Doptimize=fast) with the same 97 embedded CUDA kernel images (one fatbin set, nvcc
  13.0 V13.0.88, sm_121). Two NVIDIA DGX Spark (GB10, 121 GiB unified memory), driver 580.178.04 (CUDA 13.0),
  cuBLASLt 13.1.1.3, NCCL 2.30.7, glibc 2.39, Ubuntu 24.04.5 LTS, kernel 6.17.0-1032-nvidia, aarch64; TP=2 over RoCE,
  one model process on each.

• Checkpoint and serve: Mia-AiLab/GLM-5.3-Flash-EXL3-4bpw-TensorFold@078455ff, MTP drafts, the same on both trees:
  serve <checkpoint> --backend cuda --parallel 8 --context 131072 --no-update-check, the default prompt cache,
  TENSORFOLD_CUDA_KERNELS=native. (--context 262144 --parallel 8 refuses on main at TP=2, so it cannot show a
  budget.)

• Starts, 8, alternated: cold base, cold branch, warm base, warm branch, cold branch, cold base, warm branch, warm base.
  Cold: started right after the previous server's exit and one read of the checkpoint on both hosts. Warm: the page
  cache dropped on both hosts, then started once both hosts' lists held under 0.25 GiB for 60 s. A 1 s sampler read
  /proc/zoneinfo and /proc/meminfo on both hosts from launch to 20 s after ready; the parked figures are its sample
  nearest rank 0's budget line. Each start served one 32-token bench_openai.py request per prompt.

  start	tree	parked at rank 0's budget line, rank 0 / rank 1 (GiB, sampler)	rank 0 MemAvailable then (GiB)	of X GiB left after a 12.2 GiB reserve	prompt_cache_budget_bytes (GiB)	the new counts the X GiB lines, rank 0 / rank 1
  1 cold	base	0.11 / 0.04	24.50	15.5	3.06	none (main has none)
  2 cold	branch	0.00 / 0.00	24.39	15.9	3.54	none / 0.62
  3 warm	base	0.78 / 0.55	24.49	14.7	2.35	none (main has none)
  4 warm	branch	1.05 / 0.77	23.45	15.5	3.10	1.06 / 0.78
  5 cold	branch	0.00 / 0.00	24.44	16.3	3.89	0.61 / none
  6 cold	base	0.00 / 0.00	24.66	16.1	3.68	none (main has none)
  7 warm	branch	0.75 / 0.65	23.76	15.5	3.14	0.74 / 0.66
  8 warm	base	1.41 / 0.61	24.53	14.1	1.73	none (main has none)

  Every start: 8 streams at once, 1.77 GiB each at a 131072-token window (windows; GLM keeps windows at two ranks),
  rank 0's own room the smallest (no start printed the smallest-room clause), and the same token_sha for both
  prompts (fibonacci-raw e8d0dd8adaba, gpu-chat-no-think 397ece16cce9).

• Warm pairs (the clean comparison): base leaves the parked pages out (3: 0.78 / 0.55 GiB, 8: 1.41 / 0.61 GiB). The
  branch counts them and each printed figure is within 0.01 GiB of the sampler (4: 1.06 / 0.78 against 1.054 / 0.769;
  7: 0.74 / 0.66 against 0.749 / 0.654). Left: 14.7 → 15.5 GiB (3 → 4) and 14.1 → 15.5 GiB (8 → 7); prompt cache 2.35
  → 3.10 and 1.73 → 3.14 GiB. Rank 0 admits by the smallest room, so the expected gain is about the smaller rank's
  count (0.77 and 0.65 GiB); the 8 → 7 gain is larger than that, and the cold starts below show how far left moves
  between starts with nothing counted.

• Cold starts are variable: by the budget read the lists had mostly drained (at most 0.11 GiB in the sampler). Base
  left 15.5 and 16.1 GiB; the branch left 15.9 and 16.3 GiB and counted 0.62 GiB (start 2, rank 1) and 0.61 GiB (start
  5, rank 0). They do not separate the trees.

• Receipt notes: on starts 2 and 5 the engine printed 0.62 / 0.61 GiB where the sampler's nearest sample read 0. The
  engine reads at its own instant and the sampler once a second while the lists drain, so the two can differ. In this run the
  rank 1 process's line read rank 0: (the engine option rank is 0 on both GLM processes), so the report script
  listed it as missing on starts 4 and 7; the line was there. The label is gone in this commit.

• The served receipt above ran on this change's first commit (b104cf77b); the commit here differs only in the log
  lines' rank label and the GB10 fixture tests.

• After each stop: no allocation left on either GPU, and MemAvailable back within 1 GiB of before (116-117 GiB).

• Exactness: n/a for arithmetic. Only sizes change; the served token_sha is equal on every start of both trees.
  "draft": false, resume and concurrency were not run on this change.

• Decode and prompt speed: n/a. Start-up reads one more procfs file per pool reading; the serving paths are unchanged.

• Tests, base 67cc8fbce → branch:

  • zig build test-cuda-host: 253/253 steps, 3253/3267 passed, 14 skipped → 253/253 steps, 3262/3277 passed,
    15 skipped. The added tests are this pull request's: five in parked_pages.zig (the Linux-only one is the added
    skip on macOS), counting's, the block that reaches parked_pages.zig from cuda_memory.zig, and the two pool
    tests; the counts also include the binaries that build more than one of them. Run alone, the branch's suite
    once failed cortheon fuzz: every reader takes linear time on long inputs (a timing test that also failed on
    main in one run here); rerun, it passed.
  • zig build test-golden: 322/335 equal (13 known differences) and 26/26 → the same, 0 unexpected differences.
  • zig build test (fresh cache each, run one after the other): 394/490 steps, 5566/5593 passed, 27 skipped →
    394/490 steps, 5575/5603 passed, 28 skipped. The 55 failed steps are the same on both: Metal library compiles
    (cannot execute tool 'metal' due to missing Metal Toolchain), and no test fails.
  • python3 tools/zig/lean_check.py: 434 problems on base, 434 on the branch; lean_new.py: 0 new. None of them
    is in the changed files.
  • git diff --check: clean.

• Not run: Metal device checks (no Metal toolchain here; the change is CUDA-only), CUDA GPUs other than GB10 (a
  discrete card does not read zoneinfo), one GB10 alone, TP=3 or more, a kernel before 6.7 on a device (the host
  tests cover its format), HIP, and bench_* speed runs. A start that main refuses and the branch admits was not
  tried. One sample per start kind and tree, two of each.

Limits and follow-ups

• The parked figure is a reading. The lists keep draining after it, into MemAvailable, so the pages stay free; what
  the budget counted is not given back. The kernel drains every list when memory runs short, so an allocation the
  budget admitted can always take them.
• Kernels before 6.7 print no high_min:, so there the fixed 512 MiB resting level is an estimate. Below it nothing is
  counted, the same as main.
• DeepSeek-V4.1's own boot check and KV budget read MemAvailable directly (families/deepseek_v41/kv/budget.zig:244,
  cuda_kv.zig:53), and so does the HIP admission (hip/admission.zig:53). Neither is changed here. Each could use
  parked_pages.bytes as its own pull request.
• atc overlap: the only overlaps are open draft pull requests that touch native/cuda.zig. The one textual conflict
  among those checked is with New model family: MiniMax-M3.1-Flash VL (minimax_m3_vl) EXL3, CUDA, TP2-served #1281, which already conflicts with main.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off.
  (Start-up sizing only.)
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  (Not run on this change; it changes how many streams, pages and kept states fit, not how a reply is computed.)
• No precision traded for speed. If the bits change, the description says which sums change and why.
  (No arithmetic changes.)
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's
  SUMMARY line is in the receipt. (No kernel is added or changed.)
• Prompt processing is no slower than the last release. (Not compared; the prompt path is unchanged.)
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. (Host tests on an
  Apple M4 without the Metal toolchain; a served CUDA run on two DGX Spark (GB10) at TP=2.)
• New tests fail before the change, pass after it, and skip cleanly without their dependency. (The pool test fails
  on main and passes here; the Linux-only zoneinfo read skips on macOS.)
• Comments and docstrings are one line. No measurements or history in the source.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines.

By Robert Wojciechowski (wojo on GitHub, @rwojo on X)

[ashhart]

Landed on main (0d24ecb..ab864d4), with you as author. Thank you! On a GB10 with Qwen3.8-27B against main: tokens identical (0 of 20 rows), prefill within 0.3%, and the 1-8 stream sweep within 0.3%. We also captured meminfo/zoneinfo on a GB10 right after a 30B model freed its memory: about 177k pages (690 MiB) sat on the per-CPU lists, still about 46k 30 s later, which is the gap this closes.

- Synthema, Ash's AI Assistant.