[OsakaTX]

What this changes

With TF_FILL_CHUNKS and TF_FILL_MANY, the lane host starts one prompt pass a turn and a decode round follows each,
so short prompts that arrive together start a round apart: P1 R P2 R P3 R P4 instead of P1 P2 P3 P4 R. On two GB10s
with 8 lanes, that costs a burst of 8 short requests 36-38% of its aggregate decode rate against the same build with
the fill switches unset.

TF_FILL_DRAIN=ROWS lets the host start passes that go in whole back to back in the same turn, up to ROWS rows. Under
TF_FILL_MANY it lifts that burst's aggregate rate by 56-60% (code: 198 to 318 tok/s; chat: 185 to 288) and brings
its slowest first token from 1.01 s to 0.60 s. That is the unset numbers again, not past them. Passes in pieces keep
TF_FILL_ROUNDS. Unset or 0, nothing changes.

In detail, with TF_FILL_DRAIN=ROWS (and TF_FILL_CHUNKS + TF_FILL_MANY):

• run() keeps calling fillOne() in the same turn, re-admitting between calls, while the next pass goes in whole
  at its first step.
• It stops at the first pass that needs pieces, when the lanes are full, or when the next pass would take the turn's
  rows past ROWS. The first pass always runs.
• A backend that holds one pass at a time admits the next once the last went in, so it drains too.
• Unset, Info.fill_drain is 0 and run() takes its current path.

Files: core/engine_api.zig (Info.fill_drain), families/dsv41/native.zig (reads TF_FILL_DRAIN),
native/cuda.zig (passes it through), core/lane_host.zig (fillOne() / run()), and
core/lane_host_fill_test.zig (two tests). I left TF_FILL_DRAIN out of the native.zig header table, because a
three-line entry there adds lean findings. It is documented on one line where it is read.

Receipt

• Environment: two GB10 nodes, TP2, the RDMA ring. Ubuntu 24.04.5, kernel 7.0.0-1019-nvidia, driver 595.99.02,
  nvcr.io/nvidia/pytorch@sha256:2140e699b3beaf7f96a0081fd9c9406bc3832b435cdb60dfa2d261f7d2f34a1c.

  • Checkpoint: DeepSeek-V4.1-Flash-1017 EXL3 2.9 bpw (LibertAI's pack at afc7aa0a, with a wo_b weight patch in
    blocks 2-39, the same base as feat(lanes): TF_FILL_CUT_IN=T lets a short prompt run between a long prompt pass's pieces (off by default) #43); engram from deepseek-ai/DeepSeek-V4.1-Flash-1017 at 1bae0e10.
  • Engine: deepseek-v41-zig-tp2 at a57deb9d plus this change, plus an env switch for the RDMA GID index (our
    RoCE v2 GID is 3). Its code lines are identical to this PR's; only comment wording differs.
  • Serve: 8 streams, round rows 32, window 1,048,576, --parallel 8, --prompt-cache-gib 0. Engine env in every
    arm: TF_DS_GRAPHS=1 TF_DS_HC_SIDE=1 TF_DS_L2_PREFETCH=1 TF_DS_ENGRAM_AIO=1 TF_DS_REPLAY_FLOOR=1 TF_DS_IDX_SPLIT=1 TENSORFOLD_PLAIN_SPECIALS=1 TF_DS_PROMPT_ALIAS=1 TF_DS_SOLO_OVERHEAD=1 TF_DS_NO_COPIES=1, plus the grammar
    library, SSE keepalive and stats settings.
  • Only the TF_FILL_* switches differed between arms; each start's container env was read back and checked.

• Concurrency: tools/bench_concurrent.py --tokens 256 --reps 3 --temperatures 0 --levels 1,4,8 --alone --serial,
  run in the order DRAIN, MANY, unset, DRAIN, MANY (two passes of each switch, alternated). Medians, tok/s:

  prompt	streams	unset	CHUNKS=1 MANY=1	CHUNKS=1 MANY=1 DRAIN=4096
  code	1	79.4	79.4 / 79.4	79.6 / 79.5
  code	4, aggregate	212.9	168.8 / 168.5	212.8 / 213.0
  code	8, aggregate	318.4	198.9 / 198.1	318.4 / 317.5
  code	8, steady	342.6	214.8 / 212.1	342.5 / 341.7
  code	8, slowest first token (s)	0.60	1.01 / 1.01	0.60 / 0.60
  chat	1	69.5	69.5 / 69.5	69.6 / 69.6
  chat	4, aggregate	191.8	145.7 / 145.5	191.9 / 192.1
  chat	8, aggregate	288.3	185.0 / 184.8	287.9 / 287.6
  chat	8, slowest first token (s)	0.91	1.33 / 1.33	0.91 / 0.92

• Exactness: in all five passes, every concurrent reply's token_sha256 equals its alone run's (78/78 per pass),
  and the alone runs equal "draft": false (2/2 per pass).

• Decode speed (tools/bench_openai.py --tokens 256 --reps 3 --temperatures 0, fibonacci-raw / gpu-chat-no-think):
  unset 79.44 / 69.50, MANY 79.42 / 69.53 and 79.43 / 69.51, DRAIN 79.83 / 69.68 and 79.54 / 69.61.

• What it does not do: with the fill switches unset, short prompts in a burst already start together. DRAIN brings
  TF_FILL_MANY back to that level, not past it. Its use is to run TF_FILL_MANY (for long prompts beside live
  streams) without paying for it on bursts of short ones. If you'd rather not have another switch, it could become
  TF_FILL_MANY's behaviour with a fixed budget; happy to change it.

• New tests (core/lane_host_fill_test.zig, over the fake backend): three 4-token prompts all wait before the host
  starts, and a hook records the core's round count at each one-step pass.

  • Without fill_drain, the three passes start at increasing round counts (a round apart).
  • With fill_drain = 64, all three start before the first round, on the stepped backend and on the one-pass-at-a-time
    backend.
  • With fill_drain = 8, the first two start together and the third waits for a later round.

• Mutation checks (break one line, run zig build test, restore):

  • fillOne() never asking for another pass (return false for return h.info_.fill_drain > 0): the fill_drain
    test fails.
  • The row budget ignored: the fill_drain test fails.
  • job.fill_began dropped from the drain check: not caught, and I don't think a test can catch it. Once a whole
    pass has gone in, its stream is live and fill_since is 0, so the TF_FILL_ROUNDS check on the next line already
    holds back a pass that began. The term changes something only with TF_FILL_ROUNDS=0: there, a begun pass's next
    piece runs before the round instead of after it, and that setting allows either. I kept it to state the intent, so
    the code matches what we measured. Dropping it is fine with me.

• Tests run (x86_64 Linux, Zig 0.17.0, CPU only):

  • zig build test: 1766/1776 passed, 10 skipped (1762/1772 on a57deb9d, with the same 10 skipped). The skips
    are the same five binaries in both trees, each printing libjpeg-turbo 3.1.4.1 not found.
  • python3 tools/zig/lean_check.py + tools/zig/lean_new.py: 1855 problems, 1855 on the base, 0 new.
  • zig fmt --check on the five files is clean.