Repository navigation
Conversation
akol1
force-pushed
the
qwen27/pair-docs
branch
from
October 27, 2026 03:58
e6e6e96 to
cb488f2
Compare
akol1
marked this pull request as ready for review
October 27, 2026 04:32
Owner
|
akol1
force-pushed
the
qwen27/pair-docs
branch
3 times, most recently
from
October 27, 2026 22:38
6011c77 to
0c50037
Compare
Contributor
Author
|
Both text bugs are fixed on
|
Owner
|
Thanks @akol1: the chat-256 divergence is gone from the docs, and the RUNBOOK's rank-1 exit now matches
This also needs #1432's rebase. |
Step 1 of 6 of the two-Mac 27B series; stacked on main (15a0b2a) alone, and steps 2-6 stack on this commit. The seam exchanges whole output rows instead of K halves: rank r owns 2,560 of the 5,120 output columns (8,704 of 17,408 for the down projection), reduces over the full K on its own rank, and `pair_gather` copies whole rows in bf16 without doing any arithmetic, so no exchange rounds or sums a partial product across ranks. A broken link surfaces as bf16 NaN `0x7FC0`. The seam's four kernels (`pair_stage`, `pair_gather`, `pair_argmax`, `pair_pick`) move bf16 frames and compare values only, so they carry no sums and no accumulator's error. One Mac fails a row holding any NaN or ±Inf, so `pair_argmax` reports such a row as NaN bits with bit 31 set in its id word, and `pair_pick` turns that flag into the id sentinel: both Macs then refuse a non-finite round where one Mac would, through the same downstream check as a broken link. Five planted controls (a stage on the peer's half, a gather writing its own half twice, an argmax and a pick tie to the higher id, an argmax that does not flag) each go red on the gate named in `docs/kernel-contract.md`. That transport is a new module, `pair_rows.zig` and `pair_rows.metal`, over `pair_rows_layout.zig`'s slots and the relay protocol unchanged. `pair.metal` and `pair.zig` stay for GLM-5.3-Flash, whose transport this series does not touch: a shipped path cannot change without an A/B against main. `simd_qmm` gains a K-chunk width: `chunks(n, whole)` lets a rank that still holds the whole output width keep the split one Mac derives from that width, instead of one derived from the rank's narrower slice. Gates on this tree: `test-kernel-contract` 20/20 steps, 131/131 tests on a clean cache; the filtered pair contract 4 kernels pass gates 1-4 with 5 controls red as planted, 0 limit problems; `fmt --check` silent.
Step 2 of 6, stacked on step 1 (`qwen27/pair-seam`) and main (15a0b2a); step 3 stacks on this commit. A rank's layer program now carries one `Reduce` interface instead of assuming the whole matrix. Attention's o_proj and the MLP's down_proj seam this rank's output columns, gather the whole frame, and let the producer kernel reduce over full K on its own rank; the head seams the last row's logits and takes the pair's argmax over the whole vocab, so the picked token is the single Mac's token, not rank 0's half-vocab pick. On one Mac `reduce` is null and every seam is compiled out of the program: the single-Mac program, its digests and its tests are the ones that ran before. The rank-local K-split is fixed while the slice is. A projection's split is keyed on the checkpoint tensor's rows, not the view's rows: `Triple` carries `whole` (the matrix's row count) and `prepare` derives `tensorRows` from it, while the M5 and prompt tables keep the checkpoint rows they always keyed. `in_proj_qkv` loads as three blocks of one tensor (10,240 rows), so its gate/up-style stack keeps main's 4 K slices on 6,144 rows instead of reading the q block's 2,048 rows and splitting into 8; every width main served on one Mac keeps main's split, and host tests "a rank's row slice keeps the checkpoint matrix's K split" and "every one-Mac stack keeps main's K split" cover the eight stacks (qkv 10,240 rows: 4, q 6,144: 4, against the 2,048-row view's 8). `Part` drops its unused `k0`, `Triple` its `k`/`k0`, and `Shard`/the family's private `cols`/`attentionCols`/`gdnCols`/`downCols` and the column-window code `prepare`/`packTiles` used them for are gone, with `config_test.zig`'s column-span assertions deleted and the row fields left as they were. `simd_qmm.call` takes the rank's `whole` width and `chunks(n, whole)` picks the chunk fence from it, so a rank's K tree is its own full matrix's while a one-Mac stack passes 0 and keeps exactly main's fence (the core test pins `chunks` on 6,144/12,288 and 5,120/10,240 at 8, and on 6,144 and 5,120 alone at 16, which is the pair's own-width split the `gn` forwarding is there to avoid). `quant_gpu.strided` forwards the `gn` of the linear it serves. `q27_attn_merge` takes the rank's head count (buffer 11) and strides a row by that many heads instead of 24 (`G * 4`), so the non-prompt multi-row rounds a served chat feeds every earlier reply through merge the rank's own rows; `attention.zig` binds `g.heads`. The config gains its shard parse (`config_test.zig` covers it), the embedding table stays whole on every rank — only the head is sliced — with `embedBound` reading the table's bound rather than the family's `vocab_size`, and the session keeps the pair's pick as the round's token and surfaces `error.PairFailed` from the head seam's sentinel rather than returning `maxInt(u32)`. The seam's `Reduce` is threaded through `forward.encodeTp`/`session`, and the prompt attention kernel's signature follows the seam's reopened encoder.
…ith the peer Step 3 of 6, stacked on steps 1-2 (`qwen27/pair-seam`, `qwen27/rank-slice`) and main (15a0b2a); step 4 stacks on this commit. The serving path learns the pair: a rank reads its rank-local shard, the serve loop routes every seam through the link, and the head's pick is the reply. A pair refuses what it cannot keep: vision, a rank count the shard does not match, and a drafter whose state only rank 0 holds, and it fails closed when its watchdog cannot be started. The warm path and the slot accounting understand that both ranks hold the frame. A pair also keeps two serving rules. Its prefill chunk is capped at the seam window's rows (`pair.chunk`), because each chunk stages a seam and the window holds `batch_rows` of them; the host test pins that the served chunk times the widest seam fits `pairBytes()`. And a sampled or grammar-constrained request is refused at the head of `prefillFn` (`error.PairGreedyOnly`) instead of being answered greedily and failing on the second token, with a name in `words()` and a host test. `qwen27_link.zig` is the pair's link over the existing relay; `qwen27_pair.zig` holds the rank-local shard and the seam transport for the serve loop, `qwen27_warm.zig` the warm-up order both ranks share, and `qwen27_served.zig`/`qwen27_slots.zig`/`qwen27_lanes.zig` the accounting a shard adds. GLM-5.3's transport is not touched: `pair.zig`/`pair.metal` are as on main, and no `glm_dsa` file changes in this series.
Step 4 of 6, stacked on steps 1-3 (`qwen27/pair-seam`, `qwen27/rank-slice`, `qwen27/pair-lifecycle`) and main (15a0b2a); step 5 stacks on this commit. `tf-qwen27-tp-check` is built by `zig build test` and launched by the suite. It needs `--model`, so CI's run prints `SKIP qwen27-tp-check: no --model DIR` and carries on, and a host with the checkpoint gets both stages. Stage 1 loads the model, then rank 1 through `loadShard`, and requires the same greedy ids, logits bytes and prompt-state digest; stage 2 starts two rank-local shards joined by `pair.Loopback`, a host-memory link, and requires both ranks to emit the same ids as the single Mac, with `pair_logits_byte_equal` and `pair_first_logits_difference: null` reported, and `bad_pick_at` and the seam give-up counters naming any loss. `--stage 1|2` narrows it and `--dump` prints each side's top two. `--decode-prompt` feeds the prompt through `Session.decodeChunk` in `--chunk N`-row rounds, an earlier reply's arithmetic rather than the prompt kernels: the multi-row round a served chat gives every earlier reply, which runs `q27_attn_merge` and not the prompt kernels' merge. Both stages use the same feed, and the receipt reports `prompt_arithmetic: "decode"` next to the chunk. `--chunk` may then reach 512, the served chunk (`model.promptRows()`); the pair's stages keep 128, the rows the seam window holds (`pair.stageBytes`), so `--chunk 512 --stage 2` is refused by name rather than panicking at the seam. Stage 1 at chunk 128 is where main's merge stride is exercised: it takes `q27_attn_merge` at `chunk` rows a round, so the single-Mac logits bytes are the reference the pair has to match. `zig/qwen27_core.zig` exports the pair and `zig/build/qwen27.zig` registers the program; `--forced-file` replays one side's picks on the other side for divergence localisation, and `--taps` hashes five layers' post-prompt residuals.
Step 5 of 6, stacked on steps 1-4 (`qwen27/pair-seam`, `qwen27/rank-slice`, `qwen27/pair-lifecycle`, `qwen27/pair-tests`) and main (15a0b2a); step 6 stacks on this commit. `tools/qwen27/pair_accept.py` replays the checked-in single-Mac oracle against a served pair: every case is sent twice, both replies must be identical, and their `token_sha` must equal the oracle's, with no case exempt. `tools/qwen27/pair_supervise.sh` starts rank 1 on `$PEER` and rank 0 locally, restarts both when either stops answering, stops only the pids it started (never by pattern), and verifies the peer's host key rather than turning the check off. `qwen38_27b_oracle.json` is the single-Mac, undrafted, temperature-0 record they compare against. The oracle records the GPU family it was taken on (`apple-m4` here), and `pair_accept.py` refuses to replay it against a serve that reports a different family, or one whose family it cannot read, instead of comparing tokens across two GPUs as if they were the same machine. The two-turn case (`chat-two-turn-128`) opens with the recorded `chat-128` reply and replays the oracle's second turn, so the acceptance run covers the multi-row `decodeChunk` rounds a chat takes, not just single-turn prompts; `record_oracle.py` records it the same way. `pair_supervise.sh`'s `PEER_STATE` default is now peer-relative (rank 1's own `${TMPDIR:-/tmp}/tensorfold-pair`, expanded there) rather than rank 0's local directory, so a supervision run does not need rank 0 to know rank 1's filesystem layout; a caller-set `PEER_STATE` is used as given. Nothing in either script names a machine, a user path or an address: every location is environment-overridable, and the shim's stop path is the pidfile.
Step 6 of 6, stacked on steps 1-5 (`qwen27/pair-seam`, `qwen27/rank-slice`, `qwen27/pair-lifecycle`, `qwen27/pair-tests`, `qwen27/pair-tools`) and main (15a0b2a). The recipe and the runbook now carry the two-Mac 27B case: both Macs hold the same checkpoint, each keeps its rank-local half of every row-split projection, `--parallel 1`, requests to rank 0 only, and the watchdog that ends a rank when its peer stops signalling. The known issues page says what the pair does not carry: no drafter, no prompt cache, no learning, one greedy stream, and no `pair_known` or `TF_DUMP_IDS` shortcut. The seam splits by output rows with the full K on each rank, and its gather moves bf16 halves without arithmetic, so there is no fp32 partial sum and no tie policy; the pair's logits are byte-identical to one Mac's on the recorded fixtures and `tf-qwen27-tp-check` fails on any token difference with no exemptions. The recipe says what the pair actually does at a decided token: each rank takes the argmax of its own half of the 248,320 logits and `pair_pick` chooses between the two half-vocab winners, and a non-finite logit on either rank is flagged by `pair_argmax` and fails closed through `pair_pick`'s sentinel as `error.PairFailed` — the pair never takes a whole-vocabulary argmax over a merged buffer. A sampled or grammar-constrained request is refused at the head of prefill by name, not answered greedily for one token; `--vision` is listed with the other refused flags in the recipe and in `docs/known-issues.md`. The formats section says which checkpoints a pair takes: the MLX affine stacks a single Mac reads on Metal, each rank keeping its row slice while the K split stays the checkpoint matrix's, and nothing CUDA-only.
akol1
force-pushed
the
qwen27/pair-docs
branch
from
October 28, 2026 02:56
0c50037 to
f041082
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Step 6 of 6, stacked on steps 1-5 (
qwen27/pair-seam,qwen27/rank-slice,qwen27/pair-lifecycle,qwen27/pair-tests,qwen27/pair-tools) andmain(5897e0bc). This is the last step; nothing stacks on it.The docs now describe what the code does, with nothing projected:
docs/recipes/qwen3.8-27b.mdgainsTwo Macs over Thunderbolt: the same~/models/qwen3.8-27bcheckpoint on both Macs, one speed-up settings file per rank, rank 1 started first and rank 0 within five minutes,--parallel 1, send to rank 0. It states the split exactly (12 of 24 attention heads, 2 of 4 KV heads, half the linear attention's heads and value channels, 8,704 of the 17,408-wide MLP, 124,160 of the lm_head's 248,320 rows), says there is no cross-rank partial sum and no tie policy, lists the refusals, and names the pair-vs-single gate.RUNBOOK.mdgains the two-Mac run: rank 1 first, the harness's/healthwait,pair_accept.py run,pair_supervise.shfor restarts, and what a link failure leaves behind — the request in flight fails, the pair stays down until both ranks run again.docs/known-issues.mdrecords the pair's limits where a user will look for them: one greedy stream, no drafter, no prompt cache, no learning, refusals for what needs one Mac's whole state, and apair_gatherthat joins bf16 halves with no arithmetic between them, so pair logits are byte-identical to one Mac's on the recorded fixtures.No issue closes with this step; it documents the series for #1236 and #808.
Receipt
5897e0bcplus steps 1-6 (0b79f9c9,7d57101a,277929ca,0c2df0ec,0b4f1f4e,0c500376); no MLX or PyTorch; documentation only, so no GPU is involved in producing this diff. The commands it documents are the ones in step 5 and step 4.tools/bench_concurrent.py --alone --serialortoken_shaagainst"draft": false: n/a, this commit changes no code; the receipts it refers to are step 4's four--stage 3runs, all exit 0, with pair logits byte-equal to one Mac's andfirst_differencenull.tools/bench_openai.py: n/a, documentation only; the recipes do not quote a pair speed, because this series has no two-Mac run to measure.zig build test: 565/565 steps, exit 0.zig build test-kernel-contract: 18/18 steps, 120/120 tests.zig build dev-tools: 214/214 steps.zig fmt --check build.zig zig: clean. Lean rule: 428 problems on both5897e0bcand this tree, 0 new (tools/zig/lean_new.py, exit 0). The series head's fresh-cache suite: 565/565 steps, 8,157 passed, 47 skipped, 0 failed of 8,204.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. — the docs state that a pair drafts nothing, caches nothing and serves one stream, so those comparisons are refused rather than documented as passing.SUMMARYline is in the receipt. — none in this commit; the pair'sSUMMARYis in step 1 and the docs point at the gate.applegpu_g16), and they say two Macs were not available.~/models/qwen3.8-27b,127.0.0.1and chart names, not a user path or hostname.