Repository navigation
Conversation
|
The Two places change the arithmetic, so the pair can't match the single Mac yet:
Stage 1 compares |
5fcfe81 to
7d57101
Compare
|
Refreshed head
|
|
Thanks @akol1. Carrying the checkpoint matrix's width (
This also needs #1432's rebase. |
0984521 to
b3d9b7e
Compare
|
Refreshed and force-pushed:
Gates on |
|
Thanks @akol1. All four are in on
Keeping |
Step 1 of 6 of the two-Mac 27B series; stacked on main (1b9a987) alone, and steps 2-6 stack on this commit. The seam exchanges whole output rows instead of K halves: rank r owns 2,560 of the 5,120 output columns (8,704 of 17,408 for the down projection), reduces over the full K on its own rank, and `pair_gather` copies whole rows in bf16 without doing any arithmetic, so no exchange rounds or sums a partial product across ranks. A broken link surfaces as bf16 NaN `0x7FC0`. The seam's four kernels (`pair_stage`, `pair_gather`, `pair_argmax`, `pair_pick`) move bf16 frames and compare values only, so they carry no sums and no accumulator's error. One Mac fails a row holding any NaN or ±Inf, so `pair_argmax` reports such a row as NaN bits with bit 31 set in its id word, and `pair_pick` turns that flag into the id sentinel: both Macs then refuse a non-finite round where one Mac would, through the same downstream check as a broken link. Five planted controls (a stage on the peer's half, a gather writing its own half twice, an argmax and a pick tie to the higher id, an argmax that does not flag) each go red on the gate named in `docs/kernel-contract.md`; the next commit adds the sixth, for `pair_pick`'s flag rule itself. That transport is a new module, `pair_rows.zig` and `pair_rows.metal`, over `pair_rows_layout.zig`'s slots and the relay protocol unchanged. `pair.metal` and `pair.zig` stay for GLM-5.3-Flash, whose transport this series does not touch: a shipped path cannot change without an A/B against main. `simd_qmm` gains a K-chunk width: `chunks(n, whole)` lets a rank that still holds the whole output width keep the split one Mac derives from that width, instead of one derived from the rank's narrower slice.
The pick's sentinel rule was the one pair rule without a planted control. The argmax control proves the row's non-finite flag is written, but nothing proved `pair_pick` acts on it: a pick that dropped the `((am | ap) & PAIR_FLAG) != 0u` test would still pass every gate. The sixth control replaces that test with `false`, planted only at the nonfinite fill -- at shapes both ranks' words are finite, so dropping the rule is invisible there and the control has to be red at the gate that plants it. The control is red as planted, and the pair docs now count six.
Step 2 of 6, stacked on step 1 (`qwen27/pair-seam`) and main (1b9a987); step 3 stacks on this commit. A rank's layer program now carries one `Reduce` interface instead of assuming the whole matrix. Attention's o_proj and the MLP's down_proj seam this rank's output columns, gather the whole frame, and let the producer kernel reduce over full K on its own rank; the head seams the last row's logits and takes the pair's argmax over the whole vocab, so the picked token is the single Mac's token, not rank 0's half-vocab pick. On one Mac `reduce` is null and every seam is compiled out of the program: the single-Mac program, its digests and its tests are the ones that ran before. The rank-local K-split is fixed while the slice is. A projection's split is keyed on the checkpoint tensor's rows, not the view's rows: `Triple` carries `whole` (the matrix's row count) and `prepare` derives `tensorRows` from it, while the M5 and prompt tables keep the checkpoint rows they always keyed. `in_proj_qkv` loads as three blocks of one tensor (10,240 rows), so its gate/up-style stack keeps main's 4 K slices on 6,144 rows instead of reading the q block's 2,048 rows and splitting into 8; every width main served on one Mac keeps main's split, and host tests "a rank's row slice keeps the checkpoint matrix's K split" and "every one-Mac stack keeps main's K split" cover the eight stacks (qkv 10,240 rows: 4, q 6,144: 4, against the 2,048-row view's 8). `Part` drops its unused `k0`, `Triple` its `k`/`k0`, and `Shard`/the family's private `cols`/`attentionCols`/`gdnCols`/`downCols` and the column-window code `prepare`/`packTiles` used them for are gone, with `config_test.zig`'s column-span assertions deleted and the row fields left as they were. `simd_qmm.call` takes the rank's `whole` width and `chunks(n, whole)` picks the chunk fence from it, so a rank's K tree is its own full matrix's while a one-Mac stack passes 0 and keeps exactly main's fence (the core test pins `chunks` on 6,144/12,288 and 5,120/10,240 at 8, and on 6,144 and 5,120 alone at 16, which is the pair's own-width split the `gn` forwarding is there to avoid). `quant_gpu.strided` forwards the `gn` of the linear it serves. `q27_attn_merge` takes the rank's head count (buffer 11) and strides a row by that many heads instead of 24 (`G * 4`), so the non-prompt multi-row rounds a served chat feeds every earlier reply through merge the rank's own rows; `attention.zig` binds `g.heads`. The buffered form is only what a pair runs. One Mac has its 24 heads at compile time, and main's kernel used them there (`G * 4`, and `PG * 4` in the prompt merge), so each site now has a second instance: `q27_attn_merge_single` and `q27_attn_prompt_merge_single` carry main's body verbatim — same reduction, same float arithmetic, main's indexing, no `heads` or `kv_heads` buffer — and the encoder takes a trailing `split` and picks `io[4]`/`prompt[1]` for a pair and `io[10]`/`prompt[2]` for one Mac (`decode_round.zig` passes `r.reduce != null`). The two instances compute the same values, so only the emitted code differs. A function constant would need `MTLFunctionConstantValues`, which the runtime library never creates (`PipelineBuild` would fail the contract's first gate), so the second compiled instance is the route that needs no new Metal machinery. The config gains its shard parse (`config_test.zig` covers it), the embedding table stays whole on every rank — only the head is sliced — with `embedBound` reading the table's bound rather than the family's `vocab_size`, and the session keeps the pair's pick as the round's token and surfaces `error.PairFailed` from the head seam's sentinel rather than returning `maxInt(u32)`. The seam's `Reduce` is threaded through `forward.encodeTp`/`session`, and the prompt attention kernel's signature follows the seam's reopened encoder.
b3d9b7e to
a2a94c8
Compare
What this changes
Step 2 of 6 of the two-Mac Qwen3.8-27B series, rebased on
main(db3fd5ab), stacked on step 1 (qwen27/pair-seam). Step 3 stacks on this commit.The 27B family becomes rank-aware behind one interface:
forward.Reduce(zig/src/families/qwen27/forward.zig:23) carries a seam'spart,stage,gatherandfailed, plus an optionalargmaxthat defaults tonull; a one-Mac model therefore compiles the head seam out and executes exactly the layer program it executed before this series.projection.preparederives a rank's row slice from the checkpoint matrix (gn), so the slice keeps the checkpoint's K split, tile and prompt tables (zig/src/families/qwen27/projection.zig:61); prompt rows takepromptSplitof the whole matrix, so the pair's prompt reduction is the reduction tree one Mac uses rather than a half-width tree. The width the seam carries is the tensor's own: the fused gate-up keeps the member split instead of recomputing it from the stacked width, a local qkv block keys neitherm5SplitnorpromptSplit, andsimd_qmm_bits.call(..., whole)keeps the split rank's chunk count throughtwinExactas well.The split is by output rows with the full K on the rank that owns them: 12 of 24 attention heads, 2 of 4 KV heads, 8 of 16 linear-attention q/k heads, 24 of 48 value channels and 3,072 of 6,144 z rows, 8,704 of the MLP's 17,408 rows, 2,560 of the 5,120 output rows for
o_proj/down_projat full K, and 124,160 of the 248,320-row lm_head.o_projanddown_projcompute a rank's own output rows at full K, and the other rank's rows come over the seam as whole bf16 rows:stagecopies this rank's columns of a frame row andgathercopies each rank's half to its own offset, so no two partial sums meet and no row is computed twice. Embeddings and norms stay whole on every rank, so a token id means the same thing on both, and the embedding gather's bound stays the whole 248,320-row table (config.embedBound). The head seam isargmaxover each rank's 124,160-logit half andpickover the pair: no rank computes or gathers the whole 248,320-row logit vector, the pick is the better of the two half-vocab winners, and a non-finite logit on either half fails closed through the sentinel instead of letting the other half's winner stand in for the whole vocabulary's.session.zigkeeps the merged top two for its cross-block tie test and reads the pick'smaxInt(u32)sentinel aserror.PairFailed, never as a logit.Two host-path details the maintainer asked for: the prompt attention merge strides the rank's own head count —
q27_attn_mergetakesconstant uint& heads(zig/kernels/metal/qwen27/attention_mpp.metal:270) and indexes((int64_t)node * (int64_t)heads + h) * D, withattention.zig:413setting it from the rank's geometry; the prompt merge writes the rank's KV node rows (kv_headsinQ27PromptArgs) instead of assuming the model's four (attention_prompt.metal:171). Thein_proj_qkv2,048-row trap is keyed on the tensor's rows, not on a split table, so a whole 6,144-row qkv block stays whole;projection.zig's tests pin it: "every one-Mac stack keeps main's K split", "a rank's row slice keeps the checkpoint matrix's K split", and "stacked gate-up keeps the member K split rather than recomputing it from stacked width".config_testpins the rank spans against the checkpoint's real shapes — 12 heads, 2 KV heads, 8,704 intermediate rows, 124,160 vocab rows, 2,560 output rows, q rows from 6,144 — while rank 0's spans start at 0, andsynthetic_graphruns the same graph on both ranks. The qkv blocks start at 1,024 rather than 2,048, which is the trap the maintainer's comment names: a block boundary keyed on the stacked tensor's rows instead of the tensor's own.Receipt
b3d9b7e74ad63c700b098a4f00b5f18752ec1bdf) onmain(db3fd5abae34e85a3df7b38467ef118022b24908) plus step 1; no MLX or PyTorch; macOS 27.0.1 on an Apple M4 Pro (Mac16,11), Metal only (applegpu_g16, no tensor units), Apple metal version 32023.921. No checkpoint in this commit's gates:config_testandsynthetic_graphbuild the graph from a synthetic config, and the model gates arrive in step 4.tools/bench_concurrent.py --alone --serialortoken_shaagainst"draft": false: n/a at this commit, nothing serves yet; the split is exact by construction and pinned by the host tests above — a rank's slice keeps the checkpoint matrix's K split, the fused gate-up keeps its member split, and every one-Mac stack keepsmain's split (0 new problems in the lean run is the same diff's shape check, not exactness).PASS limits: 313 registered Metal launches against M1-M5, 0 problemsand the filtered contract runs below cover the kernels the split talks to.tools/bench_openai.py: n/a, no serving path at this commit; the pair's decode rate needs two Macs and arrives with step 4's receipts.zig build test:557/557 steps succeeded; 1153/1172 tests passed (19 skipped)(Zig does not re-count cached test steps, so these are the counts of what this checkout re-ran; every step succeeded), exit 0.zig build test-kernel-contract:43/43 steps succeeded, exit 0.zig build dev-tools:215/215 steps succeeded.zig build tf-qwen27-tp-check:no such stepat this commit (step 4 adds it).zig fmt --check build.zig zig: clean. The filtered contract runs:SUMMARY 15 kernels: 15 pass gates 1-4, 0 fail, 0 known findings; 35 controls: 35 red as planted, 0 missed; 0 limit problems; gates 300 ms, limits 146 msandSUMMARY 4 kernels: 4 pass gates 1-4, 0 fail, 0 known findings; 0 controls: 0 red as planted, 0 missed; 0 limit problems; gates 34032 ms, limits 121 ms. Lean rule:lean: 428 problems, 428 on the base, 0 new(tools/zig/lean_new.py, exit 0).tf-qwen27-tp-check, which step 4 adds); a two-Mac run (the serving link is step 3 and the scripts step 5); the unfiltered kernel corpus (it stops atUNSUPPORTED metal moss image: the family runs on tensor units (M5) alone; its kernels unchecked here, so the runs above use--only); CUDA, HIP/RDNA, M1-M3, M5 and Windows are not in this room.Checklist
"draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. — no serving path at this commit; step 3 refuses drafts and step 4's fixtures are greedy, so there is nothing drafted to compare yet.Reduceisnull, so its arithmetic ismain's.SUMMARYline is in the receipt. — the two qwen27 attention kernels this commit edits have no contract record onmaineither; the edit is their head-stride argument, and it is pinned by the one-Mac split tests and by step 4's stage-3 fixtures rather than by a registration this series does not add. The pair's four records are step 1.applegpu_g16); no CUDA, HIP, M1-M3, M5 or Windows.config_test's shard spans andprojection.zig's split tests fail on the parent commit (noShard, nowholeargument), and the model-requiring paths skip without a checkpoint as before.lean: 428 problems, 428 on the base, 0 new.