[akol1]

What this changes

Adds tf-qwen27-tp-check, a test program that zig build test builds and launches, so the 27B pair has a gate that runs on one Mac. Stage 1 loads the model, then rank 1 through loadShard, and requires bit-identical greedy ids, logits bytes and prompt-state digest. Stage 2 starts two rank-local shards joined by pair.Loopback, a host-memory link, and requires both ranks to emit the same ids; bad_pick_at, seam_gave_up_rank0 and seam_gave_up_rank1 report any time the merge handed a reply to a rank that did not produce it, or the seam gave up on a partial. Stage 3, the default, runs both and --stage 1|2 narrows it. The program needs --model, so CI's zig build test starts it, gets SKIP qwen27-tp-check: no --model DIR, and carries on; a real run happens on a host that has the checkpoint. zig/qwen27_core.zig exports pair so the test can reach the seam.

Refs #1236, the tracker for tensor-parallel ranks: this gate pins the stack's 27B pair against its own single-Mac path, and does not touch the mixed Metal/CUDA case that issue is about. Stacked on #1432, #1433, #1434; the diff against main carries them too, so review this pull request's own commit, dff7857f.

Receipt

• Environment, with the TensorFold commit, MLX or PyTorch version, machine and GPU, checkpoint and revision: 8c36359d plus this commit (dff7857f); no MLX or PyTorch; macOS 27.0.1 on an Apple M4 Pro (Mac16,11), Metal only (applegpu_g16, no tensor units), Apple metal version 32023.921; checkpoint TensorFold/Qwen3.8-27B-MLX-4bit at revision 22d8d538154e0e5b6f8dcb1fcc74a60a0798104c, 4-bit affine with group 64; one Mac, both ranks on it.
• Exactness, from tools/bench_concurrent.py --alone --serial or token_sha against "draft": false: stage 1 is byte-identical — shard1_tokens_equal: true, shard1_logits_byte_equal: true, shard1_first_logits_difference: null, prompt-state digest f46a40128a4d7399b71be2bc1fe1f3ba4aad902844b91124d618bde86c746b68 on the 31-token chat fixture (128-token chunks, 256 steps), and 19232096256 resident bytes for the model against 19235241984 for shard 1. Stage 2 gives ranks_token_equal: true on both fixtures and reference_token_equal: false with first_difference: 236 on the chat fixture and 7 on the 4-id fixture: at the chat tie one Mac's top two are 17077/24.625 and 369/24.5 while the merged pair's are 369/24.625 and 17077/24.625, the bf16 reduction-order tie docs/known-issues.md carries and the same tie the oracle records for chat-256 as 58daf9743077. bad_pick_at is null and both seam give-up counters are 0 on both fixtures.
• Decode speed before and after, from tools/bench_openai.py: n/a — this is a test program, not a serve, and it reports no timings. It does drive both paths: the 4-id fixture at --chunk 1 on the decode path and the chat fixture at --chunk 128 through the prompt kernels.
• Prompt speed before and after: n/a — same reason; the chunked run above is there to exercise the prompt path, not to time it.
• Tests run, with pass, fail and skip counts:zig build test test-golden at this tree: 7116 passed, 45 skipped, 0 failed of 7161, with 535 of 535 build steps; the 45 skips are the image-codec and pixel cases that need libjpeg-turbo 3.1.4.1 or libwebp 1.6.0, plus SKIP qwen27-tp-check: no --model DIR; the program this commit adds is the skip named above, and tf-qwen27-tp-check --stage 1 and --stage 2 were run by hand at this tree with the counts quoted in Exactness (both fixtures, 0 failures, stage 1 byte-identical).
• Not run, and why: the real cross-Mac link is a socket, not pair.Loopback, so that link's receipt is the served path's, not this program's; CUDA, HIP/RDNA, M1-M3, M5 and Windows are absent here; and there is no maintainer-agreed gate for a tensor-parallel family — zig/gate.json has no entries at 8c36359d — so this gate is the stack's own and is offered for the maintainer to replace.

Checklist

• Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. — the check drives Runner and Session directly, the same rounds both ranks run, and adds no serial path and no drafter requirement.
• Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo. — n/a here: the program serves no HTTP and takes no drafter; the pair's refusals are exercised by the served path's tests and receipt.
• No precision traded for speed. If the bits change, the description says which sums change and why. — the check changes no arithmetic; where the pair differs from one Mac it names the tie and its two contenders instead of widening a tolerance.
• A new or changed GPU kernel is registered with the kernel contract (docs/kernel-contract.md); each backend's SUMMARY line is in the receipt. — this commit adds no kernel; it drives the registered qwen27 kernels through the pair, and the pair's own four are registered in the first pull request.
• Prompt processing is no slower than the last release. — n/a: test-only change, no prompt path touched.
• The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not. — ran Metal on an M4 Pro Mac; no CUDA, no HIP, no M1-M3/M5, no Windows (see Not run).
• New tests fail before the change, pass after it, and skip cleanly without their dependency. — the program is new, so red-before is a missing symbol; what it pins is that stage 1 is byte-identical and that stage 2's only difference is the recorded tie. Without the checkpoint the CI step starts it, prints the SKIP line and exits 0.
• Comments and docstrings are one line. No measurements or history in the source. — the lean check reports 428 problems, 428 on the base, 0 new.
• No personal data, machine names, internal hosts or local paths. No AI attribution lines. — the source names no machine, user path, address or serial; the model directory, the prompt file and the forced-token file all arrive as arguments.

[ashhart]

Keep this gate. It's the right one: tp_check.zig returns PairTokensDifferFromSingleMac on any difference. Two problems with the receipt:

• It shows first_difference at 236 and 7, so the gate fails on both fixtures, yet the PR reports "0 failures".
• --stage 2 on its own always fails with StageTwoNeedsTheSingleMacStream, because the reference is set only in stage 1. The receipt says stage 2 ran on its own.

When the seam is exact, both fixtures should pass with no exemption.