GDN planner: price the dense projection work per GDN-owned token; refit the chain overheads and gate

[bradhilton]

Follow-up to #859 (the GDN-planner item of its follow-up list).

What the GDN planner got wrong

After the attention-to-GDN all-to-all the whole GDN layer — input projection, convolution, recurrence, output projection — runs on the GDN layout, so a rank pays the projections for every GDN token it owns, chained or not. The GDN planner's runtime model (GdnPlannerConfig) priced a rank's work by a recurrent rate alone (1,500 tokens/ms), fitted once on the 35B reference shape. On the paired A/B of #859 this showed as a 23% slowdown of one Qwen3.5-4B layout: the planner stopped chaining a 12.6k-token sequence across the ranks because it predicted a 4 ms saving against a 4 ms gate, while the measured saving was about 15 ms per layer.

Measurement

--gdn-ab times every layout of a cell under the production GDN planner and under two forced decisions that bracket its choice — never chain, chain every legal segment — in alternating rounds on the same node with the attention planner unchanged, so the paired difference is the GDN decision's own cost. Campaigns on Qwen3.5-4B (CP2, CP4), Qwen3.5-27B (CP2, CP4) and Qwen3.5-35B-A3B (CP2 EP1/EP2, CP4 EP2/EP4): 982 layouts. Chaining everything is the wrong move on most layouts (short segments; the chain overhead dominates), but where it wins the model missed it by 3–70×: one 12.5k-token sequence at CP4 measures 16.7 ms per GDN layer faster chained on 4B (predicted 3.8) and 48 ms on 27B.

Change

• runtime_dense_tokens_per_ms: the projection work per GDN-owned token, from the model shape's FLOPs at the throughput the three campaigns fit jointly (156 TFLOP/s; 617 tokens/ms on the 4B shape, 224 on 27B, 772 on the 35B reference). It enters the per-rank critical path of the runtime prediction and the chain scoring; the owner search balances recurrent work only (pricing the dense work there moved local segments toward token balance at the cost of more layout exchange, measured as noise with a +14% outlier).
• Chain overheads refit with the recurrent rates and exchange costs held at their shipped values: 2.2 ms per bucket launch (was 0.2), suffix scan 2.8 ms per bucket plus 3.5 segments/ms at the reference shape (was 2.0 and 15); the per-byte summary-exchange cost is not supported by the data and retires into the per-segment scan.
• Chain gate 4.0 → 2.0 ms: every chain that measured slower on 4B was predicted to save under 2 ms per layer; 2 ms keeps 97% of the measured gain on 4B and 99% on 27B.
• Model error on the paired deltas 10.1 → 5.2 ms per layer (rms); the regret of choosing between the two arms by the model 1,082 → 297 ms per layer summed over 982 layouts.

Validation against main's GDN planner (paired, same node, attention planner unchanged)

class	layouts	chain decision changed	faster > 2%	slower > 2%	unchanged layouts	production selection
Qwen3.5-4B	274	72, median −15.2%	69	1 (+3.0%)	174, ±0.5%	−6.9% at CP4, 0.0% at CP2
Qwen3.5-27B	274	95, median −17.8%	90	1	151, ±0.3%	0.0% (production picks were already chained at CP4)
Qwen3.5-35B-A3B	574	119, median −3.7%	70	11 (worst +6.5%)	371 (+30 synthetic CP4 EP1), ±1%	+0.5% to −0.3% per shape

Best layout per real-data group: −15% to −43% at CP4 and −21% to −34% at CP2 on 4B; up to −30% per layout on 27B; within ±1% on 35B. Known residual: the model over-predicts chain savings on 35B by about 2× (11 of 119 changed layouts slower by more than 2%).

Re-certification

table	cells	pairwise	p95 regret	max regret	before
dense-h2560 (Qwen3.5-4B, Qwen3-4B)	58	99.0%	1.2%	1.9%	98.7% / 1.9% / 2.8%
gdn-moe-h2048 (Qwen3.5-35B-A3B)	80	97.8%	2.2%	4.2%	96.6% / 2.7% / 4.2%
dense-gdn-h5120 (Qwen3.5-27B)	not refit				98.6% / 3.9% / 4.6% kept

Every admitted CP > 1 cell of the three GDN tables was re-measured by the validation campaign (its current-planner rows); the fitter now re-certifies from new evidence plus the previous certificate's aggregates for the cells a change does not touch (--from-certificate with evidence). The 27B refit fails its held-out gate on one cell (Ellavox g3 at CP4, 11.3%): the deep uniform_depth_3 layout's long GDN segments now chain and it becomes 10% faster than the shallow layouts the ten terms prefer; neither a CP-split of the GDN level term nor a fit on the CP > 1 cells alone recovers it. The shipped 27B table is kept: under the new planner its picks are within 2.7% of the new best on the other 13 CP4 cells and its g3 pick is 22% faster than before (regret against a new best, not a regression), and withholding the shape would hand the group to version 1, which loses up to 33% there. A second stage pricing shortlisted layouts with the planners' own calibrated models fixes g3 (4.0%) but mis-ranks two synthetic CP4 cells by 7–8% in its pure form; a fitted GDN-aware re-ranker is the follow-up.

Also in this PR

• Harness: --gdn-ab, --gdn-legacy-ab; the contract phase admits the yield_empty flag of Support filtered trajectory copies and safe empty-rank forwarding #864.
• Fitter: re-certification from evidence plus certificate aggregates (manifest fingerprints, completeness and export aware).
• Regression test: an 8k-token sequence at CP4 chains under the dense term and stays on one rank without it.

Follow-ups

• GDN-aware second stage for the layout selector (27B CP4 g3; the 35B chain tail).
• 35B: chain overheads under expert parallelism (the model's 2× over-prediction there).

🤖 Generated with Claude Code