[shinaoka]

Closes #91. Part of #95.

Problem

Every parallel gate except the x86 GEMM kernel's chose between two outcomes: the token's whole degree, or one thread. On a 64-wide team (threadteam 0.1.2's Zen2 row, linear above 64), a region that pays at 8 or 16 therefore ran serially. The high-thread failure mode was "too serial".

Change

• One rule, one helper.

  • utils::cost::par_width / par_width_time return the widest width that pays, pays_at(estimate, sections, dispatch_cost(w)), or 1. The search runs from the full degree down, so every width is considered and a falling row is safe.
  • par_width_upto / par_width_time_upto search inside a cap.
  • ParThreshold::region_width adds the policy: Never gives 1; Fixed(floor) gives the whole degree at or above the floor.
  • kernel_width (x86 GEMM) is the same function.

• Narrowing. utils::thread::with_width(par, w, f):

  • w >= degree hands over the token itself;
  • w < 2 hands over Par::Seq;
  • otherwise it is a one-child split(&[w]), led by the caller, with no worker dispatch.

  It lives in utils::thread, so builds without threadteam compile. Each site wraps only its own region; the caller keeps its whole par.

• Sites converted:

  • Full gates: partial-pivoting LU (trailing update, swap stage), the matvec row and column splits, spicy_matmul's generic path, the sparse numeric product, gemm_tiled.
  • Capped splits, which search inside their cap: the triangular-solve job split, the block-apply column split, upgrade_householder_factor's two reductions (cap: rows per participant).

• Per-step factorizations. Hessenberg, tridiagonal, unblocked bidiagonal, full-pivot LU, Bunch-Kaufman and col-pivot QR latched par = Par::Seq for the rest of the call; they now decide per step.

  • Never still makes the whole call sequential, including at the public boundaries of col-pivot QR and bidiag.
  • Col-pivot QR's non-delayed product at the head of step k uses step k-1's width.

• Behaviour changes, deliberate:

  • Hessenberg and tridiagonal: the compact-factor upgrades after the loop ask their own gate. They used to inherit the last step's latch, which was always sequential.
  • Blocked bidiagonal panel: the gate stays a floor (sequential or the whole par), so the panel's own products choose their widths.
  • Bunch-Kaufman blocked swap loop: it split over the whole degree with no gate at all; it now has one.
  • Removed: par_pays, region_pays, and the GEMM route's full-width predicate, which was Seq-only after the team arm and so always 1.

• Instrumentation. par-trace records the chosen width. utils::team::stats gains width, counting narrowed regions by width up to 256; run saturates at 16.

Verification (primerose, -j 4, 4 cores, no timing)

• Tests:
  • cargo test -p t4a-faer --lib: 527 passed.
  • --doc: 63 passed.
• Builds: default and par-trace are clean. --no-default-features --features std has the same warnings as origin/main.
• New tests:
  • par_width_takes_the_widest_width_that_pays: a rising row, a falling row, and a cap inside a falling row.
  • with_width_runs_on_a_sub_team_of_that_width: checks the participants of a section.
  • the_families_narrow_their_regions_and_match_sequential: 14 regions on a team of 8 whose row makes 4 the widest paying width. Each region runs in its own counter window and must narrow to 4, run its own split or section on 4, never split wider than 4, and match Seq within 1e-10.
  • never_reaches_the_compact_factor_upgrades: fails without the boundary guards.
• Rewritten tests: the LU region-rule, Fixed/Never and par_width tests.
• LAPACK validation (scripts/run-lapack-validation.py): hessenberg, schur, schur_reorder, balance, tridiagonal_evd, self_adjoint_evd, lu, cholesky, qr, triangular and lstsq pass. The remaining failures are the same on origin/main 31f059c:
  • svd dchkbd_15_to_19 and bidiagonal_family: NoConvergence on DCHKBD type 16 at n = 127 to 129, in all four dtypes. Reproduced on 31f059c, so it predates this change: it started with perf(svd): blocked bidiagonalisation, dqds shift, padded work copy, caller-held worker group; values-only SVD at or below the vendor #52 (svd: DCHKBD type 16 returns NoConvergence at n = 127-129 since the dqds path (#52) #110).
  • nonsymmetric_evd (known residual ratios) and indefinite (unpivoted LDL): the same failing tests as the faer#101 run on ddc55a0.
  • The svd and cholesky families now run to completion. In the faer#101 run on ddc55a0, their parallel tests timed out on four cores.
• Reviews: independent pre- and post-implementation reviews (gpt-6.1-sol). Every finding is applied or answered; see the proxy-log.

Residuals (evo gate, 2026-10-29)

• full_piv_lu f64 128 at 8T, +6.6 to +10.4%. Main latched serial after 42 steps. This PR takes 19 more steps at widths 2-7, each exactly at the rule's floor (estimate 1.8-2.9 us against 3 x D(w)).
  • On evo those steps lose. The cause is per-step costs that the empty-section dispatch row does not contain: allocations, the merge, and column ownership moving between cores.
  • The mechanism's own cost is small: the width search is 20-55 ns and the sub-team split 31 ns, at most about 4 us of the 20-30 us.
  • The three per-step allocations (best, and for_each_child's item and slot vectors) are a defect of the mechanism, tracked as par: no heap allocation inside per-step parallel sections (full_piv_lu best vector, for_each_child item/slot vectors) #111. A fixed-cost term in the cost model will be decided after that.
• cholesky f64 128 at 4T, +9.5 to +10.5%. The gate decisions and dispatches are identical to main, and no region narrows, so the change's mechanism is not involved. Left as a residual (layout or code-size candidate).
• The analysis is in the PR comment.

Not in this PR

• Performance. This PR changes gate decisions only. Evidence comes from the monitor's runs: broccoli at 1/8/16/32/64 threads and evo.
• Follow-ups:
  • The Hessenberg step estimate (n-k-1)^2 does not track the A02 matvecs, k (n-k-1).
  • cost ゲートなしで全幅を渡す経路: 三角積の x86 カーネル、K==1 の rank_update、GEMM のセクション数過少計上 #94's regions: rank_update, the SIMD GEMM loops, the schoolbook product, native K == 1.
  • The section counts some gates pass are inherited models.

🤖 Generated with Claude Code

[shinaoka]

evo regression gate, same procedure as #99. No regression beyond the noise floor except two of the smallest cells (+6 to +10%); one baseline pathology is gone.

Setup. base origin/main 1ac38c6 vs this PR 2bfbc90, both through fbench (which measures the fork row on a team the loop owns), mkl arm, campaign protocol (--prime-ms 250 --reps 5 --statistic median), two process sets per cell combined by the median of their medians, boost off (checked: 0), measured process pinned to 0 / 0-3 / 4-11 for 1T/4T/8T with the observer off that L3 domain. 10 families (qr_factor, qr, lu, full_piv_lu, cholesky, svd_values, svd_thin, eigh, gemm, householder_apply) x 128/256/512 x f64,c64 x both orders = 120 cells per run. OpenBLAS arm alongside at the same cells (the existing lapack-openblas binary).

budget	cells	median cand/base	min	max	outside +-5%
1T	120	0.9997	0.9705	1.0240	0
4T	120	1.0030	0.0919	1.1810	5
8T	120	1.0038	0.9662	1.1041	3

Machine state: the vendor row is the same binary in both runs; its two process sets differ by a median of 1.004 (max 1.069) per cell.

What moved

• qr_factor f64 f 256 at 4T: base {0.72 ms, 15.13 ms} in its two process sets (the bimodal 4T row-major pathology), candidate {0.72, 0.73} — the PR is at the good value in both sets (the 0.09 ratio above is against the base's bad set).
• Consistent, and the only regressions: cholesky f64 128 at 4T both orders (+9.5%, +10.5%; base {0.0695, 0.0662} against cand {0.0745, 0.074} and the same in f-order) and full_piv_lu f64 128 at 8T both orders (+6.6%, +10.4%). Both are the smallest cells of the grid (54 us and 330 us), so this is 4-30 us of per-region cost.
• Not established (the two sets disagree by more than the machine state): qr c64 f 128 at 4T {0.945, 0.724} against base {0.709, 0.704}, and cholesky c64 f 512 at 4T {2.99, 2.70} against base {2.67, 2.67}.
• Everything else is within the spread, including every householder_apply, gemm, svd_values and eigh cell.

OpenBLAS alongside (median of cand/vendor over the cells both arms share): qr 0.78 / 0.44 / 0.37 at 1T/4T/8T, qr_factor 0.74 / 0.54 / 0.47, cholesky 0.71 / 0.65 / 0.62, gemm 0.83 / 0.86 / 0.88, svd_values 0.81 / 0.66 / 0.62, svd_thin 0.90 / 0.66 / 0.62, eigh 1.00 / 0.93 / 0.94, lu 0.78 / 1.25 / 1.44. The PR does not change any of those rows materially (they are the same cells as above).

If the two small-cell regressions matter, the place to look is the sub-team setup for regions below the old width; the rest of the arm is clean.

[shinaoka]

Where the two small-cell regressions in the evo gate come from (04:33Z comment)

I compared the gate decisions and team dispatches of the two cells on 1ac38c6 and on this PR. Method: one call, then counted with par-trace and utils::team::stats. The run uses evo's tables: faer's Rates and threadteam's dispatch row both for x86_64-avx512, injected into a primerose build. Decisions and counts depend only on the tables, not on the host. Nothing was timed except the two micro-costs below. The probe is in the advisor run dir: work/runs/faer-91/gate_probe.rs.

full_piv_lu f64 128, team of 8

	gate decisions	steps at width 8	steps narrowed to 2-7	serial steps
main 1ac38c6	43	42	0	85 (latched after the first refusal)
this PR	127	42	19 (2:3, 3:3, 4:5, 5:5, 6:1, 7:2)	66

The 19 extra parallel steps are what the rule says to take. Each one sits at the floor: the estimate is 1.8-2.9 us against 3 x D(w) of 1.8-2.8 us (D(w) = 0.60-0.94 us on the avx512 row). Main never asked about them because of the latch.

The mechanism's own costs, measured on primerose:

• Width search: par_width_time, a cheap rejection then a descending scan, costs 20-55 ns.
• Sub-team: with_width to a narrower width costs 31 ns. That is the one-child split, with no worker dispatch.

That gives at most 84 × ~40 ns + 19 × 31 ns ≈ 4 us against the measured 20-30 us. So the cost is not in the search or the split. It comes from the 19 floor-level steps losing on evo where the model says they gain about 1.2 us each.

The table's D(w) is the steady-state cost of an empty section. A full-pivoting step section also pays costs that the table does not contain:

• three allocations per step: the best vector, plus the item and slot vectors of for_each_child;
• the reduction merge;
• the trailing block's columns moving between cores when the width changes from step to step.

At the floor, those dominate. The rule and its narrowing work as designed. Fixing this would mean changing the margin or the section count of this one region, which is host-constant tuning, so it is left as a residual. It is recorded in the PR body.

cholesky f64 128, team of 4

Main and this PR are identical:

• the same 2 gate decisions (the triangular join, width 2);
• the same dispatches: run [4:2], 2 kernel GEMMs and 2 below the floor;
• no region narrowed.

The llt path reaches none of the converted per-step or split regions at this size. The +5 us therefore does not come from a decision or a dispatch of this change. A layout or code-size effect is the remaining candidate; I have no mechanism to point at. It is also recorded as a residual.

Possible follow-ups (not in this PR)

• The 16-20 ns cheap rejection copies DispatchCosts, through dispatch_costs().min_cost(). A cached minimum would remove that.
• A per-region fixed-overhead term in the cost model, so that floor-level steps of the per-step factorizations account for their allocations and merge.

Both are outside #91's scope. The broccoli high-thread results are still pending.

[shinaoka]

High-thread check on broccoli: origin/main 1ac38c6 against the PR head 2bfbc90

f64, C order, 1 / 8 / 16 / 32 / 64 threads, the arms interleaved cell by cell, 3 process sets per
cell, harness faer-benchmark 0e685d7, prime 250 ms / 5 reps / median, per-process whole-L3-domain
idle guard (EPYC 7702P, 64 cores Zen2, SMT off, boost 0, governor performance). Both binaries on the
team route (fork-team), both linking threadteam 0.1.2. The OpenBLAS row is from the same binary's
vendor arm, one process per cell.

The #107 stacking mode hit 15 of the 118 fork processes here, on both arms alike (gemm 16/32/64T:
main 7.33 / 6.56 / 6.27 ms against PR 7.34 / 6.69 / 6.40 in the same cells; qr_factor 16T: main
clean, PR 55.27; lu 16T: main clean, PR 57.92). It is a 2-6x jump, so the table below quotes the
minimum of the three processes per cell — the fast level is reproducible, the slow one is the
outlier — and the cells whose medians looked like 15-50% regressions (lu 16T and 64T, qr_factor
16T and 64T, householder_apply 32T) differ only in which process was hit, not in the level.

Fast-mode level (ms), and the vendor column

cell	T	main	PR	main/PR	OpenBLAS
gemm 256x4096x256	1	21.11	21.06	1.002	20.59
	8	3.417	3.410	1.002	4.371
	16	2.168	2.155	1.006	2.407
	32	1.632	1.648	0.990	1.862
	64	1.093	1.082	1.011	1.800
qr_factor 1024	1	72.49	73.23	0.990	139.08
	8	18.42	18.34	1.004	91.64
	16	14.73	14.76	0.998	80.72
	32	13.04	13.23	0.986	80.71
	64	12.95	13.56	0.955	78.32
lu 1024	1	43.11	44.21	0.975	62.90
	8	24.13	26.14	0.923	41.30
	16	24.33	24.53	0.992	40.42
	32	24.59	24.22	1.015	41.76
	64	25.52	38.64	(0.660, unresolvable)	54.01
householder_apply 4096x256	1	49.74	48.70	1.021	skipped
	8	10.71	10.53	1.017	skipped
	16	6.165	6.386	0.965	skipped
	32	4.946	4.738	1.044	skipped
	64	3.942	3.846	1.025	skipped

Reading

• No cell improves, and nothing moves beyond this host's resolution except two small regressions
  in the same direction: lu 8T 24.13 -> 26.14 ms (+8%) and qr_factor 64T 12.95 -> 13.56 (+4.5%,
  and its 64T PR cell had a mode-hit process at 25.64). lu 64T is not resolvable: the PR's two
  processes were 38.64 and 47.28 ms against main's 25.52-42.51, i.e. both arms were in and out of the
  mode there.
• Everything else is 1.00 +/- 0.04, in both directions, with gemm dead flat.
• If the sub-team width selection is meant to help somewhere, these four cells are not where: the
  shapes on this host are far enough above the floors that the region sizes do not change
  (the earlier cost: read the team dispatch cost from threadteam 0.1.2; drop team_section_cost (threadteam-rs#6 stage 4) #104 run made the same point for the cost table itself).
• The vendor column is the context for all of it: the fork is 1.7-6x faster per cell at every budget,
  and OpenBLAS is flat from 16T on (qr_factor 80.7 -> 78.3 ms, lu 40.4 -> 54.0), so nothing in this
  table is a scaling ceiling of the machine.

Before merging I would want the lu 8T pair reproduced on a quiet host (lu is the only cell that
moved in a direction, and 8T is where this host's noise is smallest, so it is the cheap one to check).

[shinaoka]

lu f64 C 1024 at 8T, 5 interleaved process sets on one CCX pair (CPUs 48-55)

arm	the five processes (ms)	median
main 1ac38c6	25.24, 25.34, 25.52, 25.21, 25.49	25.34
PR 2bfbc90	25.61, 26.02, 25.97, 25.87, 63.63 (one #107-mode process)	25.97

The +8 % does not reproduce; a 2.4 % difference does, and it is real. The two arms do not
overlap: main is 25.21-25.52 (1.2 % spread) and the PR is 25.61-26.02, so every one of the five sets
puts the PR above main by 1.4-3.1 %, median 0.976 (min-of-five 0.984). The earlier +8 % came from
comparing minima of three processes whose cells both contained mode-hit members; with five sets on a
single CCX pair the estimate is stable and three times smaller.

Not a blocker from this side — it is a 0.6 ms difference on a 25 ms call — but it is the one cell in
the #109 grid that separates reproducibly, so if the sub-team width selection has a cost anywhere in
these four families, lu at 8T is where to look for it.

[shinaoka]

lu f64 1024: how the width choice differs from main (prepared for the broccoli re-measurement, no timing)

Method: the same as the small-cell analysis. One call of lu_in_place per configuration, counted with par-trace and utils::team::stats, comparing 1ac38c6 against this PR's 2bfbc90.

primerose is the same host class as broccoli, x86_64-avx2, so faer's Rates and threadteam's Zen2 dispatch row are the ones broccoli uses. No table was injected, and the decisions are broccoli's.

8 threads, F order (C order is identical apart from the copy's splits)

	main	PR
trailing updates at width 8	23	23
trailing updates narrowed	0 (serial)	9: 5 at width 2, and one each at 3, 5, 6, 7
trailing updates serial	40	31
swap stages	8 at width 8, the rest serial	identical
triangular-solve job splits	12, widths 2/4/8	identical (12, jobs 2/4/8, region width 8)
GEMMs on the team kernel / below the floor	52 / 398	60 / 406
sections (run)	2:24 8:28	2:29 3:1 5:1 7:1 8:28

The only decision that differs is the trailing update at the levels where width 8 stops paying. Main ran those levels serially; this PR runs nine of them on a narrower sub-team. These are the shapes, and their widths, per call:

• rows 416, 352, 288, 224 at b = 32: widths 7, 5, 3, 2;
• rows 64 at b = 64: width 6;
• rows 1008, 976, 944, 912 at b = 16: width 2.

Their solves and products run on the sub-team too, which accounts for the 8 extra kernel GEMMs and 8 extra below-floor GEMMs.

What this can and cannot explain

All nine sit at or near the rule's floor, the same situation as the full_piv_lu 128 cell. Their total serial work is bounded by the largest of them, about 32 * (32*32/2 + 416*32) ≈ 0.44 M multiply-adds, around 40-50 us serially, and the rest are smaller. Even if every one of them lost its whole serial time, the loss would stay below about 0.2 ms per call.

That is the right size for a 1-2% effect. It is too small for +8% of an 8-thread lu 1024 call, unless that call is well under 3 ms. If the re-measurement reproduces +8%, the width choice above does not account for it, and I would look at what this decision table cannot see: placement or the per-process mode of #107. Otherwise it is these narrowed floor-level levels, and #111's per-section allocations, plus a fixed-cost term if one is still needed after #111, are the place to address them.

64 threads (for the qr_factor 1024 64T note): LU, for reference

• The trailing updates at width 64 are the same as main.
• The narrowed regions now reach widths 9-61 in the larger levels: the narrowed histogram has entries at 22, 24, 29, ..., 61. Main ran those levels at the full 64 or serially.
• Kernel GEMMs: 92 against 73.

This is the intended effect of #91 at high width, so it is not by itself a regression signal. If qr_factor 1024 at 64T stays at +4.5% on re-measurement, I can produce the same table for qr_factor.

[shinaoka]

Merged. The broccoli and evo runs show no mechanism regression: the only cells that separate (lu 1024 8T, 2.4%; full_piv_lu 128 8T) are floor-level steps that now run narrowed, and the analysis above traces their loss to per-section costs the dispatch table does not model (allocation inside per-step sections, the reduction merge). Those are tracked in #111, after which the floor-level steps are re-measured.