[shinaoka]

Closes #111. Part of #95.

Problem

Every parallel region on the team route allocated on the heap:

• utils::team::for_each_child (behind every for_each and for_each_with_par) collected all the iterator's items into a Vec, one per column for par_col_iter_mut, and then a slot Vec.
• The full-pivoting LU step and both Bunch-Kaufman full-pivoting updates built an nthreads-long vector of pivot candidates per step.
• join_sized boxed its two closures.

These allocations were suspected of causing #109's floor-level loss (full_piv_lu f64 128 at 8T on evo, +6.6 to +10.4%). The evo re-measurement of this PR shows they were not: the cell is at 1.0025 of main with them gone, and nothing else regressed. The change stands on its own as a cleanup of the mechanism; the monitor ruled the allocations a mechanism defect.

The remaining candidates for #109's floor-level loss are recorded, not chased here: the fixed per-section costs of the 19 steps that narrow, namely the reduction merge and the trailing block's columns moving between cores as the width changes.

Change

• for_each_child splits the producer on the leader into the blocks (split_at, relative to the remainder, at the par_split_indices boundaries). Each block's leader takes its piece and iterates it there, so P::IntoIter never crosses threads. Everything runs inside the producer callback, so borrowing producers stay valid.
  • Up to 64 blocks, when a producer slot is at most 256 bytes: the pieces live in an inline Option<P> array (at most 16 KiB) and are dealt in one fork.
  • Wider or heavier splits: the producer is halved recursively, two slots and one section per level, and nothing is allocated.
• The pivot candidates come from the call's scratch. lu_in_place_scratch and cholesky_in_place_scratch grow by par.degree() entries of (usize, usize, Real); in BK's they are in an or with the blocked workspace. The buffer is passed down and reused by every step.
• join_sized takes two typed slots instead of Box<dyn FnOnce>.
• Scope of the promise: faer's dispatch and these workspaces. An iterator adapter that allocates in its own split_at, such as rayon's map_with, still does.

Verification (primerose, -j 4, 4 cores, rebased on 0c806f1)

• faer/tests/no_step_alloc.rs is its own binary, with a process-wide counting #[global_allocator], gated on std and threadteam.
  • Teams of 4, 8 and 70; width 70 takes the recursive path. The regions are forced parallel with ParThreshold::Fixed(0), with a warm-up first.
  • Covered: full-pivoting LU 96, Bunch-Kaufman with full pivoting 96 (both pivot sizes asserted), and join_sized with capturing closures. Each has 0 allocations in the counted call and dispatched a parallel section.
  • On origin/main the same test counts 190 allocations in one full-pivoting LU 96 call at width 4.
• New unit tests:
  • for_each_child_deals_every_item_once: flat and recursive paths, a producer over the inline-size limit, counts that do not divide each other, exact item ownership, child widths within the even shares.
  • for_each_child_unwinds_and_keeps_the_team.
• Suites: cargo test --lib 534 passed before the rebases onto 0c806f1 and 7e5e9e3 (the utils, lu and cholesky subsets pass after it); --doc 63 passed.
• Builds: cargo check passes for --no-default-features, --no-default-features --features std, and par-trace.
• Reviews: independent pre- and post-implementation reviews (gpt-6.1-sol). The pre-review's BLOCKER (a spill path above 64 jobs) shaped the design above. The post-review's two IMPORTANT findings (feature gate, focused handoff tests) are fixed.

Performance (monitor, evo, 2026-10-29 07:54Z)

No regression. full_piv_lu f64 128 at 8T is at 1.0025 of main, so #109's +7 to 10% on that cell remains. Whether to chase it, and whether the cost model needs a per-region fixed-cost term, is decided later.

🤖 Generated with Claude Code

[shinaoka]

evo regression check, narrowed as asked (same procedure as #99). No regression at 8T; the cell #109 moved is unchanged against main. One mixed pair at 4T is the only thing outside +-5%.

Setup. base origin/main 7e5e9e3 vs this PR a96737b, both through fbench (mkl arm, campaign protocol: --prime-ms 250 --reps 5 --statistic median), two process sets per cell combined by the median of their medians, boost off (checked: 0), measured process pinned to 0-3 (4T) / 4-11 (8T) with the observer off that L3 domain. full_piv_lu, lu, cholesky, qr_factor, f64, 128/256, both orders = 16 cells per run, OpenBLAS arm alongside.

budget	cells	median cand/base	min	max	outside +-5%
4T	16	0.9998	0.9185	1.0821	2
8T	16	1.0046	0.9980	1.0393	0

Machine state: the vendor row (same binary in both runs) differs by a median of 1.002 (4T) / 1.005 (8T) between its two process sets, max 1.033.

full_piv_lu f64 128 at 8T (the cell #109 moved by +6.6 to +10.4% against the older base): base 0.3418 ms (sets 0.345, 0.339) against candidate 0.3426 ms (sets 0.341, 0.344) — ratio 1.0025, i.e. unchanged. The #109 regression stays where it is and this PR neither adds to it nor removes it; if it is worth chasing, it is #109's cost, not this change's.

The one mixed pair, 4T full_piv_lu f64 128: c-order 0.919 (0.365 -> 0.335 ms, both sets) and f-order 1.082 (0.318 -> 0.346 ms, both sets) — the same 128x128 factorisation, one order faster and the other slower by 8%, and at 8T both orders are at 1.00. Two process sets agree in each direction, so it is not noise, but it is not a mechanism this check can name; worth a second look only if the arm matters at 4T.

Everything else: lu 0.99-1.01, cholesky 0.99-1.04, full_piv_lu 256 0.99-1.01, qr_factor 0.95-1.01.

OpenBLAS alongside (median cand/vendor): cholesky 0.62 / 0.62, lu 0.84 / 0.87, qr_factor 0.49 / 0.46 at 4T / 8T. full_piv_lu has no vendor arm (LAPACK only has partial pivoting), so it is the one family here without a yardstick.

[shinaoka]

The lu f64 C 1024 8T cell: the 2.4 % is not gone (1.7 %, and the ranges now overlap)

Same procedure as the #109 re-check: 5 interleaved process sets, taskset -c 48-55 (one CCX pair),
faer-fork row only, prime 250 ms / 5 reps / median, per-process whole-domain idle guard.
origin/main 7e5e9e3 against the PR head a96737b.

arm	the process medians (ms)	median
origin/main 7e5e9e3	25.26, 24.74, 25.58, 25.54, 25.72	25.54
PR a96737b	25.49, 26.25, 26.30, 25.98, 25.94	25.98

ratio 0.983 (0.978 over the four clean sets — the fifth PR process was guard-rejected at 7.7 % busy
by two other agents' processes on CPUs 50/53; its own value, 25.94, is in line, so the conclusion does
not depend on it).

Reading

• The difference is still there and is now inside the overlap: main spans 24.74-25.72, the PR
  spans 25.49-26.30, so 1.7 % nominal with both ranges touching.
• Most of the apparent improvement since par: every region at the widest width that pays, on a sub-team of that width #109 is origin/main drifting, not this PR. Against
  1ac38c6 the pair was main 25.34 / PR 25.97 (+2.4 %); against 7e5e9e3 it is 25.54 / 25.98
  (+1.7 %). The PR's own level did not move (25.97 -> 25.98); main moved up 0.8 %.
• So on this cell par: no heap allocation inside a parallel region #126 is neutral, and the 2.4 % should be carried as an open question about lu's
  C-order row-major adaptation rather than attributed to par: every region at the widest width that pays, on a sub-team of that width #109 or closed by par: no heap allocation inside a parallel region #126. The per-step regions
  this PR fixes (for_each_child's collection, the full-pivoting pivot candidates) are not what
  dominates lu 1024 at 8T; the PR body's own evidence is full_piv_lu 128 at 8T, a different cell.

If you want this settled rather than carried: a 4T/8T pair of lu 1024 C with 10 interleaved sets on
a quiet host, and a full_piv_lu 128 8T pair (the PR's own regression cell) would show whether #126's
win exists where it was claimed and whether this cell is a real 1-2 % cost.