par: every region at the widest width that pays, on a sub-team of that width

PATCHSET.md

@@ -124,6 +124,7 @@ gate has changed and two kinds of adoption are now in the ledger.
 | Serial path overhead of a team token | A QR 128 on a `Par::Team` that cannot pay was +11 % over `Par::Seq`: the 59 block applies asked every product and triangular step. The block apply asks one floor (`utils::cost::par_can_pay_time`, against `min_cost`, the cheapest width of the table) and runs the inner steps sequentially; `kernel_width` rejects against the same cheapest width instead of assuming a table non-increasing in width; gates read the token's table without a `ParShape` copy; the Rayon host table is cached. threadteam pinned at `7095893`. Fork-only (`Par::Team`). | KEEP (with the team executor) |
 | Threadteam is the only parallel route | `Par::Rayon`, `Par::rayon`, `ParShape::Rayon`, `ParShape::rayon` and `Par::__Lifetime` are removed, with the Rayon arms of `thread::join_raw`/`for_each`/`for_each_with_par`/`split`, of `ParShape::dispatch_cost`/`min_dispatch_cost`, of `par_pays_time`/`par_can_pay_time` and of the GEMM route; `parallelism_degree` folds into `Par::degree` and `is_team` goes. The Rayon cost model in `utils::cost` (`outside_cost_of`, `in_pool_cost_of`, `width_model`, `rayon_cost`, `min_rayon_cost`, `caller_in_pool`, the outside/in-pool anchors) and its tests are deleted; `ParShape::dispatch_cost` reads the costs of the team token the shape came from. faer's own serial `Rates` table and its `HostClass` key stay, and the `PAYS`/`pays_at` rule has one copy in `utils::cost` with a test against `threadteam`'s. `spindle` leaves `Cargo.toml` (no production code opened a group; the last users were tests and the two example harnesses). New: `faer::team::with_team(width, |par| ..)` owns a `threadteam::Pool` for a closure and hands out `Par::Team` (a width of one hands out `Par::Seq`), which is the caller-side answer #59 was waiting for and what the benchmark harness and the threshold probe use instead of `Par::rayon(n)`. `threadteam` becomes the feature that compiles the route (on by default, pulling `rayon` as its substrate); with it off there is no `Par::Team` and no team module, so a no_std build has `Par::Seq` only and nothing pulls std in through Rayon. 81 lib-test and 15 example sites converted to `with_team`; the tests that only asserted the removed route are deleted with the route. | KEEP | tensor4all/faer#85, tensor4all/faer#59 | Mechanism, not a speed-up: the acceptance measurement pairs the branch against `origin/main` with both rows on the team route, and the claim is only that deleting the second route did not regress the team route. Verification: `cargo test -p t4a-faer --lib`, `cargo test -p t4a-faer --doc`, `cargo check` for `--no-default-features`, `--no-default-features --features std`, `--no-default-features --features std,rayon`, default and `std,rayon,threadteam`, and `cargo +nightly-2025-01-08 run --profile nostd` in `faer-no-std-test` (the inherited nostd job). Record: [worklog](docs/worklogs/2026-10-29-threadteam-only.md). |
 | Team dispatch cost read from threadteam | `utils::cost::team_section_cost` and `MEASURED_MAX_WIDTH` (faer's own linear extrapolation above width 8, #97) are deleted: the extrapolation lives in threadteam (`WidthCosts` 0.1.1, linear above a row's largest anchor; 0.1.2, a measured Zen2 team row for `x86_64-avx2` at widths 2 to 64). `ParShape::dispatch_cost` reads `DispatchCosts::cost(Team, w)`; `par_pays_time`, `team::kernel::kernel_width` and the tiled team product's floor ask `TeamExec::dispatch_cost(w)`. `kernel_width` searches from the full width down instead of stopping at the first narrow width that does not pay (measured rows fall with the width). Requirement `threadteam = "0.1.2"`. Decisions change at widths 7 and 8 for rows whose largest anchor is 6, at width 8 on `x86_64-avx512` (its 0.98 us anchor) and at every width on `x86_64-avx2`. | EXPERIMENT (pending the maintainer's acceptance decision) | tensor4all/threadteam-rs#6, tensor4all/faer#91 | No timing (gate inputs only; acceptance run left to the maintainer). Tests: `utils::cost::tests::the_team_cost_is_threadteams_table_at_every_width`, `utils::team::tests::kernel_width_finds_the_widest_paying_width_on_a_falling_row`. Record: [worklog](docs/worklogs/2026-10-29-team-cost-from-threadteam.md). |
+| Every region at the widest width that pays | The full-width gates (`par_pays`, `ParThreshold::region_pays`, `ParShape::pays_full` at the call sites, the block apply's `n_tasks`) chose between the token's whole degree and one thread, so on a wide team a region that pays at 8 or 16 ran serially. Every region now asks `utils::cost::par_width` / `ParThreshold::region_width` for the widest width that pays (every width searched, falling rows included; the GEMM kernel's `kernel_width` is the same rule) and runs on a sub-team of that width (`utils::thread::with_width`, a split into one child: no worker dispatch). Splits whose task count is capped (triangular solve jobs of 64 columns, block apply columns, `upgrade_householder_factor`'s rows per participant) search inside their cap. The factorizations that latched `par = Par::Seq` for the rest of the call (Hessenberg, tridiagonal, unblocked bidiagonal, full pivoting LU, Bunch-Kaufman, column pivoting QR) decide per step; `Never` still makes the whole call sequential; the post-loop compact-factor upgrades of the Hessenberg and tridiagonal reductions now ask their own gate (they ran under the last step's latch, i.e. sequentially). The blocked bidiagonal panel gate is a floor (sequential or the whole `par`), so the panel's products choose their own widths. The Bunch-Kaufman blocked swap loop, split over the whole degree without a gate, now has one (row swaps at the LU swap rate). `Fixed` keeps its rules (the whole degree above the floor). `par_pays`, `region_pays` and the Seq-only GEMM predicate are removed; `par-trace` records the chosen width. | EXPERIMENT (pending the maintainer's acceptance run on broccoli / evo) | tensor4all/faer#91, tensor4all/faer#95 | No timing here (performance evidence is the monitor's: broccoli 1/8/16/32/64T, evo). Tests: `utils::team::tests::{par_width_takes_the_widest_width_that_pays, with_width_runs_on_a_sub_team_of_that_width, the_families_narrow_their_regions_and_match_sequential}`, the rewritten `region_rules_follow_the_serial_time`, `fixed_and_never_keep_their_meaning`, `par_width_reads_the_degree_of_the_token`. Record: [worklog](docs/worklogs/2026-10-29-width-per-region.md). |
 
 ## Ground rules
 

docs/worklogs/2026-10-29-width-per-region.md

@@ -0,0 +1,88 @@
+# Every region at the widest width that pays (tensor4all/faer#91)
+
+## Decisions
+- **One rule, one helper.** The maintainer and monitor decided the rule: a region runs at the widest width in
+  `2..=degree` that pays, `pays_at(estimate, sections, dispatch_cost(w))`, on a sub-team of that width.
+  - `utils::cost::par_width` (units at a rate) and `par_width_time` (a serial time) search the widths from
+    the full degree down and stop at the first width that pays. Every width is considered, so a falling row
+    (`aarch64-apple`) is safe. The cheap rejection against the cheapest width of the table comes first.
+  - `par_width_upto` and `par_width_time_upto` search inside a cap, for splits whose task count is bounded
+    by their shape.
+  - `ParThreshold::region_width` adds the parameter policy: `Never` gives 1, and `Fixed(floor)` gives the
+    whole degree when `units >= floor`.
+  - `kernel::kernel_width` (the x86 GEMM kernel's two sections) is the same function.
+- **Narrowing is a single-child split.** `utils::thread::with_width(par, w, f)` gives:
+  - `f(par)` when `w >= degree`;
+  - `f(Par::Seq)` when `w < 2`;
+  - otherwise `team.split(&[w])`. A one-child split dispatches no worker share: the child is led by the
+    caller, and threadteam's `section` still runs its join guard and its panic handling.
+
+  The helper lives in `utils::thread`, so callers compile without `threadteam`.
+- **Only the region narrows.** Each site wraps only its own region in `with_width`, and the caller keeps its
+  whole `par` for later, larger work.
+- **Per-step factorizations.** Hessenberg, tridiagonal, unblocked bidiagonal, full pivoting LU, Bunch-Kaufman
+  and column pivoting QR set `par = Par::Seq` the first time a step did not pay and kept it for the rest of
+  the call. They now decide per step.
+  - `Never` still makes the whole call sequential, through an explicit `par = Seq` at the top.
+  - Column pivoting QR's non-delayed product at the head of step k uses the width decided at step k-1, which
+    is the decision its comment says covers it.
+  - **Behaviour change:** the post-loop `upgrade_householder_factor` of the Hessenberg and tridiagonal
+    reductions ran under the last step's latch, which was always sequential by then. It now asks its own
+    gate.
+- **Capped splits.** A split whose task count has an upper bound searches inside the cap; it does not clamp
+  the answer afterwards, which a non-monotone row would make wrong.
+  - Triangular solve: jobs of 64 columns. The region's width is searched separately and shared out among the
+    jobs, which keeps the nested parallelism the old code gave each job.
+  - The block apply's column split: `K` tasks, likewise.
+  - `upgrade_householder_factor`: four rows per participant, at both gates. A narrower outer width re-runs
+    the call on a sub-team of that width.
+- **Blocked bidiagonal panel.** The gate stays a floor: sequential, or the whole `par`. Its estimate is the
+  remaining area, and narrowing the whole panel to it would cap the panel's two trailing products, which
+  choose their own widths.
+- **The tiled product (the non-x86 route) takes the same rule.** `gemm_tiled` chose between the whole
+  team and one thread. It now asks `par_width_team`. When the tile grid uses fewer participants than the
+  chosen width, that smaller width must pay as well, because a row need not be monotone.
+- **The `V^H V` reduction splits into exactly the chosen number of row ranges.** It uses balanced
+  `par_split_indices`, where ceiling-sized chunks could produce fewer tasks than the width the gate chose.
+- **`Never` at the public boundaries.** Column pivoting QR and the bidiagonalization apply `Never` before
+  forwarding `par` to their compact-factor upgrades. Otherwise the upgrade's own gate could open a split
+  under `Never`, now that it caps by rows per participant instead of rejecting.
+- **New gate on the Bunch-Kaufman blocked swap loop.** It split over the whole degree with no gate at all,
+  and the family test caught it. It now uses `par_width` at the LU swap rate.
+- **Removed:**
+  - `par_pays`, `region_pays` and `region_pays_impl`;
+  - the GEMM route's full-width predicate at `matmul/mod.rs`. It was reached only after the team arm
+    returns, so it was always 1.
+
+  `ParShape::pays_full` stays as public API.
+- **Trace.** `par-trace` records the chosen width (a new `width` field), and `trace_pub` takes it too.
+- **Counter.** `utils::team::stats` gains `width`: the regions narrowed to a sub-team, by width, up to 256.
+  The `run` histogram saturates at 16.
+
+## Verification conclusions and constraints
+- **Tests, no timing:**
+  - `par_width_takes_the_widest_width_that_pays`: a rising row, a falling row, a cap inside a falling row.
+  - `with_width_runs_on_a_sub_team_of_that_width`: the participant count of a section run on the narrowed
+    token, and the parent's width afterwards.
+  - `the_families_narrow_their_regions_and_match_sequential`, on a team of 8 whose row makes width 4 the
+    widest that pays. Families: full and partial pivoting LU, column pivoting QR, Hessenberg, tridiagonal,
+    bidiagonal, Bunch-Kaufman, triangular solve and matvecs. For each:
+    - regions narrowed to 4, and some of the family's own splits ran at 4;
+    - no split wider than 4;
+    - the outputs equal the sequential ones within `1e-10` relative.
+  - The family test runs every converted region in its own counter window: the triangular solve and both
+    matvecs separately, Bunch-Kaufman with full pivoting (the default `PartialDiag` does not reach the
+    converted steps), the block apply, the compact-factor upgrade, the sparse product and the tiled product.
+  - `never_reaches_the_compact_factor_upgrades`: a 33 x 9 column pivoting QR at block 9 and a
+    bidiagonalization under `Never`, on a cheap row, open no section. It fails without the boundary guards.
+  - The rewritten gate tests: `region_rules_follow_the_serial_time` (partial pivoting LU),
+    `fixed_and_never_keep_their_meaning`, `par_width_reads_the_degree_of_the_token`.
+- **Builds:** default, `par-trace`, and `--no-default-features --features std`. The last has the same
+  warnings as the base.
+- **Not changed (follow-ups):**
+  - The Hessenberg step estimate is `(n-k-1)^2`. The work it gates after the first step is the `A02` matvecs,
+    `k (n-k-1)`, which grows while the estimate shrinks. Re-modelling it needs calibration.
+  - The regions with no gate or a degree-wide split that #94 owns: `rank_update`, the SIMD GEMM loops, the
+    schoolbook product, native `K == 1`.
+  - The section counts passed by some gates are inherited models, not dispatch counts.
+- **Performance evidence is the monitor's:** broccoli at 1/8/16/32/64 threads, and evo.

faer/src/linalg/cholesky/bunch_kaufman/factor.rs

@@ -1,5 +1,6 @@
 use crate::internal_prelude::*;
 use crate::utils::cost::{ParThreshold, rate};
+use crate::utils::thread::with_width;
 use crate::{assert, perm};
 use linalg::matmul::triangular::BlockStructure;
 const TOP_BIT: usize = 1 << (usize::BITS - 1);
@@ -280,6 +281,10 @@ fn lblt_full_piv<T: ComplexField>(
 	let mut A = A;
 	let mut subdiag = subdiag.column_vector_mut();
 	let mut par = par;
+	// `Never` keeps the whole call sequential (the steps decide their own width otherwise)
+	if matches!(params.par_threshold, ParThreshold::Never) {
+		par = Par::Seq;
+	}
 	let n = A.nrows();
 	let scale_fwd = &A.norm_max();
 	let scale_bwd = &scale_fwd.recip();
@@ -331,14 +336,8 @@ fn lblt_full_piv<T: ComplexField>(
 		let rem = n - npiv;
 		// One section per step (the rank-1 or rank-2 update fused with the pivot search);
 		// the unit is an element of the trailing block.
-		if !params.par_threshold.region_pays::<T>(
-			par.shape(),
-			rem * rem,
-			rate::bunch_kaufman(),
-			1,
-		) {
-			par = Par::Seq;
-		}
+		// The step runs at the widest width that pays.
+		let width = params.par_threshold.region_width::<T>(&par, rem * rem, rate::bunch_kaufman(), 1);
 		if i0 != 0 {
 			swap_self_adjoint(A.rb_mut(), 0, i0);
 			perm::swap_rows_idx(Aprev.rb_mut(), 0, i0);
@@ -359,8 +358,9 @@ fn lblt_full_piv<T: ComplexField>(
 				A[(i, i)] = (A[(i, i)].real() - &diag * L[i].abs2()).to_cplx();
 			}
 			if n != 0 {
-				(max_i, max_j, max_offdiag) =
-					rank_1_update_and_argmax(A.rb_mut(), L.rb(), diag, par.reborrow());
+				(max_i, max_j, max_offdiag) = with_width(par.reborrow(), width, |par| {
+					rank_1_update_and_argmax(A.rb_mut(), L.rb(), diag, par)
+				});
 			}
 		} else {
 			let a00 = A[(0, 0)].real();
@@ -380,16 +380,18 @@ fn lblt_full_piv<T: ComplexField>(
 			let (mut L0, mut L1) = L.two_cols_mut(0, 1);
 			let n = A.nrows();
 			if n != 0 {
-				(max_i, max_j, max_offdiag) = rank_2_update_and_argmax(
-					A.rb_mut(),
-					L0.rb(),
-					L1.rb(),
-					d.copy(),
-					d00.copy(),
-					d11.copy(),
-					d10.copy(),
-					par.reborrow(),
-				);
+				(max_i, max_j, max_offdiag) = with_width(par.reborrow(), width, |par| {
+					rank_2_update_and_argmax(
+						A.rb_mut(),
+						L0.rb(),
+						L1.rb(),
+						d.copy(),
+						d00.copy(),
+						d11.copy(),
+						d10.copy(),
+						par,
+					)
+				});
 			}
 			for j in 0..n {
 				let x0 = &L0[j].copy();
@@ -761,35 +763,45 @@ fn lblt_blocked<T: ComplexField>(
 		let pivots = &pivots[k..next];
 		let A_left = A.rb_mut().get_mut(.., ..k);
 		if A_left.ncols() > 0 {
-			match par {
-				Par::Seq => {
-					for mut col in A_left.col_iter_mut() {
-						for (i, &j) in core::iter::zip(k..next, pivots) {
-							let j = j & !TOP_BIT;
-							linalg::lu::partial_pivoting::factor::swap_elems(
-								col.rb_mut(),
-								i,
-								j,
-							);
-						}
-					}
-				},
-	#[cfg(feature = "rayon")]
-				_ => {
-				let nthreads = core::num::NonZeroUsize::new(par.degree()).unwrap();
-					let nthreads = nthreads.get();
-					crate::utils::thread::for_each(par.reborrow(), 
-						nthreads,
-						A_left.par_col_iter_mut(),
-						|mut col| {
+			// one section over the columns left of the block, at the widest width that pays for
+			// its row swaps (it was split over the whole degree, whatever its size)
+			let width = crate::utils::cost::par_width::<T>(
+				&par,
+				A_left.ncols().saturating_mul(next - k),
+				rate::lu_swap(),
+				1,
+			);
+			with_width(par.reborrow(), width, |par| {
+				match par {
+					Par::Seq => {
+						for mut col in A_left.col_iter_mut() {
 							for (i, &j) in core::iter::zip(k..next, pivots) {
 								let j = j & !TOP_BIT;
-								linalg::lu::partial_pivoting::factor::swap_elems(col.rb_mut(), i, j);
+								linalg::lu::partial_pivoting::factor::swap_elems(
+									col.rb_mut(),
+									i,
+									j,
+								);
 							}
-						},
-					);
-				},
-			}
+						}
+					},
+		#[cfg(feature = "rayon")]
+					_ => {
+					let nthreads = core::num::NonZeroUsize::new(par.degree()).unwrap();
+						let nthreads = nthreads.get();
+						crate::utils::thread::for_each(par, 
+							nthreads,
+							A_left.par_col_iter_mut(),
+							|mut col| {
+								for (i, &j) in core::iter::zip(k..next, pivots) {
+									let j = j & !TOP_BIT;
+									linalg::lu::partial_pivoting::factor::swap_elems(col.rb_mut(), i, j);
+								}
+							},
+						);
+					},
+				}
+			});
 		}
 		k = next;
 	}