cost: extrapolate WidthCosts linearly above the largest anchor; variable anchors (#6 stage 1); threadteam 0.1.1

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "threadteam"
-version = "0.1.0"
+version = "0.1.1"
 edition = "2021"
 license = "MIT OR Apache-2.0"
 authors = ["Satoshi Terasaki", "Hiroshi Shinaoka"]

README.md

@@ -59,7 +59,7 @@ the token to kernels by value. Kernels never store a token and never mint one.
 | `Pool::into_team(width)` -> `Team` (owns the pool; `IntoTeamError` gives it back), `Team::exec(budget)` / `try_exec`, `Team::pool`, `Team::into_pool` | persistent reservation for an engine that owns its pool (`Pool` is not `Clone`); minted tokens are ordinary `Exec`s whose `fork`, `for_ranges`, `broadcast`, `team` dispatch on the reserved members; one leader at a time (`exec` blocks, `try_exec` -> `Refused::Busy`); idle members park while reserved |
 | `PoolBuilder::qos(Qos)`, `raise_current_thread_qos(Qos)` | worker QoS on macOS (default `UserInitiated`); raise the leading thread without ever lowering it |
 | `PoolBuilder::idle_spin`, `idle_runs_rayon`, `acquire_timeout`; `Pool::release_idle` | waiting policy of idle team members |
-| `DispatchCosts`, `PoolBuilder::dispatch_costs`, `Exec::dispatch_cost(width)`, `TeamExec::dispatch_cost(width)` | cost of one parallel section per kind of token (team, in pool, outside), steady state; one row per `HostClass` (`HostClass::detect()`, keyed by kernel dispatch: `x86_64-avx512`, `x86_64-avx2`, `x86_64-scalar`, `aarch64-apple`, `aarch64`, `generic`), `DispatchCosts::for_class(class)`; measured rows for `x86_64-avx512` and `aarch64-apple`, every other class takes the conservative `generic` row (3 / 15 / 40 us); `DispatchCosts::host_default()` is the detected class's row; `Duration::MAX` when serial; override per pool |
+| `DispatchCosts`, `WidthCosts`, `PoolBuilder::dispatch_costs`, `Exec::dispatch_cost(width)`, `TeamExec::dispatch_cost(width)` | cost of one parallel section per kind of token (team, in pool, outside), steady state; one row per `HostClass` (`HostClass::detect()`, keyed by kernel dispatch: `x86_64-avx512`, `x86_64-avx2`, `x86_64-scalar`, `aarch64-apple`, `aarch64`, `generic`), `DispatchCosts::for_class(class)`; measured rows for `x86_64-avx512` (team anchors at widths 2, 4, 6 and 8) and `aarch64-apple`, every other class takes the conservative `generic` row (3 / 15 / 40 us at the anchors 2, 4, 6); a row holds measured anchors (`WidthCosts::new`, `WidthCosts::flat`, or `WidthCosts::from_anchors` for measured anchors up to 64 and beyond), interpolates between them and grows linearly above the largest one, so a width above the anchors is never charged the cost of the largest anchor; `DispatchCosts::host_default()` is the detected class's row; `Duration::MAX` when serial; override per pool |
 | `pays_at(est, sections, cost)`, `Exec::pays(est, sections, width)`, `Exec::width_for(est, sections)`, `width_for_by`, `PAYS` | the one rule: parallel iff `est >= PAYS * sections * D(width)`; `width_for` minimises `sections * D(k) + est / k` |
 | `Exec::external(f)` | enter the pool once and hand the budget as a plain width to an outside library (faer, a vendor BLAS); how it divides the width is that library's concern |
 

benches/cost_lookup.rs

@@ -1,12 +1,15 @@
-//! Cost of the dispatch-cost lookups themselves: `Exec::dispatch_cost`, `Exec::pays` and
-//! `Exec::width_for` on a team token of a pool of 8 (what a kernel calls per GEMM and per gate).
+//! Cost of the dispatch-cost lookups themselves: `Exec::dispatch_cost`, `Exec::pays`,
+//! `Exec::width_for`, `DispatchCosts::min_cost` and a by-value copy of `DispatchCosts`, on a
+//! team token of a pool of 8 (what a kernel calls per GEMM and per gate). The `width_for`
+//! row takes a budget of 64 through `width_for_by`, so it reads the anchors up to 64 whatever
+//! the pool size.
 //!
 //! `cargo bench --bench cost_lookup`
 
 use std::hint::black_box;
 use std::num::NonZeroUsize;
 use std::time::{Duration, Instant};
-use threadteam::{DispatchCosts, DispatchKind, WidthCosts};
+use threadteam::{DispatchCosts, DispatchKind};
 
 fn ns_per_call(mut f: impl FnMut(u64) -> u64) -> f64 {
     let iters = 2_000_000u64;
@@ -26,10 +29,16 @@ fn ns_per_call(mut f: impl FnMut(u64) -> u64) -> f64 {
 fn main() {
     let costs = black_box(DispatchCosts::host_default());
     let w8 = NonZeroUsize::new(8).unwrap();
-    let budget = black_box(w8);
+    let budget8 = black_box(w8);
+    let budget64 = black_box(NonZeroUsize::new(64).unwrap());
     // Varying input so nothing is hoisted: 5 us .. 1 ms.
     let est = |i: u64| Duration::from_nanos(5_000 + (i % 1024) * 1_000);
     let w = |i: u64| NonZeroUsize::new(2 + (i % 7) as usize).unwrap();
+    let kind = |i: u64| match i % 3 {
+        0 => DispatchKind::Team,
+        1 => DispatchKind::InPool,
+        _ => DispatchKind::Outside,
+    };
     println!(
         "at (width 2..8)       {:>7.2} ns",
         ns_per_call(|i| costs.cost(DispatchKind::Team, w(i)).as_nanos() as u64)
@@ -44,9 +53,65 @@ fn main() {
     println!(
         "width_for (budget 8)  {:>7.2} ns",
         ns_per_call(|i| {
-            threadteam::width_for_by(est(i), 1, budget, |k| costs.cost(DispatchKind::Team, k)).get()
-                as u64
+            threadteam::width_for_by(est(i), 1, budget8, |k| costs.cost(DispatchKind::Team, k))
+                .get() as u64
         })
     );
-    let _ = WidthCosts::flat(Duration::ZERO);
+    println!(
+        "width_for (budget 64) {:>7.2} ns",
+        ns_per_call(|i| {
+            threadteam::width_for_by(est(i), 1, budget64, |k| costs.cost(DispatchKind::Team, k))
+                .get() as u64
+        })
+    );
+    println!(
+        "min_cost              {:>7.2} ns",
+        ns_per_call(|i| costs.min_cost(kind(i)).as_nanos() as u64)
+    );
+    println!(
+        "copy DispatchCosts    {:>7.2} ns",
+        ns_per_call(|i| {
+            // A by-value copy, as `TeamExec::dispatch_costs` and a consumer's
+            // shape hand it on; read a field so the copy cannot be elided.
+            let copy: DispatchCosts = black_box(costs);
+            copy.cost(DispatchKind::Team, w(i)).as_nanos() as u64
+        })
+    );
+
+    #[cfg(feature = "parallel")]
+    token_rows(costs);
+}
+
+/// The lookups on a real team token: the kind is read at call time, so these are
+/// the paths a kernel takes.
+#[cfg(feature = "parallel")]
+fn token_rows(costs: DispatchCosts) {
+    use threadteam::{Pool, WaitOverride};
+
+    let pool = Pool::builder()
+        .threads(8)
+        .dispatch_costs(costs)
+        .acquire_timeout(Duration::from_secs(10))
+        .build()
+        .expect("pool");
+    // Park the reserved members at once: the timing loop must not compete with
+    // idle spinners on a host of fewer cores.
+    let team = pool
+        .into_team_with(8, WaitOverride::new().idle_spin(Duration::ZERO))
+        .expect("idle pool");
+    let exec = team.exec(8);
+    let w = |i: u64| NonZeroUsize::new(2 + (i % 7) as usize).unwrap();
+    let est = |i: u64| Duration::from_nanos(5_000 + (i % 1024) * 1_000);
+    println!(
+        "Exec::dispatch_cost   {:>7.2} ns",
+        ns_per_call(|i| exec.dispatch_cost(w(i)).as_nanos() as u64)
+    );
+    println!(
+        "Exec::width_for (b 8) {:>7.2} ns",
+        ns_per_call(|i| exec.width_for(est(i), 1).get() as u64)
+    );
+    println!(
+        "Exec::pays            {:>7.2} ns",
+        ns_per_call(|i| exec.pays(est(i), 1, w(i)) as u64)
+    );
 }

docs/design/placement.md

@@ -570,15 +570,25 @@ serial_estimate >= PAYS * sections * D(width)        (PAYS = 3)
   `fork` children) is `DispatchKind::Team`; a plain token whose caller is a worker
   of its pool is `InPool`; a plain token called from outside is `Outside`. No
   thread-local state is involved and nothing allocates.
-* `DispatchCosts` holds one `WidthCosts` per kind (values at widths 2, 4 and 6,
-  linear in between, flat above 6).
+* `DispatchCosts` holds one `WidthCosts` per kind. A `WidthCosts` stores the
+  measured anchors of its row (widths 2, 4 and 6 from `new` / `flat`, or any
+  strictly increasing widths from 2 to `u16::MAX` from `from_anchors`, at most
+  `WidthCosts::MAX_ANCHORS` = 12 of them): the value at the first anchor below
+  it, linear interpolation between two anchors, and **linear extrapolation above
+  the largest anchor**, `cost(w_m) * w / w_m`. A wide section therefore never pays
+  the cost of the largest measured anchor; a row measured at width 8 keeps its
+  measured value there instead of the extrapolation. The model is evaluated on
+  every lookup by a scan over at most 12 anchors (no table, no allocation).
   `DispatchCosts::for_class(HostClass)` is the row of a host class, and
   `DispatchCosts::host_default()` the row of `HostClass::detect()`. Classes are keyed by the kernel dispatch the
   machine takes, not by the CPU model: `x86_64-avx512` (runtime `avx512f`), `x86_64-avx2` (`avx2` + `fma`),
   `x86_64-scalar`, `aarch64-apple` (aarch64 on macOS), `aarch64`, `generic`; detected once per process.
-  `x86_64-avx512` holds the measured Ryzen row (team 0.6/0.7/0.9, in pool 3.0/3.6/3.5, outside 9.7/10.1/8.7 us),
-  `aarch64-apple` the measured Apple M5 row, and every class without a measurement the conservative flat
-  `generic` row (3, 15, 40 microseconds). The rates of the consumer crates (strided-rs, tcontract-rs, faer) are
+  `x86_64-avx512` holds the measured Ryzen row (team 0.6/0.7/0.9 at widths 2/4/6 and the measured 0.98 at width 8,
+  in pool 3.0/3.6/3.5, outside 9.7/10.1/8.7 us), `aarch64-apple` the measured Apple M5 row at widths 2, 4 and 6, and
+  every class without a measurement the conservative generic row (3, 15, 40 microseconds at the anchors; the
+  extrapolation charges 32, 160 and 427 microseconds at width 64). A consumer that measured its own host at more
+  widths overrides the row with `WidthCosts::from_anchors` (up to `WidthCosts::MAX_ANCHORS` = 12 anchors). The rates
+  of the consumer crates (strided-rs, tcontract-rs, faer) are
   tables of their own, keyed by the same class names, and a class's rates and dispatch costs come from the same
   machine in the same clock state: boost on/off moves both by about 1.66x and leaves the ratio stable within a
   few percent (`faer-benchmark` `raw-exec3-boost`, `tables-*-exec3-boost.md`). Each crate carries an identical

docs/design/thresholds.md

@@ -282,7 +282,9 @@ checked (0.019 and 0.084). A scalar that is not native takes the `f64` or `c64`
 ```rust
 pub const PAYS: f64 = 3.0;
 pub enum DispatchKind { Team, InPool, Outside }
-pub struct WidthCosts;     // values at widths 2, 4, 6: const new([Duration; 3]), const flat(Duration), at(width), min()
+pub struct WidthCosts;     // measured anchors: const new([Duration; 3]) (widths 2, 4, 6), const flat(Duration),
+                           //   const from_anchors(&[(usize, Duration)]) -> Option<Self>, const MAX_ANCHORS = 12,
+                           //   at(width), min()
 pub struct DispatchCosts;  // new(team, in_pool, outside), host_default(), for_class(HostClass), with(kind, costs), cost(kind, width), min_cost(kind)
 impl PoolBuilder { pub fn dispatch_costs(self, c: DispatchCosts) -> Self; }
 impl Pool        { pub fn dispatch_costs(&self) -> DispatchCosts; }
@@ -302,18 +304,31 @@ estimate / k` over `1..=budget`, counting only widths at which the region pays,
 authority (including `fork` children of a team token), else a worker of the pool (in pool), else outside. No
 thread-local state, no allocation (`tests/dispatch_cost.rs`).
 
-Lookups are cheap: `WidthCosts::new` evaluates the model once into a table of nanoseconds for widths 1 to 6 (wider
-tokens read the last entry, which is the flat model), so `at` is an index, and `pays_at` / `width_for_by` compare
-integer nanoseconds (the objective of `width_for_by` stays `f64`). `cost_lookup` bench, cores 4-11, per call:
-`at` 7.9 to 1.2 ns, `pays_at` after `at` 10.1 to 2.2 ns, `width_for` at budget 8 83 to 32 ns. `min_cost(kind)` /
-`WidthCosts::min()` is the cheapest cost over all widths: a region below `PAYS * min_cost` pays at no width, a
-rejection that stays right for a table whose cost rises with width. A test compares the table with the former
-interpolation for widths 1 to 64.
-
-Width model: linear interpolation between the widths 2, 4 and 6, the width 2 value below, and above width 6
-the width 6 value held flat (the Linux width 8 data, section 3.6, is below the width 6 value, so flat is
-conservative; it replaced growth in proportion to the width). A host that is not measured gets the flat generic table
-(team 3, in pool 15, outside 40 microseconds), which errs towards staying serial.
+Lookups are cheap: a `WidthCosts` stores its measured anchors (integers, widths and nanoseconds) and `at` is a
+forward scan over at most `MAX_ANCHORS` = 12 of them, so the lookup allocates nothing and `pays_at` /
+`width_for_by` compare integer nanoseconds (the objective of `width_for_by` stays `f64`). `cost_lookup` bench on
+cores 4-11 (previous phase, flat table whose `at` was an index): `at` 7.9 to 1.2 ns, `pays_at` after `at` 10.1 to
+2.2 ns, `width_for` at budget 8 83 to 32 ns. The anchor scan costs an integer division per interpolated width where
+the table held the value: on the 4-core run of issue #6 (cores 56-59, best of 9 x 2M calls, same bench file on both
+trees, with the 64-bit fast path that falls back to 128 bits only when a product overflows) `at` 0.89 -> 4.12 ns,
+`pays_at` after `at` 1.97 -> 7.84 ns, `width_for` budget 8 25.9 -> 61.8 ns and budget 64 222 -> 522 ns, `min_cost`
+6.7 -> 4.8 ns, a by-value `DispatchCosts` copy 4.5 -> 11.0 ns (144 -> 384 bytes), `Exec::dispatch_cost` on a team
+3.3 -> 6.3 ns. That stays far below one section, which the rows above charge 0.6 to 427 us, but a consumer that
+calls `width_for` on every tiny region should use `min_cost` first. `min_cost(kind)` /
+`WidthCosts::min()` is the cheapest cost over all widths: the minimum over the anchors (an interpolated value lies
+between its two anchors, and the extrapolation never falls below the last anchor), so a region below `PAYS * min_cost` pays at no width.
+A test compares the rows with the former interpolation for widths 1 to 6 and pins the extrapolation above them.
+
+Width model: the value at the first anchor below it, linear interpolation between two anchors, and above the
+**largest** anchor `w_m` linear extrapolation, `cost(w_m) * w / w_m` (issue #6). The largest anchor is 6 for the
+rows built with `new` (widths 2, 4 and 6); the `x86_64-avx512` team row also carries the measured width-8 anchor,
+and `from_anchors` accepts measured anchors up to `u16::MAX`, so a row measured on a many-core host replaces the
+extrapolation where it has data. Above the last anchor the cost is never flat: a wide section is charged at least
+in proportion to its width, so `width_for` no longer answers the full width just because the largest anchor pays (a
+flat 1 us row gives 1, 2, 4 and 8 us at widths 6, 12, 24 and 48; a 24 us region on such a row selects width 12 of a
+budget 64). Section 3.6 keeps the historical reason for the earlier flat model. A host that is not measured gets the
+flat generic row (team 3, in pool 15, outside 40 microseconds at the anchors 2, 4 and 6, and the width proportion
+above), which errs towards staying serial.
 
 faer: `ParShape::Team(n, DispatchCosts)` carries the costs of its token (copied by `Par::shape`);
 `ParShape::dispatch_cost(width)` and `pays(estimate, sections, width)` delegate to threadteam for team shapes.
@@ -522,8 +537,10 @@ pool of 8, median of 3 processes, microseconds per section:
 After the per-member parker fix (see "Width dependence" below; same bench, pool of 8, 10 us share rows, median of 3
 processes, all accepted; `tables-dispatch-exec2.md`), the team row is 0.6, 0.7, 0.9 us at widths 2, 4, 6 and 0.98 at
 w=8; in pool 3.0, 3.6, 3.5 and outside 9.7, 10.1, 8.7 were re-measured and are essentially unchanged. The table
-above predates the parker fix; `for_class(HostClass::X86_64Avx512)` holds the post-fix row. The flat model above width 6 gives 0.9 us for a
-team at w=8, 8 % optimistic against the measured 0.98; the model is left as it is.
+above predates the parker fix; `for_class(HostClass::X86_64Avx512)` holds the post-fix row. The team row carries all
+four measured widths as anchors, so width 8 keeps the measured 0.98 us; in pool and outside keep the widths of the
+post-fix run only (their width-8 columns are from the pre-fix run and are not used), so the model extrapolates
+there: width 8 is 4.67 us in the pool and 11.6 us from outside.
 
 | kind | w=2 | w=4 | w=6 | w=8 |
 | --- | --- | --- | --- | --- |
@@ -535,7 +552,9 @@ The row is `DispatchCosts::for_class(HostClass::X86_64Avx512)` in `cost.rs`, and
 
 ```rust
 HostClass::X86_64Avx512 => DispatchCosts {
-    team: WidthCosts::new([us_f(0.6), us_f(0.7), us_f(0.9)]),
+    team: WidthCosts::from_anchors(&[
+        (2, us_f(0.6)), (4, us_f(0.7)), (6, us_f(0.9)), (8, us_f(0.98)),
+    ]).unwrap(),
     in_pool: WidthCosts::new([us_f(3.0), us_f(3.6), us_f(3.5)]),
     outside: WidthCosts::new([us_f(9.7), us_f(10.1), us_f(8.7)]),
 },
@@ -545,10 +564,13 @@ What the Linux data says about the design:
 
 * The team cost is the M5's (1.1 to 2.8 us); a plain token costs about half the M5's. Parked members are dearer:
   a team after a 300 ms gap costs 82 to 84 us (M5 27), the first share starts 63 to 83 us after a park.
-* The width model above the last anchor was 2 times pessimistic (it has since been changed to hold the width 6
-  value flat, section 3.1): width 8 measures 1.11 us on a team, 3.0 in the
-  pool and 6.7 outside against 2.21, 4.69 and 11.7 from `cost(6) * 8 / 6`. With the generic table this refuses a
-  team GEMM of `64^3` at width 8 (estimate 9.7 us against `3 * 4.0`) that pays at width 4 (0.32 of serial).
+* The width model above the last anchor was 2 times pessimistic when it grew in proportion to the width; the flat
+  model that replaced it (section 3.1) had the opposite error and was 8 % optimistic at width 8 on the post-fix team
+  row (0.9 against 0.98 us). Since issue #6 the model grows in proportion above the largest anchor again, with the
+  measured width-8 team anchor covering width 8 exactly. The historical numbers: width 8 measures 1.11 us on a team,
+  3.0 in the pool and 6.7 outside (pre-fix run) against 2.21, 4.69 and 11.7 from `cost(6) * 8 / 6`. With the generic
+  row the extrapolation refuses a team GEMM of `64^3` at width 8 (estimate 9.7 us against `3 * 4.0`) that pays at
+  width 4 (0.32 of serial).
 * Every gate measured follows `estimate >= PAYS * sections * D` exactly; the deviations are in the inputs: the
   generic table on Linux, the M5 serial rates (the 2.0 GHz rates here are 1.4 to 3.7 times larger: GEMM 0.064 ns
   per MAC f64, 0.257 c64, 0.032 f32, 0.128 c32), and strided's class assignment (a contiguous `copy_into` and every