Own worker threads: a process-wide reserve of std::thread workers instead of a private Rayon pool (#15)

Cargo.toml

@@ -13,9 +13,13 @@ categories = ["concurrency", "science"]
 
 [features]
 default = ["parallel"]
-# Run entries on a private Rayon pool. Without it the same types exist, every
-# pool has width 1 and every entry runs inline on the calling thread.
-parallel = ["dep:rayon"]
+# Run entries on the pool's own worker threads (docs/design/own-threads.md).
+# Without it the same types exist, every pool has width 1 and every entry runs
+# inline on the calling thread.
+parallel = []
+# Opt-in Rayon interop: `Exec::external_rayon` runs a callback inside a private
+# Rayon pool of the token's width. Not needed by anything else.
+rayon = ["parallel", "dep:rayon"]
 # Test support: per-root live-task ledger and entry counters. Zero cost when off.
 ledger = []
 # Test support: hooks that pause threads inside the park/wake handshake and
@@ -45,6 +49,8 @@ libc = "0.2"
 
 [dev-dependencies]
 libc = "0.2"
+# The Rayon reference rows of benches/wait_scenarios.rs.
+rayon = "1.10"
 
 [[bench]]
 name = "team_dispatch"
@@ -83,3 +89,8 @@ unexpected_cfgs = { level = "warn", check-cfg = ['cfg(loom)'] }
 [[bench]]
 name = "cost_lookup"
 harness = false
+
+[[bench]]
+name = "wait_scenarios"
+harness = false
+required-features = ["parallel"]

README.md

@@ -5,7 +5,16 @@ that starts tasks: kernels take an `Exec` token and split it, and no scheduler
 type appears in any public signature.
 
 Used by strided-rs, tcontract-rs, cpueinsum-rs, tlinalg-rs and tenferro-rs.
-Design record: [`docs/design/placement.md`](docs/design/placement.md).
+Design records: [`docs/design/placement.md`](docs/design/placement.md) (the token),
+[`docs/design/own-threads.md`](docs/design/own-threads.md) (the worker threads),
+[`docs/design/waiting.md`](docs/design/waiting.md) (how they wait).
+
+The workers are threadteam's own `std::thread`s. A pool leases them from a
+process-wide reserve and gives them back when dropped, so a pool built after another
+one was dropped reuses running threads (faer's `with_team` builds one per call);
+a pool with a thread name, a start hook, a stack size or a QoS of its own gets
+dedicated workers that its drop joins. Rayon is not used unless the opt-in `rayon`
+feature asks for `Exec::external_rayon`.
 
 ## Usage
 
@@ -58,8 +67,9 @@ the token to kernels by value. Kernels never store a token and never mint one.
 | `Exec::team(f)` / `team_with(over, f)` | reserve `budget - 1` awake pool members for a scope and run barrier-bearing sections through `TeamExec` (`run`, `run_with`, `split`, `fork`, `reborrow`), or `Err(Refused::Unavailable)`; see the design record |
 | `Pool::into_team(width)` -> `Team` (owns the pool; `IntoTeamError` gives it back), `Team::exec(budget)` / `try_exec`, `Team::pool`, `Team::into_pool` | persistent reservation for an engine that owns its pool (`Pool` is not `Clone`); minted tokens are ordinary `Exec`s whose `fork`, `for_ranges`, `broadcast`, `team` dispatch on the reserved members; one leader at a time (`exec` blocks, `try_exec` -> `Refused::Busy`); idle members park while reserved |
 | `PoolBuilder::qos(Qos)`, `raise_current_thread_qos(Qos)` | worker QoS on macOS (default `UserInitiated`); raise the leading thread without ever lowering it |
-| `PoolBuilder::idle_spin`, `idle_runs_rayon`, `acquire_timeout`; `Pool::release_idle` | waiting policy of idle team members; a reserved member spins with one PAUSE per poll (never `sched_yield`), parks after the delay, and parks after a short spin in a root scope that has more participants than the caller's CPUs |
-| `PoolBuilder::pin_participants(bool)`, `Pool::pin_participants()` | opt-in placement of the **members** of a root team scope (Linux; off by default, which is consistent with the rule it protects): member `k` (participant `k >= 1`) pins itself onto the `k`-th CPU of the calling thread's affinity set, `S[0]` is left to the leader, and a member restores its own mask before it returns to Rayon or serves foreign work. **The leader is never pinned**, because a vendor library creates its thread team from the calling thread and a one-CPU caller would confine the whole provider team; a member that calls such a library directly inside a share does confine it, so use `Exec::external` (which restores the member's mask first) or leave pinning off. For tensor4all/faer#107; give each team its own CPU set (`taskset -c`), there is no CPU allocator |
+| `PoolBuilder::idle_spin`, `idle_runs_rayon`, `acquire_timeout`; `Pool::release_idle` | waiting policy of idle team members (an idle member spins for the spin budget unless `idle_spin` is zero, then sleeps; `idle_runs_rayon` has no effect and is kept for compatibility) |
+| `PoolBuilder::spin_polls`, `Pool::spin_polls` | spin budget of every team wait, in polls (the shape of libgomp's `GOMP_SPINCOUNT`): a reserved member waiting for work, the leader in a join and a participant in a barrier spin that many polls, yield a few times, then sleep until woken; throttled to 100 polls in an oversubscribed scope or after a waiter was descheduled ([`docs/design/waiting.md`](docs/design/waiting.md)) |
+| `PoolBuilder::pin_participants(bool)`, `Pool::pin_participants()` | opt-in placement of the **members** of a root team scope (Linux; off by default, which is consistent with the rule it protects): member `k` (participant `k >= 1`) pins itself onto the `k`-th CPU of the calling thread's affinity set, `S[0]` is left to the leader, and a member restores its own mask before it serves the pool's queue, leaves the pool, or serves foreign work. **The leader is never pinned**, because a vendor library creates its thread team from the calling thread and a one-CPU caller would confine the whole provider team; a member that calls such a library directly inside a share does confine it, so use `Exec::external` (which restores the member's mask first) or leave pinning off. For tensor4all/faer#107; give each team its own CPU set (`taskset -c`), there is no CPU allocator |
 | `DispatchCosts`, `WidthCosts`, `PoolBuilder::dispatch_costs`, `Exec::dispatch_cost(width)`, `TeamExec::dispatch_cost(width)` | cost of one parallel section per kind of token (team, in pool, outside), steady state; one row per `HostClass` (`HostClass::detect()`, keyed by kernel dispatch: `x86_64-avx512`, `x86_64-avx2`, `x86_64-scalar`, `aarch64-apple`, `aarch64`, `generic`), `DispatchCosts::for_class(class)`; measured rows for `x86_64-avx512` (team anchors at widths 2, 4, 6 and 8) and `aarch64-apple`, a measured Zen2 team row for `x86_64-avx2` (anchors at ten widths from 2 to 64; its plain kinds stay generic), every other class takes the conservative `generic` row (3 / 15 / 40 us at the anchors 2, 4, 6); a row holds measured anchors (`WidthCosts::new`, `WidthCosts::flat`, or `WidthCosts::from_anchors` for measured anchors up to 64 and beyond), interpolates between them and grows linearly above the largest one, so a width above the anchors is never charged the cost of the largest anchor; `DispatchCosts::host_default()` is the detected class's row; `Duration::MAX` when serial; override per pool |
 | `pays_at(est, sections, cost)`, `Exec::pays(est, sections, width)`, `Exec::width_for(est, sections)`, `width_for_by`, `PAYS` | the one rule: parallel iff `est >= PAYS * sections * D(width)`; `width_for` minimises `sections * D(k) + est / k` |
 | `Exec::external(f)` | enter the pool once and hand the budget as a plain width to an outside library (faer, a vendor BLAS); how it divides the width is that library's concern |
@@ -72,11 +82,13 @@ the token to kernels by value. Kernels never store a token and never mint one.
   once is at most its budget at every nesting depth. A parent waiting for its
   children keeps no slot.
 * **Lazy and inline.** A serial token, budget one, or one task runs inline on the
-  calling thread and makes no scheduler call, so the process-global Rayon pool is
-  never started implicitly.
-* **Placement by caller.** A worker of the target pool spawns into the current
-  scope without installing; a plain thread or a worker of another pool enters the
-  target pool once and waits. No thread-local state exists in the crate.
+  calling thread and makes no scheduler call; no worker starts before a pool is
+  built, and Rayon's process-global pool is never started.
+* **Placement by caller.** A worker of the target pool runs its own batch's
+  tasks itself and serves the pool while it waits; a worker of another pool
+  publishes on the target pool and serves its own pool while it waits (so
+  chains between pools always make progress); a plain thread publishes and
+  sleeps until its tasks finished. No thread-local state exists in the crate.
 * **`broadcast`** refuses with `Refused::ForkChild` for a token derived from
   `fork` (through `reborrow` or `limit` too), and with `Refused::CallerInPool`
   when the caller is a worker of the same pool. Roots, and their `reborrow`s and
@@ -86,11 +98,14 @@ the token to kernels by value. Kernels never store a token and never mint one.
   blocking work of another token on those workers delays the broadcast.
   `can_broadcast` applies the same rule through one private helper, so the two
   cannot disagree.
-* **Deterministic shutdown.** Dropping a `Pool` joins every worker thread: when
-  `drop` returns, all workers have exited and their thread-local destructors
-  (including state created in an `on_start` hook) have run. A pool dropped from
-  one of its own workers cannot join itself, so that drop only signals shutdown
-  and returns without joining. Without `parallel` there is nothing to join.
+* **Shutdown.** Dropping a `Pool` detaches every worker and waits until each
+  one acknowledged; after that no worker touches the pool. Dedicated workers (a
+  pool with a thread name, an `on_start` hook, a stack size or a QoS of its own)
+  are then joined: their thread-local destructors, including state created in
+  an `on_start` hook, have run when `drop` returns. Shared workers go back to the
+  process-wide reserve, asleep, for the next pool. A pool dropped from one of its
+  own workers does not wait for or join that worker, which acknowledges and
+  leaves after its task returns. Without `parallel` there is nothing to join.
 * **No scheduler type is public.** There is no constructor from, and no accessor
   returning, a scheduler pool. `scripts/check-no-public-rayon.sh` runs
   `cargo public-api -p threadteam` (when installed) and fails if `rayon::`
@@ -100,12 +115,58 @@ the token to kernels by value. Kernels never store a token and never mint one.
 
 | Feature | Effect |
 | --- | --- |
-| `parallel` (default) | run on a private pool. Without it the same types exist, `PoolBuilder::build` returns a width-1 pool and every entry runs inline |
+| `parallel` (default) | run on the pool's own worker threads. Without it the same types exist, `PoolBuilder::build` returns a width-1 pool and every entry runs inline |
+| `rayon` | opt-in Rayon interop: `Exec::external_rayon(f)` runs `f(width)` inside a private Rayon pool of the pool's width (separate threads, built on first use) |
 | `ledger` | test support: `ledger::max_live(root)`, `live`, `installs`, and `Exec::ledger_root`. Records live tasks per root token and pool installs; compiled out when off |
 
 Consumers assert `ledger::max_live(root) <= budget` in their own tests, and keep
-a test binary that runs serial and pooled work and then checks that
-`rayon::ThreadPoolBuilder::new().build_global()` still succeeds.
+a test binary that runs serial and pooled work and then checks
+`ledger::global_pool_unstarted()` (with the `rayon` feature it checks that Rayon's
+process-global pool was never started).
+
+## Wait benchmarks
+
+`benches/wait_scenarios.rs` measures how team threads wait, the way callers use them; the design
+is in [`docs/design/waiting.md`](docs/design/waiting.md). A change to the waits is judged by running
+it on the base and on the head of the change, on a quiet host and on a many-core shared host.
+
+| scenario | a call is | reports |
+| --- | --- | --- |
+| `sections` | 200 sections of 5, 20 or 50 µs, team held | per call, and the overhead over `sections * work` |
+| `gaps` | 10 sections of 5 µs, the caller sleeps 0 / 100 µs / 1 ms / 10 ms between calls | the first section (wake latency) and the call |
+| `acquire` | 50 sections of 20 µs: the team held (a token per call), the pool held (members acquired per call), a new pool per call | per call; for a new pool also the total from build to drop |
+| `nested` | 100 splits of the team into pairs, each pair one 20 µs section | per call |
+
+Each scenario runs at width = all CPUs of the affinity set and at three quarters of them, on
+threadteam and on a Rayon `scope`; `benches/omp/wait_omp.c` runs `sections` and `gaps` on libgomp
+(one `omp parallel` region per section, runtime defaults). `--load K:P` adds `K` child processes,
+each busy `P` % of every 10 ms, for the whole run: the stand-in for another user's processes.
+
+```sh
+# one checkout, both passes (quiet, then 8 processes at 10 %), rows appended to
+# docs/design/waiting-data/<host>.csv
+scripts/bench-waits.sh <label>                 # e.g. main-4c73c2b, then the PR head
+scripts/bench-waits.sh <label> 64,48 8:10      # explicit widths and load
+cargo bench --bench wait_scenarios -- --scenario gaps --widths 8   # one scenario; options in the file header
+```
+
+* **evo** (quiet): `scripts/bench-waits.sh <label> "" none` on the base and on the head. The
+  acceptance check is that `sections` and `gaps` do not regress.
+* **broccoli** (64 CPUs, shared): `taskset -c 0-63 scripts/bench-waits.sh <label> 64,48 8:10` on the
+  base and on the head, plus the faer#107 arms of `acquire_probe_107` (tensor4all/faer branch
+  `probe/107-acquire-per-call`). The acceptance check is that width 64 under load is no longer
+  slow and width 48 does not regress.
+
+To compare two commits on a shared host, `scripts/bench-ab.sh <base-rev> <head-rev> <rounds>
+[bench args]` builds both and alternates them, so a drift of other users' load falls on both sides;
+`HEAD_VARIANTS='--spin-polls 300000;--own-hold-ms 100'` adds head runs with tuning flags (this
+version only), and `OMP=1` runs the libgomp driver in every round with the same scenario, widths,
+calls and load (`acquire` is libgomp's held team). The base of the waits change is its bench commit `a3a9320` (the bench on the 0.1.3
+waits). Under every threadteam row, a `waits per call` line counts join and barrier sleeps, member
+parks and the preemptions waiters saw.
+
+Results are committed as CSV under [`docs/design/waiting-data/`](docs/design/waiting-data/), one
+file per host, with the label naming the commit; the summary tables live in `waiting.md`.
 
 ## License