diff --git a/Cargo.toml b/Cargo.toml index 43299cc..e85464b 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -13,9 +13,13 @@ categories = ["concurrency", "science"] [features] default = ["parallel"] -# Run entries on a private Rayon pool. Without it the same types exist, every -# pool has width 1 and every entry runs inline on the calling thread. -parallel = ["dep:rayon"] +# Run entries on the pool's own worker threads (docs/design/own-threads.md). +# Without it the same types exist, every pool has width 1 and every entry runs +# inline on the calling thread. +parallel = [] +# Opt-in Rayon interop: `Exec::external_rayon` runs a callback inside a private +# Rayon pool of the token's width. Not needed by anything else. +rayon = ["parallel", "dep:rayon"] # Test support: per-root live-task ledger and entry counters. Zero cost when off. ledger = [] # Test support: hooks that pause threads inside the park/wake handshake and @@ -45,6 +49,8 @@ libc = "0.2" [dev-dependencies] libc = "0.2" +# The Rayon reference rows of benches/wait_scenarios.rs. +rayon = "1.10" [[bench]] name = "team_dispatch" @@ -83,3 +89,8 @@ unexpected_cfgs = { level = "warn", check-cfg = ['cfg(loom)'] } [[bench]] name = "cost_lookup" harness = false + +[[bench]] +name = "wait_scenarios" +harness = false +required-features = ["parallel"] diff --git a/README.md b/README.md index efd0948..a919e8f 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,16 @@ that starts tasks: kernels take an `Exec` token and split it, and no scheduler type appears in any public signature. Used by strided-rs, tcontract-rs, cpueinsum-rs, tlinalg-rs and tenferro-rs. -Design record: [`docs/design/placement.md`](docs/design/placement.md). +Design records: [`docs/design/placement.md`](docs/design/placement.md) (the token), +[`docs/design/own-threads.md`](docs/design/own-threads.md) (the worker threads), +[`docs/design/waiting.md`](docs/design/waiting.md) (how they wait). + +The workers are threadteam's own `std::thread`s. A pool leases them from a +process-wide reserve and gives them back when dropped, so a pool built after another +one was dropped reuses running threads (faer's `with_team` builds one per call); +a pool with a thread name, a start hook, a stack size or a QoS of its own gets +dedicated workers that its drop joins. Rayon is not used unless the opt-in `rayon` +feature asks for `Exec::external_rayon`. ## Usage @@ -58,8 +67,9 @@ the token to kernels by value. Kernels never store a token and never mint one. | `Exec::team(f)` / `team_with(over, f)` | reserve `budget - 1` awake pool members for a scope and run barrier-bearing sections through `TeamExec` (`run`, `run_with`, `split`, `fork`, `reborrow`), or `Err(Refused::Unavailable)`; see the design record | | `Pool::into_team(width)` -> `Team` (owns the pool; `IntoTeamError` gives it back), `Team::exec(budget)` / `try_exec`, `Team::pool`, `Team::into_pool` | persistent reservation for an engine that owns its pool (`Pool` is not `Clone`); minted tokens are ordinary `Exec`s whose `fork`, `for_ranges`, `broadcast`, `team` dispatch on the reserved members; one leader at a time (`exec` blocks, `try_exec` -> `Refused::Busy`); idle members park while reserved | | `PoolBuilder::qos(Qos)`, `raise_current_thread_qos(Qos)` | worker QoS on macOS (default `UserInitiated`); raise the leading thread without ever lowering it | -| `PoolBuilder::idle_spin`, `idle_runs_rayon`, `acquire_timeout`; `Pool::release_idle` | waiting policy of idle team members; a reserved member spins with one PAUSE per poll (never `sched_yield`), parks after the delay, and parks after a short spin in a root scope that has more participants than the caller's CPUs | -| `PoolBuilder::pin_participants(bool)`, `Pool::pin_participants()` | opt-in placement of the **members** of a root team scope (Linux; off by default, which is consistent with the rule it protects): member `k` (participant `k >= 1`) pins itself onto the `k`-th CPU of the calling thread's affinity set, `S[0]` is left to the leader, and a member restores its own mask before it returns to Rayon or serves foreign work. **The leader is never pinned**, because a vendor library creates its thread team from the calling thread and a one-CPU caller would confine the whole provider team; a member that calls such a library directly inside a share does confine it, so use `Exec::external` (which restores the member's mask first) or leave pinning off. For tensor4all/faer#107; give each team its own CPU set (`taskset -c`), there is no CPU allocator | +| `PoolBuilder::idle_spin`, `idle_runs_rayon`, `acquire_timeout`; `Pool::release_idle` | waiting policy of idle team members (an idle member spins for the spin budget unless `idle_spin` is zero, then sleeps; `idle_runs_rayon` has no effect and is kept for compatibility) | +| `PoolBuilder::spin_polls`, `Pool::spin_polls` | spin budget of every team wait, in polls (the shape of libgomp's `GOMP_SPINCOUNT`): a reserved member waiting for work, the leader in a join and a participant in a barrier spin that many polls, yield a few times, then sleep until woken; throttled to 100 polls in an oversubscribed scope or after a waiter was descheduled ([`docs/design/waiting.md`](docs/design/waiting.md)) | +| `PoolBuilder::pin_participants(bool)`, `Pool::pin_participants()` | opt-in placement of the **members** of a root team scope (Linux; off by default, which is consistent with the rule it protects): member `k` (participant `k >= 1`) pins itself onto the `k`-th CPU of the calling thread's affinity set, `S[0]` is left to the leader, and a member restores its own mask before it serves the pool's queue, leaves the pool, or serves foreign work. **The leader is never pinned**, because a vendor library creates its thread team from the calling thread and a one-CPU caller would confine the whole provider team; a member that calls such a library directly inside a share does confine it, so use `Exec::external` (which restores the member's mask first) or leave pinning off. For tensor4all/faer#107; give each team its own CPU set (`taskset -c`), there is no CPU allocator | | `DispatchCosts`, `WidthCosts`, `PoolBuilder::dispatch_costs`, `Exec::dispatch_cost(width)`, `TeamExec::dispatch_cost(width)` | cost of one parallel section per kind of token (team, in pool, outside), steady state; one row per `HostClass` (`HostClass::detect()`, keyed by kernel dispatch: `x86_64-avx512`, `x86_64-avx2`, `x86_64-scalar`, `aarch64-apple`, `aarch64`, `generic`), `DispatchCosts::for_class(class)`; measured rows for `x86_64-avx512` (team anchors at widths 2, 4, 6 and 8) and `aarch64-apple`, a measured Zen2 team row for `x86_64-avx2` (anchors at ten widths from 2 to 64; its plain kinds stay generic), every other class takes the conservative `generic` row (3 / 15 / 40 us at the anchors 2, 4, 6); a row holds measured anchors (`WidthCosts::new`, `WidthCosts::flat`, or `WidthCosts::from_anchors` for measured anchors up to 64 and beyond), interpolates between them and grows linearly above the largest one, so a width above the anchors is never charged the cost of the largest anchor; `DispatchCosts::host_default()` is the detected class's row; `Duration::MAX` when serial; override per pool | | `pays_at(est, sections, cost)`, `Exec::pays(est, sections, width)`, `Exec::width_for(est, sections)`, `width_for_by`, `PAYS` | the one rule: parallel iff `est >= PAYS * sections * D(width)`; `width_for` minimises `sections * D(k) + est / k` | | `Exec::external(f)` | enter the pool once and hand the budget as a plain width to an outside library (faer, a vendor BLAS); how it divides the width is that library's concern | @@ -72,11 +82,13 @@ the token to kernels by value. Kernels never store a token and never mint one. once is at most its budget at every nesting depth. A parent waiting for its children keeps no slot. * **Lazy and inline.** A serial token, budget one, or one task runs inline on the - calling thread and makes no scheduler call, so the process-global Rayon pool is - never started implicitly. -* **Placement by caller.** A worker of the target pool spawns into the current - scope without installing; a plain thread or a worker of another pool enters the - target pool once and waits. No thread-local state exists in the crate. + calling thread and makes no scheduler call; no worker starts before a pool is + built, and Rayon's process-global pool is never started. +* **Placement by caller.** A worker of the target pool runs its own batch's + tasks itself and serves the pool while it waits; a worker of another pool + publishes on the target pool and serves its own pool while it waits (so + chains between pools always make progress); a plain thread publishes and + sleeps until its tasks finished. No thread-local state exists in the crate. * **`broadcast`** refuses with `Refused::ForkChild` for a token derived from `fork` (through `reborrow` or `limit` too), and with `Refused::CallerInPool` when the caller is a worker of the same pool. Roots, and their `reborrow`s and @@ -86,11 +98,14 @@ the token to kernels by value. Kernels never store a token and never mint one. blocking work of another token on those workers delays the broadcast. `can_broadcast` applies the same rule through one private helper, so the two cannot disagree. -* **Deterministic shutdown.** Dropping a `Pool` joins every worker thread: when - `drop` returns, all workers have exited and their thread-local destructors - (including state created in an `on_start` hook) have run. A pool dropped from - one of its own workers cannot join itself, so that drop only signals shutdown - and returns without joining. Without `parallel` there is nothing to join. +* **Shutdown.** Dropping a `Pool` detaches every worker and waits until each + one acknowledged; after that no worker touches the pool. Dedicated workers (a + pool with a thread name, an `on_start` hook, a stack size or a QoS of its own) + are then joined: their thread-local destructors, including state created in + an `on_start` hook, have run when `drop` returns. Shared workers go back to the + process-wide reserve, asleep, for the next pool. A pool dropped from one of its + own workers does not wait for or join that worker, which acknowledges and + leaves after its task returns. Without `parallel` there is nothing to join. * **No scheduler type is public.** There is no constructor from, and no accessor returning, a scheduler pool. `scripts/check-no-public-rayon.sh` runs `cargo public-api -p threadteam` (when installed) and fails if `rayon::` @@ -100,12 +115,58 @@ the token to kernels by value. Kernels never store a token and never mint one. | Feature | Effect | | --- | --- | -| `parallel` (default) | run on a private pool. Without it the same types exist, `PoolBuilder::build` returns a width-1 pool and every entry runs inline | +| `parallel` (default) | run on the pool's own worker threads. Without it the same types exist, `PoolBuilder::build` returns a width-1 pool and every entry runs inline | +| `rayon` | opt-in Rayon interop: `Exec::external_rayon(f)` runs `f(width)` inside a private Rayon pool of the pool's width (separate threads, built on first use) | | `ledger` | test support: `ledger::max_live(root)`, `live`, `installs`, and `Exec::ledger_root`. Records live tasks per root token and pool installs; compiled out when off | Consumers assert `ledger::max_live(root) <= budget` in their own tests, and keep -a test binary that runs serial and pooled work and then checks that -`rayon::ThreadPoolBuilder::new().build_global()` still succeeds. +a test binary that runs serial and pooled work and then checks +`ledger::global_pool_unstarted()` (with the `rayon` feature it checks that Rayon's +process-global pool was never started). + +## Wait benchmarks + +`benches/wait_scenarios.rs` measures how team threads wait, the way callers use them; the design +is in [`docs/design/waiting.md`](docs/design/waiting.md). A change to the waits is judged by running +it on the base and on the head of the change, on a quiet host and on a many-core shared host. + +| scenario | a call is | reports | +| --- | --- | --- | +| `sections` | 200 sections of 5, 20 or 50 µs, team held | per call, and the overhead over `sections * work` | +| `gaps` | 10 sections of 5 µs, the caller sleeps 0 / 100 µs / 1 ms / 10 ms between calls | the first section (wake latency) and the call | +| `acquire` | 50 sections of 20 µs: the team held (a token per call), the pool held (members acquired per call), a new pool per call | per call; for a new pool also the total from build to drop | +| `nested` | 100 splits of the team into pairs, each pair one 20 µs section | per call | + +Each scenario runs at width = all CPUs of the affinity set and at three quarters of them, on +threadteam and on a Rayon `scope`; `benches/omp/wait_omp.c` runs `sections` and `gaps` on libgomp +(one `omp parallel` region per section, runtime defaults). `--load K:P` adds `K` child processes, +each busy `P` % of every 10 ms, for the whole run: the stand-in for another user's processes. + +```sh +# one checkout, both passes (quiet, then 8 processes at 10 %), rows appended to +# docs/design/waiting-data/.csv +scripts/bench-waits.sh