[shinaoka]

Comparison experiment. Do not merge. main keeps the own-threads version (#16), and 0.2.0 is not published. This PR's base is exp/rayon-pool-reuse-base (= 625a593), so the diff shows only the experiment.

Refs #18, #14, #15, #16, tensor4all/faer#107. Assigned to shinaoka.

Question

Can faer#107 be solved on Rayon by reusing the pool between pools of one width, measured against the same acceptance criteria (a)-(d)?

Hypothesis: #14 failed (a) because every per-call pool started fresh threads, not because of how it waited.

What changes (on 625a593: Rayon, with libgomp's waits only)

• Cache. A Pool with default thread settings (no thread_name, on_start, stack_size or qos) keeps its rayon::ThreadPool in a process-wide cache on drop. The next Pool::build of the same width takes it instead of spawning threads.
  • Only idle entries are handed out, so two live pools never share a Rayon pool (tested).
• Drop. Before the threads go to the cache, the drop waits until the pool's members have left their loops (resident() == 0; shutdown makes idle members leave at once). So no member loop of the old pool runs on the threads when the next pool takes them.
• Affinity. On reuse, build reads the caller's CPU set and each worker applies it to itself (a Rayon broadcast), as a thread the caller spawned would get it.
• Cap. As in Own worker threads: a process-wide reserve of std::thread workers instead of a private Rayon pool (#15) #16: the larger of the available parallelism and the widest shared pool built so far, counted in threads. The oldest entries beyond it are dropped and joined, outside the lock.
• Unchanged paths.
  • Dedicated pools (any hook or setting) drop and join as before.
  • A pool dropped on one of its own workers is not cached: it detaches as before (drop_from_a_worker_of_the_pool_does_not_hang, unchanged).
• Diagnostic. threadteam::spawned_workers() (#[doc(hidden)], as on main) counts started worker threads.
• scripts/bench-ab.sh.
  • The summary parser fix from main (78b59a1).
  • HEAD2=<rev>: a third side, labelled head2, run in every round with the order rotated, so the three-way comparison stays interleaved.

Size and complexity (for the maintainer's comparison)

	this experiment (625a593..9dfcbdb, src/)	own threads (625a593..302c37d, src/)
lines	+190 / −31 in 4 files (pool.rs +126 code lines, including one unit test; the rest is docs and .inner → .inner())	+1562 / −543 in 12 files (about 1040 code lines)
new unsafe	none	43 unsafe blocks, fns or impls
new synchronization	a Mutex<Vec<_>> cache touched only in build and drop; a fetch_max width; a yield-then-sleep wait in drop for resident() == 0	slot state word, intrusive batch queue, broadcast list, cooperative waits (five new loom models)
idle_runs_rayon, external	keep their Rayon meaning	no-op; external lends to the pool's own queue
feature parallel	pulls Rayon	pulls nothing

There is no new loom model: the drop wait reads the existing slot states and adds no handshake; the existing 7 loom models pass.

Verification (primerose, loaded)

• Local gate on 9dfcbdb:
  • fmt; clippy with all features and with no default features;
  • tests with default features, all features (twice) and no default features;
  • loom: 7 models;
  • deny bans, no-public-rayon, cargo doc, and the aarch64-apple-darwin check;
  • taskset -c 0-2 runs of idle_clock, oversubscribed, team, rayon_reuse, team_owner and alloc_free, three times.
• New tests in tests/rayon_reuse.rs:
  • reuse: the same thread ids and zero new spawns across 5 rounds, including into_team on reused workers;
  • two live pools of one width have disjoint workers;
  • hook pools are dedicated and joined;
  • (Linux) a reused worker takes the building caller's CPU set.
    Plus the unit test the_cache_keeps_at_most_its_cap_of_threads.
  • Mutation checks: without the cache lookup the reuse test fails, and without the mask broadcast the affinity test fails.
• faer probe smoke: both new probe branches build acquire_probe_107 and run fresh_pool gemm128 4 5.
• bench-ab.sh with HEAD2: a 3-round smoke run at width 4 produced all three sides. The values are not meaningful: 4 CPUs and load average 35.

Acceptance runs (the monitor sends the requests)

Three sides in each run: base a3a9320 (bench commit, 0.1.3 waits), head exp/rayon-pool-reuse (9dfcbdb), and head2 302c37d (main, 0.2.0). Run from a checkout of this branch, so scripts/bench-ab.sh has HEAD2.

broccoli (64 threads):

HEAD2=302c37d OMP=1 taskset -c 0-63 scripts/bench-ab.sh a3a9320 exp/rayon-pool-reuse 10 --scenario acquire --widths 64 --calls 40 --impl threadteam
HEAD2=302c37d OMP=1 taskset -c 0-63 scripts/bench-ab.sh a3a9320 exp/rayon-pool-reuse 10 --scenario acquire --widths 64 --calls 40 --impl threadteam --load 8:10
HEAD2=302c37d OMP=1 taskset -c 0-63 scripts/bench-ab.sh a3a9320 exp/rayon-pool-reuse 10 --scenario gaps --widths 64 --calls 40 --impl threadteam --load 8:10

(a): acquire_probe_107, every arm (held, team_per_call, exec_team_per_call, into_team_per_call, fresh_pool, with_team), on three tensor4all/faer branches:

side	faer branch	threadteam pin
base	probe/107-acquire-per-call	0.1.3
head	probe/107-acquire-per-call-rayon-reuse (092234e)	9dfcbdb
head2	probe/107-acquire-per-call-v0.2.0 (498e4ec)	302c37d; faer's requirement raised to 0.2.0 for the probe only

The existing probe/107-acquire-per-call-own pins 0e259b5, which is before #16's post-review fixes, so it is not the 0.2.0 side. The head and head2 probes read PROBE_SPIN_POLLS only.

evo (quiet):

HEAD2=302c37d OMP=1 scripts/bench-ab.sh a3a9320 exp/rayon-pool-reuse 10 --scenario gaps --impl threadteam
HEAD2=302c37d OMP=1 scripts/bench-ab.sh a3a9320 exp/rayon-pool-reuse 5 --scenario acquire --calls 40 --impl threadteam
HEAD2=302c37d OMP=1 scripts/bench-ab.sh a3a9320 exp/rayon-pool-reuse 5 --scenario sections --impl threadteam

Not for merge. No publication.

🤖 Generated with Claude Code

[shinaoka]

evo acceptance run for #19. I ran the three evo commands from the PR body (HEAD2=302c37d OMP=1) from a checkout of this branch at 9dfcbdb.

side	commit
base	a3a9320 (bench commit, 0.1.3 waits)
exp	9dfcbdb (exp/rayon-pool-reuse)
0.2.0	302c37d (main)

I passed the head as the SHA 9dfcbdb, which is the branch head.

Conditions were the same as on #16:

• quiet host, with boost=0 checked before each command;
• observer (the agent and herdr) on 0-3,12-15;
• measurement under taskset -c 4-11;
• threadteam at widths 8 and 6, libgomp at width 8;
• foreign load none in all 60 passes. The load average of 4 to 7 during acquire and sections is the bench's own spinning members.

Verdict: exp passes (b) and (d).

• The largest gap to the base is 1.024 on gaps 0 to 1 ms and 1.008 on held_team/held_pool, both within 10 %.
• The 10 ms gap is 0.99 to 1.02 with the default settings.
• Next to 0.2.0, exp is consistently 1 to 2 % slower: on the gaps call, on sections 5 µs (1.021 to 1.027 against 1.004 to 1.008), and on the fresh_pool total (1096 / 1100 against 1060 / 1076 µs).
• That is far below the 10 % in criterion (b), so evo does not separate the two designs. The decision rests on broccoli's (a) and (c).

gaps, 10 rounds (median of the round medians, µs; first section / whole call; ratio to base in parentheses)

width	gap	base	exp 9dfcbdb	0.2.0 302c37d	libgomp
6	0	5.9 / 57.7	6.0 / 58.9 (1.017 / 1.020)	5.9 / 58.4 (0.998 / 1.013)	-
6	100 µs	5.9 / 57.7	6.0 / 58.9 (1.014 / 1.021)	5.9 / 58.4 (0.997 / 1.013)	-
6	1 ms	5.9 / 57.7	6.0 / 59.0 (1.020 / 1.022)	5.9 / 58.5 (1.007 / 1.013)	-
6	10 ms	5.9 / 57.6	5.8 / 58.6 (0.993 / 1.017)	5.9 / 58.4 (1.002 / 1.013)	-
8	0	6.1 / 59.2	6.2 / 60.4 (1.024 / 1.020)	6.0 / 59.5 (0.993 / 1.005)	6.5 / 64.3
8	100 µs	6.1 / 59.4	6.2 / 60.5 (1.011 / 1.019)	6.0 / 59.5 (0.985 / 1.003)	6.5 / 64.2
8	1 ms	6.1 / 59.4	6.2 / 60.5 (1.017 / 1.018)	6.1 / 59.6 (0.993 / 1.004)	6.7 / 64.5
8	10 ms	6.0 / 59.0	6.1 / 60.1 (1.012 / 1.018)	6.0 / 59.5 (0.993 / 1.008)	15.3 / 73.1

acquire --calls 40, 5 rounds (µs)

width	arm	base	exp	0.2.0	libgomp
6	held_team	1037.4	1045.2 (1.008)	1041.8 (1.004)	-
6	held_pool	1045.4	1047.7 (1.002)	1045.1 (1.000)	-
6	fresh_pool, call	1038.8	1044.4 (1.005)	1041.9 (1.003)	-
6	fresh_pool, build to drop	1229.2	1096.5 (0.892)	1060.2 (0.862)	-
8	held_team	1048.1	1052.6 (1.004)	1048.6 (1.001)	1071.4
8	held_pool	1054.4	1056.3 (1.002)	1052.7 (0.998)	-
8	fresh_pool, call	1046.3	1052.4 (1.006)	1048.2 (1.002)	-
8	fresh_pool, build to drop	1316.9	1100.1 (0.835)	1075.6 (0.817)	-

The fresh_pool phases at widths 6 / 8 (µs; the base bench predates them):

phase	exp	0.2.0
build	20.9 / 15.0	6.8 / 12.1
acquire	6.0 / 6.1	4.5 / 6.1
release	0.6 / 0.9	0.6 / 0.8
drop	24.0 / 24.4	5.3 / 8.5

Both reuse paths remove most of the base's one-off cost of about 180 to 270 µs.

• exp's drop is about 24 µs because it waits for resident() == 0 before caching. 0.2.0's drop is 5 to 9 µs.
• exp's build at width 6 is 21 µs, against 0.2.0's 7 µs. That is the affinity broadcast on reuse.

sections, 5 rounds (per call, µs)

width	work × 200	base	exp	0.2.0	libgomp
6	5 µs	1158.5	1182.4 (1.021)	1162.8 (1.004)	-
6	20 µs	4150.4	4180.8 (1.007)	4165.7 (1.004)	-
6	50 µs	10155.1	10179.2 (1.002)	10166.8 (1.001)	-
8	5 µs	1179.6	1210.9 (1.027)	1189.4 (1.008)	1275.5
8	20 µs	4178.4	4208.8 (1.007)	4187.8 (1.002)	4275.7
8	50 µs	10177.4	10203.1 (1.003)	10186.5 (1.001)	10274.4

Against the criteria (evo covers (b) and (d))

• (b) passes for exp.
  • held_team and held_pool are 1.002 to 1.008 of the base; gaps 0 to 1 ms are 1.011 to 1.024.
  • 0.2.0 is 0.985 to 1.013 on the same cells.
• (d) passes for exp.
  • The 10 ms gap is 0.993 / 1.012 on the first section and 1.017 / 1.018 on the call, at widths 6 / 8.
  • 0.2.0 is 1.002 / 0.993 and 1.013 / 1.008. libgomp is 15.3 / 73.1 µs on that cell.
• exp against 0.2.0 on this host: exp is about 1 % slower on the gaps call, 2 % slower on sections 5 µs, and 3 to 4 % slower on the fresh_pool total. None of these approach a criterion.
• (a) and (c) need the broccoli runs; this run does not cover them.

Raw rows and the waits per call lines are in shinaoka-advisor/work/runs/threadteam-19/ (gaps10, acquire5, sections5, .csv and .log).

[shinaoka]

Acceptance runs on broccoli — measured head 9dfcbdb (exp/rayon-pool-reuse), head2 302c37d (0.2.0)

Conditions as on #16: cpufreq/boost read back 0, governor performance, taskset -c 0-63, OMP=1,
a 5 s load logger alongside, HEAD2=302c37d scripts/bench-ab.sh a3a9320 9dfcbdb 10 (base a3a9320,
head2 = main/0.2.0), width 64, acquire, --calls 40, then gaps under load. (a) is
acquire_probe_107 on the three probe branches of the table (base c1d8e4c, exp 092234e, v0.2.0
498e4ec), built against threadteam 0.1.3 / 9dfcbdb / 302c37d respectively, 6 processes per arm
(gemm128, 50 calls) and 5 for chol2048.

(a) faer#107 arms, 64T, ms per call

arm	kernel	base (0.1.3)	exp 9dfcbdb	0.2.0 302c37d
held	gemm128	0.02	0.02	0.02
team_per_call	gemm128	0.02	0.02	0.02
exec_team_per_call	gemm128	0.02	0.02	0.02
into_team_per_call	gemm128	0.02	0.02	0.02
fresh_pool	gemm128	15.38 (6/6)	0.04	0.02
with_team	gemm128	15.38 (6/6)	0.04	0.02
held	chol2048	13.02	13.85	13.36
fresh_pool	chol2048	150.91	17.95	19.72
with_team	chol2048	135.01	23.10	18.14

(a) met by both heads: no process in any of the six arms, on either head, is in the slow mode, and the
chol2048 collapse (151 / 135 ms) is gone (18-23 ms).

(b) quiet, µs per call

arm	base	libgomp	exp	0.2.0
held_team	1 548	1 684	1 494 (−3.5 %)	1 484 (−4.1 %)
held_pool	1 539	–	1 648 (+7.1 %)	1 605 (+4.3 %)
fresh_pool total	150 459	–	4 005	13 671
fresh_pool phases: build / acquire / call / release / drop	–	–	112 / 2 248 / 1 555 / 14 / 57	139 / 11 547 / 1 583 / 13 / 115

(b) met by both, and the experiment's whole gain is visible in one column pair: the acquire phase of a
one-call pool is 2.25 ms with the reused Rayon pool against 11.5 ms when each pool builds its own, which
is why its fresh_pool total is 4.0 ms against 13.7 ms.

(c) under --load 8:10, µs per call

arm	base	libgomp	exp	0.2.0
held_team	1 502	1 680	1 533 (0.91x)	1 523 (0.91x)
held_pool	1 747	–	1 681 (0.96x base)	1 548 (0.89x base)
fresh_pool total	121 074	–	5 981	16 483
fresh_pool acquire	–	–	3 229	13 943

gaps under load, first section: exp 16.6 / 17.3 µs, 0.2.0 16.8 / 16.4 µs, base 17.1 / 17.1,
libgomp 19.3 / 77.5.

(c) met by both, at 0.91x libgomp on held_team and with the gap wake at 0.1.3's level.

Verdict

criterion	base (0.1.3)	exp 9dfcbdb	0.2.0 302c37d
(a) no slow mode in any arm	not met (fresh_pool/with_team 15.38 ms at gemm128, 151/135 ms at chol2048)	met (0.04 ms; 18/23 ms)	met (0.02 ms; 20/18 ms)
(b) quiet held within 10 %	reference	met (−3.5 / +7.1 %)	met (−4.1 / +4.3 %)
(c) loaded within libgomp	reference (0.89x libgomp)	met (0.91x)	met (0.91x)

Both heads pass all three criteria; the experiment and 0.2.0 differ only in the cost of a one-call pool:
acquire 2.2-3.2 ms against 11.5-13.9 ms, so fresh_pool total 4.0-6.0 ms against 13.7-16.5 ms — a 2.7-2.9x
advantage for the pooled variant on that arm, with the held arms, the gap rows and the faer#107 arms equal
within noise. That is the comparison the branch set out to make, and it is the only place the two differ on
this host.

Nothing here is a merge recommendation: the held arms, gaps and every acquire_probe_107 arm are the same
for both, so the choice turns on whether a per-call pool that costs 2.2 ms instead of 11.5 ms to acquire is
worth a process-wide reused Rayon pool.

[shinaoka]

Superseded by #21 (main back to Rayon with this pool reuse, 0.2.0). Kept as the record of the three-way acceptance runs.