team route: some processes run every repetition at the 1-thread time (16T, broccoli)

Observation (broccoli, faer#99 re-measurement, 2026-10-29T02:26Z)

qr_factor f64 512 at 16T, 20 processes. Five of them were slow on every repetition (11.8 to 13.5 ms) against 3.6 to 4.2 ms for the rest. 11.8 ms is about the 1-thread time. The slow mode belongs to the process: it hits both arms and both orders, and every repetition inside a slow process is slow. The shape matches the ~0.62 ms TeamExec section reported on #91.

A 3x process-level mode is a hazard for any single-process comparison at these budgets.

What is ruled out by the code

A silent fall back to Par::Seq. utils::team::with_team panics if the pool or the team cannot be acquired, and the team is the leader plus width - 1 reserved members.
A gate that varies between processes. The gates read HostClass::detect() (cached feature detection) and a constant table, DispatchCosts::for_class. Nothing is calibrated at run time.
"The leader takes every chunk." Every chunk is assigned statically:
- faer's for_each_child hands block c to child c (par_split_indices);
- threadteam's split_raw and section run participant 0 on the leader and publish every other share to a fixed member slot, then join on a completion counter.
A late member delays the join, but its work cannot move to the leader. So option (2) in the monitor's note ("wait for the members' arrival before the leader takes the chunks") does not apply in that form. The leader already waits.

Probe and what it shows so far (primerose, width 4, not 16)

The probe is faer/examples/team_mode_probe.rs on branch fix/team-serial-mode (b113f83). Build it with --features threadteam/trace. For each process it records:

the time of each repetition, and faer's team counters;
per repetition, from threadteam's trace: the number of publishes and shares, the members that took part, park and unpark events, the latency from a publish to a member's SectionBegin, and the members' busy time;
per thread, from /proc/self/task: the last CPU, CPU ticks, and voluntary and involuntary context switches; plus Cpus_allowed_list.

qr_in_place 512 runs entirely through splits: 80 two-way and 90 four-way per 5 repetitions, and no flat sections.

CPUs	result
0-3 (shared with other users, load average ~50)	10-120 ms. Two team threads often sat on the same CPU with ~6k-40k involuntary context switches each. Publish-to-share latency 0.2-1.4 ms on average, 3-7 ms at most. 0 park events.
35-38 (idle)	6/6 processes fast at 2.62-2.66 ms, 4 distinct CPUs. Latency 65 us on average, 260 us at most. 0 parks. One process had a transient stacked pair, 6.9k switches, in its first repetition.

Reading so far. Within a call the members never park: they spin, then yield. The slow runs line up with team threads stacked on one CPU and preempting each other, not with a wake-from-park delay. The first repetition of every process costs ~37 ms against 2.6 ms afterwards, even on idle CPUs. The harness's 250 ms prime covers that, so it does not explain a mode that lasts the whole process.

Width 4 on idle CPUs does not reproduce the mode. It needs broccoli at 16T.

Request: run the probe on broccoli at 16T (via the monitor)

20 processes of team_mode_probe 16 512 20, under the same pinning and idle guard as the faer#99 run.
Keep the full stdout of every process, slow and fast.
Questions to answer:
- Placement: do slow processes show stacked team threads, with fewer distinct CPUs than 16 and large involuntary-switch counts?
- Parking: do any of them have park events?
- Latency: what is the publish-to-share latency in slow versus fast processes?
- CPU set: what is Cpus_allowed_list for 16T on that host (one L3 domain is 4 cores)?

How OpenMP runtimes and OpenBLAS wake members and wait, compared with threadteam

I read the sources only; no code was copied.

Source	Revision read
GCC libgomp	gcc-mirror `6aae4093ee77`
LLVM libomp (`openmp/runtime/src`)	llvm-project `ad13d6f017a2`
OpenBLAS (`driver/others`)	`e805f60561b6`
threadteam	threadteam-rs `416198e` (0.1.2)

Waking members, barriers, reuse, affinity

	libgomp	libomp	OpenBLAS pthreads	threadteam
Wake at region entry	Idle workers wait on one central barrier ("dock"). The last thread to arrive bumps a generation word and wakes all of them with one futex call (`config/linux/bar.c:37-43`).	Release goes down a tree with fan-out 4 (`kmp_global.cpp:79-87`). A waiter is woken with `pthread_cond_signal` only if it is marked as sleeping (`kmp_wait_release.h:806-819`).	The caller writes a job into each worker's slot, then signals each sleeping worker's condvar one by one (`blas_server.c:700-751`).	The leader bumps each member's generation counter. A parked member is woken through its own condvar. Linear, not a tree.
Barrier	A central counter plus a futex generation word	Hyper-tree by default; linear, tree, hierarchical and distributed are selectable	none; fixed per-thread jobs	A sense-reversing counter; the leader joins on a completion counter
Static chunks: can a late member's chunk be taken by another thread?	No (`iter.c:73`)	No (`kmp_sched.cpp:314-329`)	No	No (`split_raw` / `section`)
What persists between regions	The pool's threads stay docked (`team.c:460-488`)	Hot teams (`kmp_runtime.cpp:981-1003`)	A persistent pool	The team lives for the whole `with_team` scope
Threads pinned by default?	No (`env.c:75`)	No (`kmp_settings.cpp:6468-6483`)	No (`NO_AFFINITY=1`, `Makefile.rule:177`)	No

How spinning is kept cheap

	libgomp	libomp	OpenBLAS	threadteam
Pause per spin	1 PAUSE on x86 (`x86/futex.h:121-124`). On aarch64 only a compiler barrier, no instruction.	1 PAUSE on x86, none on aarch64 (`kmp.h:1563-1585`)	8 `nop`s on x86_64 and ARMV8 (`common.h:389-403`)	1 `spin_loop()` (PAUSE on x86, YIELD on aarch64)
Default spin before sleeping	300,000 spins with `OMP_WAIT_POLICY` unset, about 3 ms (`env.c:2494-2508`). ACTIVE: 3×10¹⁰. PASSIVE: 0, straight to the futex.	blocktime 200 ms, but 0 on hybrid CPUs (`kmp.h:1249`)	2²⁸ TSC ticks, roughly 0.1 s at 2-3 GHz (`blas_server.c:160-164`)	`idle_spin`, 200 ms by default
`sched_yield` while spinning	Never: spin, then futex	Yes. The default is to yield after 512 spins and then every 256 (`kmp.h:1350-1351, 1661-1673`). When oversubscribed, it yields on every spin.	No on x86_64/ARMV8	After 16,384 spins, every poll yields until the park delay ends (`awake.rs:60` `backoff`)
Backoff	None; fixed length	Only through TPAUSE, which is off by default	None	None
How it sleeps	futex	condvar	condvar	condvar, one per member
When there are more threads than CPUs	Counts the CPUs in the affinity mask (`proc.c:94-100`). With more threads than that, it switches to a throttled spin count: 100 with the policy unset, 1000 for ACTIVE (`env.c:2512-2517`, `wait.h:52-55`).	Detects `__kmp_nth > __kmp_avail_proc` and then yields on every spin (`kmp.h:1594-1601`)	Caps the thread count at the CPU count and does not adapt	Nothing
Leader at the join	(not read)	spin, yield, then sleep, as above	spins on `nop`s and never sleeps (`blas_server.c:764-777`)	the same `backoff`: spin, then yield

What threadteam does differently

All four runtimes assign chunks statically, never pin threads by default, and spin before they sleep. threadteam is the odd one out in three ways:

Its spin turns into an unbounded run of sched_yield calls. After 16k PAUSEs, every poll yields until the park delay (200 ms) ends, and the leader's join and barrier do the same.
- A thread in that loop is always runnable, so the scheduler sees every CPU as busy.
- If two team threads land on one CPU, nothing pulls one of them away, and the stacking can last for the whole process. That fits a mode that is a property of the process. The primerose probe showed exactly that pattern in the slow runs: stacked threads with tens of thousands of involuntary switches. broccoli at 16T still has to confirm it.
- libgomp never yields; it sleeps after about 3 ms. libomp yields, but sleeps after its blocktime.
It does not detect oversubscription. libgomp throttles its spin to about 100 iterations when there are more threads than CPUs in the mask; libomp yields on every spin.
Its default spin is long, 200 ms, measured from the last share. This is the same length as libomp's blocktime, but libomp sets it to 0 on hybrid CPUs and libgomp uses about 3 ms. On a shared host, a member left between calls burns its CPU for 200 ms.

Recommendation

These are changes in threadteam's waiting policy, so they are tracked in a threadteam-rs issue. The order assumes broccoli confirms the stacking.

Spin cheaply and for a bounded time, without a yield phase, then park on the futex/condvar. This is libgomp's shape.
- Inside one faer call, members keep spinning between sections. This is option (1) of the monitor's note, the in-call scope: no park inside a factorization.
- Between calls, park after a short delay: the order of 1 ms instead of 200 ms.
- Replace the yield loop with the park. A parked thread is not runnable, so the scheduler can move a stacked thread onto a free CPU.
- The cost is the wake latency of the first section after a park. threadteam's own docs (cost.rs:338-340) put a section after the members parked at about 27 us on the measured host. On idle CPUs the probe measured 65 us on average from publish to share, but that figure includes nested publishes.
Throttle the spin when oversubscribed, as libgomp does. If the reserved members of all teams in the process outnumber the CPUs in sched_getaffinity, spin only about 100 PAUSEs before parking. On a shared host this is what keeps the spin from eating other users' CPU.
Keep threads unpinned by default, as all four runtimes do. Pinning would fix the stacking, but it takes CPUs away from other users and fights the caller's taskset. If it is wanted, it should be an opt-in, like OMP_PROC_BIND.
Defaults for how fast and how long to spin:
- one PAUSE per iteration (YIELD on aarch64);
- inside a call, spin until the next section;
- between calls, a park delay of about 1 ms, configurable per pool or through WaitOverride, the way GOMP_SPINCOUNT and KMP_BLOCKTIME are;
- when oversubscribed, about 100 spins.
These numbers are provisional. They are what libgomp ships, and they should be checked against the broccoli measurement before they are adopted.

Not settled: whether the slow mode on broccoli is the stacking. The 16T probe run is still pending. If it is not, recommendation 1 is still justified by the CPU cost on a shared host, but the fix for #107 is something else.

Not verified

Whether libomp's monitor thread is built by default, which decides whether its forced zero blocktime takes effect.
How a libgomp master waits at the join.
Whether every aarch64 OpenBLAS target defines ARMV8.
The wall-clock length of OpenBLAS's TSC-tick timeout on a given machine.

threadteam side: tensor4all/threadteam-rs#9 (waiting policy). The fix PR waits for the broccoli 16T probe run to confirm the stacking.

20 processes of `team_mode_probe 16 512 20` on broccoli: 10 fast, 10 slow, and what separates them

Run via the monitor, 2026-10-29, EPYC 7702P (Zen2, 64 cores, 16 MiB L3 per 4-core CCX, SMT off),
boost 0, governor performance, probe built at b113f83 with --features threadteam/trace, each
process pinned to CPUs 48-63 (four whole L3 domains) with a 3 s whole-domain idle gate before and
after it, agent pinned off the measured set. Cpus_allowed_list: 48-63 in every process, w=16,
n=512, 20 repetitions. Every process's full stdout is kept.

The mode reproduces exactly and splits the sample in half: 10 processes at 3.47-3.53 ms median,
10 at 12.39-12.47 ms. 12.4 ms is the one-thread time of this call, so in the slow mode the members'
work stops overlapping. Six of the ten fast processes also contain one to six individual 12.4 ms
repetitions, so the state is per repetition; a process is slow when it never leaves it.

	fast (n=10)	slow (n=10)
median per process	3.47-3.53 ms	12.39-12.47 ms
publishes / shares / members per rep	15 / 197 / 15	identical
`split` histogram, `below_floor`	`[.., 40@3, 260@17]`, 15320	identical
park / unpark	0 / 0	0 / 0
publish-to-share latency (median of reps)	221-225 us	828-842 us
distinct CPUs, busy threads	16 of 16, all ten processes	15 of 16 in 6 processes, 16 in 4
CPU ticks summed over the team	190-296	363-433 (1.66x)
involuntary switches, max thread (median)	26.7k	31.1k (ranges overlap; one fast process: 34)

What the differences say

Not the schedule. Publishes, shares, members, the split-width histogram and below_floor are
byte-for-byte the same in fast and slow processes. Nothing decided differently.
Not parking. park 0 unpark 0 in all twenty processes, every repetition.
The "members are woken from park late" reading in the issue is ruled out: the members never park.
The latency is the symptom, not the cause. 15 publishes per repetition at 15 us each (fast)
against 56 us each (slow): the hand-off is what stretches, by the same factor as the call.
CPU burned, and where it ran. A slow process burns 1.66x the CPU for identical work, and in six
of the ten the team ends up with two threads on one CPU while one of the sixteen allowed CPUs is
idle, where all ten fast processes end on sixteen distinct CPUs. The four slow processes that still
show 16 distinct CPUs weaken it, but the probe samples each thread's CPU once, at the end, so a
stacked pair that is later rescheduled leaves no trace.

So the standing candidate is placement/oversubscription, with the caveat above: the probe cannot yet
say whether a share actually ran on a CPU shared with another share. What would answer it: sample
sched_getcpu() at the start of every share (per member, per publish) rather than once per thread,
and log the leader's own CPU per share; then "two shares of one publish ran on one CPU" is directly
observable. Cheaper probe-side checks that would also separate the cases: build the pool with
width - 1 members when the leader is also a participant (17 threads on 16 CPUs is the healthy
oversubscription you start from), and run the same 20 processes with the pool's threads explicitly
confined to the 16 CPUs.

If the per-share CPU sample shows the shares of one publish landing on one CPU, the fix belongs in
threadteam (member placement or the spin/yield policy at section entry) rather than in faer; if it
shows clean placement and 16 us hand-offs that simply do not overlap, the next suspect is the join.

Probe update for the next broccoli run (`fix/team-serial-mode` at `df7e803`)

One correction to the 17-on-16 reading

with_team(16) builds a pool of 16 workers, but the team is the leader (the calling thread) plus 16 - 1 reserved members. threadteam's Pool::into_team(w) acquires w - 1 members.

The spare worker sleeps. The 16th worker is never claimed. On primerose the probe samples its state as S throughout the timed loop.
So the team is 16 spinning threads on 16 CPUs, not 17. That is full subscription with no slack, and the measured "16 busy threads" in every fast process agrees.
Option (1) of the monitor's note is therefore already the layout. The comparison that adds one free CPU is a team of 15 on the same 16 CPUs (team_mode_probe 15 ... under taskset -c 48-63).

What the probe now records (both options are environment variables)

PROBE_SAMPLE=<cpu>: a sampler thread pins itself to <cpu>, which must be outside the measured set. During each repetition it loops over /proc/self/task/*/stat and reads every thread's state and CPU. Per repetition it reports:
- the number of samples;
- how many samples had two runnable (R) threads on the same CPU, meaning one of them was waiting on that CPU's run queue;
- the largest number of threads runnable at once.
This shows directly whether two team threads shared a CPU while a repetition ran, not just where each thread ended up.
PROBE_PIN=48-63: a section run on the team at setup pins participant i (the leader is 0) to list[i] with taskset -pc, each on its own thread. The spare worker is not a participant and is left alone.

On primerose, at width 4 on idle CPUs, a repetition gives 31 samples:

unpinned, the first repetition had a stacked pair in 43 % of its samples, and the later ones in 0 %;
pinned, every repetition had 0 %.

Suggested broccoli runs

Use the same protocol as the 03:07Z run, with the sampler on a CPU outside 48-63 and outside the agent's set (44 below is a placeholder), 20 processes each:

arm	command
A, as before, sampled	`PROBE_SAMPLE=44 taskset -c 48-63 team_mode_probe 16 512 20`
B, one free CPU	`PROBE_SAMPLE=44 taskset -c 48-63 team_mode_probe 15 512 20`
C, pinned	`PROBE_PIN=48-63 PROBE_SAMPLE=44 taskset -c 48-63 team_mode_probe 16 512 20`

How to read the result

The slow processes of A show stacked samples, and B and C have no slow mode. Then the mechanism is placement under exact subscription, with spinning or yielding threads that the scheduler never rebalances. The fix belongs in threadteam-rs#9: count the leader, and when the participants fill every CPU, spin with PAUSE, then park instead of an endless sched_yield. Pinning stays opt-in.
The slow processes of A show no stacked samples. Then placement is ruled out, and the next suspect is the join or hand-off itself.

A/B/C on broccoli, 20 processes each, arms interleaved (probe at `df7e803`, sampler on CPU 8)

Same protocol as the 03:07Z run: EPYC 7702P (Zen2, SMT off, boost 0, governor performance), each
process pinned to 48-63, 3 s whole-domain idle gate before and after, agent off the measured set,
PROBE_SAMPLE=8 (outside 48-63 and outside the agent's CPUs), n=512, 20 repetitions.

arm	slow (≥6 ms)	median of medians fast / slow	stacked samples, mean per process (fast / slow)	worst process	publish-to-share us (fast / slow)
A width 16, unpinned	11 / 20	3.52 / 12.43 ms	71.7 % (46.3 / 92.4)	100 %	225 / 835
B width 15, same 16 CPUs	7 / 20	3.55 / 12.46 ms	52.3 % (31.5 / 90.8)	98.6 %	196 / 757
C width 16, `PROBE_PIN=48-63`	0 / 20	3.44 / –	0.0 % (0.0 / –)	0 %	218 / –

Reading

The slow mode is stacking, and the stacked-sample counter shows it directly. In A every slow
process spends 83-100 % of its samples with two runnable team threads on one CPU (mean 92.4 %), every
fast process far less, and the publish-to-share latency splits the same way (835 against 225 us). This
is the 03:19Z "reading 1": the members are alive, unparked, and receiving the same shares; two of them
are simply queued on one CPU, so their shares stop overlapping and the call takes the one-thread time.
The fast processes' 46.3 % stacked mean also explains the single 12.4 ms repetitions inside them.

Pinning removes it completely: C is 20/20 at 3.4-3.5 ms with zero stacked samples in all
twenty processes, on the same CPUs, same width, same work.

One correction to the suggested fix. B — one free CPU, which is option (1) of the monitor's note —
still fails 7 of 20 processes, and its slow processes are as stacked as A's (90.8 %). A spare CPU
gives the scheduler somewhere to move a displaced member but does not stop it from stacking the pair
in the first place, so "count the leader, PAUSE-then-park instead of endless sched_yield" would not
by itself remove the mode on this host. What does remove it is deterministic placement: the fourteen
extra CPUs of this host are there to be assigned, and the probe's PROBE_PIN shows the assignment is
enough (0 % stacked, 0/20 slow).

So for threadteam-rs#9: participant affinity is the load-bearing part, at least when the participants
fill every allowed CPU; parking may still help the tail (the 46 % stacked samples inside fast
processes did not make them slow), but it is not what fixes A. Opt-in, as the probe keeps it, with the
caller's CPU set as the default source.

Raw output of all 60 processes is in the run dir (per process: repetition times, publish/share/park
counters, per-repetition sampler counts, per-thread CPU/ticks/context switches).

Control: OpenBLAS `dgeqrf` 512 at 16T, 20 processes — the per-process mode does not reproduce

Blocker first: the OpenMP-default control cannot be run on broccoli. The only BLAS here is
libopenblas.so.0 = OpenBLAS 0.3.8 NO_LAPACKE DYNAMIC_ARCH NO_AFFINITY Zen MAX_THREADS=64
(openblas_get_config()), i.e. the pthread flavour: ldd shows libpthread and no libgomp,
and there is no openblas-openmp build anywhere on the host (/usr/lib/.../openblas-pthread is the
only variant directory; libgomp.so.1 exists but nothing links it). So OMP_PROC_BIND,
OMP_WAIT_POLICY and GOMP_SPINCOUNT had no effect to record — they never reached a runtime. What a
GNU OpenMP build would default to (proc_bind false, active waiting with GOMP_SPINCOUNT=300000) is
documentation, not a measurement from here, so I am not quoting it as one.

What the installed flavour gives, same protocol as the A/B/C run (20 processes, taskset -c 48-63,
3 s whole-domain idle gate before and after each, openblas_set_num_threads(16) read back as 16,
512x512 f64, 20 repetitions per process, matrix restored outside the clock), plus one 1T reference
process:


1T reference	18.48 ms
16T, 20 processes	12.55 / 12.72 / 12.73 / 13.51 / 14.14 / 14.16 / 14.26 / 14.26 / 14.29 / 14.31 / 14.34 / 14.40 / 14.40 / 14.42 / 14.45 / 14.46 / 14.49 / 14.50 / 14.51 / 15.24 ms
speedup over 1T	1.29x, and 20 of 20 processes are at 68-82 % of the 1-thread time

No fast/slow split: there is only the slow level. Not one process reached anything like the
3.4-3.5 ms faer's team holds in its fast mode (5.4x over its own 1T), and the process-to-process
spread is 12.55-15.24 ms instead of two clusters 3.5x apart. On this host and with this pinning,
OpenBLAS at 512 simply does not scale at 16 threads.

That is worth reading carefully in both directions:

It weakens "any thread pool on this shared host collapses at 16T" as the explanation: a
different pool implementation, on the same CPUs, same shape, same process discipline, shows no
mode. That is consistent with the A/B/C result that the mode is about the team's own placement
under exact subscription, not about the machine being busy.
It does not make the absolute level a discriminator: OpenBLAS's 14.3 ms median sits on top of
faer's slow mode (12.4 ms), so "the call took ~14 ms" tells you nothing on its own. What
distinguishes the fault is the bimodality within a fixed configuration (and it disappears under
PROBE_PIN), not the number.

If the OpenMP-default control is still wanted, it needs an OpenMP-built vendor library on this host
(libopenblas-openmp, MKL, or BLIS with OpenMP); installing one needs root, which I do not have
here. Tell me when it is available and I will run the identical 20-process protocol against it and
record the effective OMP_* values from the process environment.

placement: opt-in participant pinning, PAUSE-only reserved spin, over…

threadteam-rs#10 re-checked against #107's mode (20 processes per arm, 16T, CPUs 48-63)

arm	slow processes (≥6 ms)	stacked samples, mean
threadteam main (0.1.2)	5 / 20	47.8 %
PR #10, pinning off	0 / 20	28.4 %
PR #10, `pin_participants(true)`	0 / 20	3.4 %

The slow mode is gone in both PR arms — the PAUSE-only spin and the oversubscription park do it on
their own, before any pinning — and member-only pinning removes most of the residual stacking. The
unpinned leader keeps the whole mask and landed on the CPU the members leave free, so it does not
stack with them. Full table and caveats (20 processes bounds the rate below ~14 %): comment on
tensor4all/threadteam-rs#10.

OpenMP-default control: it does not stack either, and the settings it runs under

Two instruments, both with OMP_PROC_BIND unset, taskset -c 48-63, 16 threads, 20 processes each,
the same whole-domain idle gate as before.

1. libopenblas-openmp dgeqrf 512 (libopenblas0-openmp installed by the maintainer today;
linked explicitly — rpath plus LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/openblas-openmp — and
openblas_get_config() confirms … NO_AFFINITY USE_OPENMP Zen MAX_THREADS=64, threads requested 16,
read back 16):


1T	18.44 ms
16T, 20 processes	13.90 / 14.16 / 14.75 ms (min / median / max), all twenty between 13.9 and 14.8

No split, no fast cluster — the same shape as the pthread build measured earlier (12.55-15.24 ms), so
this vendor code path is uniformly near its own 1T time at 512 and cannot exhibit the mode either way.

2. A gcc -fopenmp region probe (probes/hh107e/ompprobe.c): 20 iterations of 300 back-to-back
short regions, 16 shares of ~5 us each, every share recording sched_getcpu(); a region is "stacked"
when its shares do not occupy 16 distinct CPUs.


1T iteration	1.63 ms
16T iteration, 20 processes	2.71-2.87 ms (median 2.78)
stacked regions	0.0-0.1 % in every one of the 20 processes

So the default OpenMP runtime never leaves two runnable threads of one region on one CPU across
1.2 million regions on this host, where threadteam's team spent 28-48 % of its samples stacked in the
same configuration.

The settings it runs under, and the difference to threadteam

OMP_DISPLAY_ENV=VERBOSE reports, with nothing set by me: OMP_PROC_BIND='FALSE',
OMP_DYNAMIC='FALSE', OMP_WAIT_POLICY='PASSIVE', GOMP_SPINCOUNT='300000'. That is the
interesting half of the comparison: libgomp binds nothing either — its threads are free to be placed
anywhere in the mask, exactly like the team's members — and it still does not stack, because of how
its workers wait and how a region starts. Between regions a libgomp worker spins for a bounded number
of polls and then sleeps, so the OS is free to move it and there is no resident set of 16 pollers to
collapse onto fewer CPUs; a region begins with a barrier that only releases when every worker is
resident, and only then do the shares run. threadteam's reserved members instead poll continuously
across calls, so a member that the scheduler happens to deschedule shares its CPU with a peer
mid-section, and the section's join waits for it — which is precisely the 3x we measured (the slow
processes' publish-to-share latency was 835 us against 225 us). The fix in tensor4all/threadteam-rs#10
(PAUSE-only spin, and park when the scope is oversubscribed) moves threadteam towards what libgomp
does by default, which is consistent with that PR's arms showing 0/20 slow processes with pinning off.

placement: opt-in participant pinning, PAUSE-only reserved spin, over…

Is the slow mode a long-lived in-share stack? Split the stacked samples and measure how long they last

Probe change (local commit on fix/team-serial-mode, not pushed; diff and sources in the run dir):
every sample now records the timestamp and each thread's state and CPU, and after each repetition the
sample is classified against the threadteam trace — a share's window is SectionBegin..SectionEnd
per member, matched to the sampled thread through the participants' tids. A stacked sample (two
runnable threads on one CPU) is counted as

(a) in share: at least one of the two is inside a share at that instant;
(b) waiting: both are between sections.

and the consecutive length of each kind of stack is recorded (samples and the wall time it spans).
Three arms, 16T, taskset -c 48-63, 20 processes each, the usual guard, repetition 0 (the ~60 ms
warm-up) excluded from every number below.

arm	slow (≥6 ms)	(a) in share	(b) waiting	longest (a)	longest (b)
threadteam `416198e`, fast processes	– (3.53 ms)	2.5 % of samples	40 %/sample	92 samples, 19.5 ms	20 samples, 4.0 ms
threadteam `416198e`, slow processes	7 / 20 (12.42 ms)	14.2 %	164 %/sample	102 samples, 21.4 ms	58 samples, 12.2 ms
`d3d9eca` (#10), pinning off	0 / 20	0.0 % (1 sample in 6607)	3 %/sample	40 samples, 7.8 ms	189 samples, 38.2 ms
`d3d9eca`, `pin_participants(true)`	0 / 20	0	0	–	–

Answer

Yes for (a), no for "long-lived". The slow mode tracks in-share stacking: its rate rises 5.7x
(2.5 % -> 14.2 % of samples) in the slow processes, and it is the one kind the fixed library no longer
produces — 1 sample in 6607 with pinning off, 0 with pinning on, where the old library produced it in
thousands.

What does not separate them is duration: the longest in-share stack is 19.5 ms in a fast process
against 21.4 ms in a slow one. The slow mode is many recurring short in-share stacks, not one long
residence — which is why the earlier "stacked sample %" correlated but never explained the 3x, and why
a fix that only reduces the chance of one long stack would not be enough.

And (b) alone is harmless: waiting-between-sections stacking is common in fast processes on both
libraries (40 % of samples on the old one, 3 % on the new one without pinning, with a 38 ms run in one
process) and never makes a process slow. It is what the two "stacked %" numbers in my earlier comments
were mostly counting, which also explains why that rate stayed non-zero after the fix.

Corrected summary for the issue: the mode is in-share CPU sharing on the team's members, recurring
through the call; the members waiting on one CPU between sections is normal scheduling noise.

deps: threadteam 0.1.3 (requirement and lock) for the #107 wait fix (#…

Fixed by threadteam 0.1.3 (tensor4all/threadteam-rs#10), which faer now requires and locks (#120, 7cfc56e). The classification run on broccoli (06:25Z) shows that the slow mode was team threads repeatedly stacking on one CPU while a share was executing, case (a). On threadteam 0.1.3 with pinning off, the default faer uses, (a) appears in 1 of 6607 samples. With pinning on it does not appear at all. The fix is the change to how members wait: PAUSE-only reserved spin, and a yield or park when a scope is oversubscribed. Participant pinning stays an opt-in extra (PoolBuilder::pin_participants, default off, leader never pinned).

Reopened: threadteam 0.1.3 still shows the mode in the configuration the harness uses

First, the version question: everything already links threadteam 0.1.3. faer-benchmark's own
Cargo.lock has no threadteam entry at all — the crate arrives only through the path-patched
t4a-faer, whose requirement is "0.1", so cargo resolves the newest matching release at build
time. Confirmed on the binaries themselves (strings … | grep threadteam): fbench (MKL arm and
OpenBLAS arm) and the standalone faer driver all print threadteam-0.1.3. So there is no lock to
bump and no PR needed for it.

Against those binaries, 20 processes each, 64T on CPUs 0-63, the usual guard (another user's
processes sit at 5-13 %, so the wide cells are recorded as over-guard with the reading):

cell	fast level	processes in the mode	what they look like
`gemm` f64 128, harness `fbench` (fork row, C)	0.019 ms	8 / 20	15.98 ms = 841x
`cholesky` f64 2048, standalone driver	~12 ms	~16 / 20	median 17-74 ms; samples `[724, 20.7, 20.0]`, `[126, 42.5, 72.0]`

So 0.1.3 does not remove it here. The interesting part is why the earlier verification said 0/20:
that run used team_mode_probe, which acquires one team and runs 20 repetitions inside it, with a
warm-up first. fbench — and every real caller — takes the team per call (with_team per row, 5
repetitions after a 250 ms prime). The fix apparently helps a team that is already warm and held, and
not a fresh acquisition per call. That distinction is the thing to measure next: the same probe with
with_team taken between repetitions, and a count of how often a fresh acquisition lands stacked.

cholesky 2048 is the worse cell of the two, and it is also where the mode costs the most (17-74 ms
against ~12 ms, i.e. up to 6x on a call that is 27x faster at 64T in the fast mode).

probe: #107 report the spare worker (state, run time, runnable/stacke…

The spare worker: what it runs, how long it spins, and a probe to measure it on broccoli

A pool of w workers gives a team of w participants: the caller and w - 1 reserved workers (threadteam-rs#12). So one worker of every pool is a spare, and a with_team(64) process runs 65 threads. Below is what that spare does, from the code (threadteam 0.1.3 4c73c2b, rayon-core 1.13.0), and the probe that will measure whether it matters here.

What the spare runs (code reading)

It is never a threadteam member. Awake::claim_cold spawns exactly missing = w - 1 claim jobs on the pool (src/awake.rs, pool.inner.spawn(move || claim_job(..))). The worker that gets none never enters member_loop, so threadteam's time-based idle spin (WaitPolicy::DEFAULT.idle_spin = 200 ms, PAUSE then yield_now per poll) does not apply to it. That spin applies only to a member whose reservation ended while it stays IDLE.
It idles as a plain Rayon worker. In wait_until_cold → Sleep::no_work_found (rayon-core src/sleep/mod.rs:88-108), every round is one find_work() (its own deque, steal attempts on the other workers' deques, the injector) followed by thread::yield_now(). After ROUNDS_UNTIL_SLEEPY = 32 rounds it announces itself sleepy. One more round later (ROUNDS_UNTIL_SLEEPING = 33) it blocks on its condvar. The spin before sleeping is therefore 33 rounds of steal plus yield, bounded by count, not by time.
When it wakes. A sleeping Rayon worker is woken only when jobs are injected or pushed (new_injected_jobs / new_jobs → wake_any_threads). In a team scope that happens at the acquisition, when the w - 1 claim jobs are spawned. Sections, fork and for_ranges on a team token dispatch on the reserved members and inject nothing. So:
- held pool (tenferro CpuEngine, team_per_call, the earlier team_mode_probe): the spare runs its 33 rounds at pool start and at the first acquisition, then sleeps for the pool's lifetime;
- fresh pool per call (with_team, fbench): every call spawns w new threads, and every one of them, the spare included, runs the 33 rounds right as the claim jobs arrive, i.e. while the team is being acquired and the call begins.
The worst case. When all CPUs are busy, a yield_now lets another runnable thread on that CPU run, and the spare runs again only when the scheduler gives the CPU back. On a 64-CPU set where 64 participants already spin, the spare is a 65th runnable thread for those 33 rounds, and the CPU it lands on has two runnable threads.

Indicative numbers (primerose, not a measurement)

primerose is saturated (load average 64-75 on 64 CPUs), so the timings below are noise. Only the spare's own CPU time is meaningful, from schedstat, at width 4 on taskset -c 0-5:

held pool, 20 calls: the spare ran 190-250 µs in its whole life, all of it before or in the first call; 0 µs in every later call; state S at the end of every call.
fresh pool per call: each new spare ran about 85-650 µs, partly inside the call (652 µs in the first call of one process).

So the spare's CPU cost is sub-millisecond per pool. Alone, it cannot account for a 16 ms gemm 128 or a 17-74 ms cholesky 2048. It can, however, make a CPU hold two runnable threads at the moment a fresh team starts. If such a stack then persists (when width == CPUs, every CPU already holds a spinning participant, so the scheduler gains nothing by moving one), the spare would be the trigger and the persistence the cost. That is the hypothesis the broccoli run should confirm or refute.

Probe (branch `probe/107-acquire-per-call`, `c1d8e4c`, not for merge)

faer/examples/acquire_probe_107.rs, <mode> <gemm128|chol2048> <width> <reps>. Modes:

mode	how the team is taken
`held`	one team and one scope for all calls (the earlier probe)
`team_per_call`	Pool and Team held, a token per call (= tenferro `CpuEngine`)
`exec_team_per_call`	Pool held, `Exec::team` per call
`into_team_per_call`	Pool held, `into_team` / `into_pool` per call
`fresh_pool`	a new pool per call, built as `with_team` builds it
`with_team`	`faer::team::with_team` itself

Options: PIN=1, PRIME_MS, SAMPLE_CPU (a /proc sampler outside the set), SPARE_WAIT=1. Per call, the probe reports:

the time and whether the participants are stacked (a census on the same team);
the spare's state at the end of the call;
the spare's CPU time during the call and since its start (schedstat);
the samples in which the spare was runnable, and in which it shared a CPU with another runnable thread;
with SPARE_WAIT, the time from pool creation until every spare sleeps.

The control for "does the +1 matter": on taskset -c 0-63, width 63 (63 workers including the spare, plus the caller = 64 threads on 64 CPUs) against width 64 (65 threads), in fresh_pool and team_per_call, 20 processes each, against the 11:55Z criteria.

Requested on broccoli (via the monitor):

the six modes without pinning;
fresh_pool and team_per_call with PIN=1;
the 63-against-64 control;
one run per mode with SAMPLE_CPU outside the set.

If the width-63 control removes the mode, the fix belongs in threadteam-rs#12: either a team that does not leave a spare, or a spare that never wakes. If it does not, the fix is about stacks that persist at width == CPUs, independently of the spare.

Acquisition modes at 64T: the trigger is a pool per call, not the team acquisition (and not the spare alone)

Probe acquire_probe_107 (branch probe/107-acquire-per-call, 8fd3b4c), 64T on CPUs 0-63,
PIN unset unless named, 20 processes per arm for gemm128 (50 calls each) and 10 for chol2048
(10 calls each), the usual whole-domain guard (another user's processes kept the wide cells busy, so
most carry over-guard and the readings are in the raw files). "slow" = a process whose median call
is >3x the fastest call seen in that arm; "stacked calls" = calls where two participants shared a CPU.

`gemm128` f64 at 64T, 50 calls per process

arm (per call)	processes	fast call	median of process medians	slow processes	stacked calls
`held` (one team for everything, the old probe)	20	0.017 ms	8.01 ms	10/20	54 %
`team_per_call` (pool reused, `team.exec(w).team(..)`)	20	0.016 ms	0.02 ms	6/20	38 %
`exec_team_per_call` (`Exec::new(...).team(..)`)	20	0.017 ms	0.02 ms	7/20	35 %
`into_team_per_call` (`into_team` … `into_pool`)	20	0.019 ms	0.03 ms	6/20	37 %
`fresh_pool` (new pool + team per call)	20	0.027 ms	15.99 ms	20/20	93 %
`with_team` (literally `faer::team::with_team`)	20	0.027 ms	15.99 ms	20/20	95 %
`fresh_pool`, `PIN=1`	20	0.021 ms	32.00 ms	16/20	71 %
`team_per_call`, `PIN=1`	20	0.015 ms	0.02 ms	6/20	29 %

`chol2048` f64 at 64T, 10 calls per process

arm	processes	fast call	median of medians	slow	stacked
`fresh_pool`	10	106.6 ms	361 ms	10/10	92 %
`fresh_pool`, `PIN=1`	10	14.2 ms	642 ms	9/10	63 %
`with_team`	10	15.6 ms	380 ms	10/10	99 %
`team_per_call`	10	15.0 ms	329 ms	10/10	76 %

At this size the mode also hits the pool-reusing arms — 10 calls are not enough for a process to
recover, and the fixed ~16 ms of the gemm128 case is a much smaller share of a 15-300 ms call.

Placement and the "one free CPU" control (`gemm128`, one process per arm)

configuration	`held`	`team_per_call`	`fresh_pool`	`with_team`
width 63 on CPUs 0-62, sampler on 63	15.98 ms	0.02 ms	16.00 ms	16.00 ms
width 48 on CPUs 0-63 (16 CPUs of slack)	–	0.02 ms, 0 % stacked	0.04 ms, 2 % stacked	–

The +1 spare worker

The pool has width workers and the team is the caller plus width-1 members, so every configuration
here runs 65 threads on 64 CPUs, and my external sampler confirms it: max_R_threads = 65 in
every arm at 64T (held, team_per_call, fresh_pool, with_team, both kernels), 64 only in a
couple of exec_team_per_call runs. So the spare is runnable alongside the 64 participants everywhere
— and it is therefore not what separates the arms: team_per_call has the same 65 runnable threads
and stays at 0.02 ms, while fresh_pool/with_team sit at a fixed 16 ms on 20 of 20 processes.

What does separate them is constructing the pool in the call. Two further data points agree:
pinning the fresh pool makes it worse (median 32 ms, and 642 ms at chol2048, against 16/361
unpinned), and with 16 CPUs of slack (width 48 on 64 CPUs, where the spare and any parked-to-spinning
worker are harmless) the mode disappears entirely.

One gap I am not papering over: the per-spare state and CPU-time columns I wanted (a), and the spin
time (c), are not usable from my outside sampler — it loses the pool's threads when the process tears
them down (threads_at_end = 1 for the chol2048 runs). Getting them needs a flag inside the probe
that prints the spare's own state/utime, or snapshots taken while the pool is alive. The 65-runnable
count above is the part that survived that check.

threadteam-rs#14 re-check on the #107 arms: the width-64 collapse is gone in threadteam's own bench

With threadteam patched to PR #14 head (904501e), the acquire scenario of the new
benches/wait_scenarios.rs reports, for the fresh_pool case (what faer::team::with_team does):

load	width	main `4c73c2b`	PR `904501e`
none	64	131.7 ms per call	6.15 ms
8x10 % foreign load	64	622.8 ms per call	6.95 ms
none	48	1.41 ms	6.28 ms
8x10 %	48	2.29 ms	6.35 ms

So the 107 mode — a fixed ~16 ms per freshly created pool that the arm fresh_pool/with_team showed
20/20 at 64T — is what this PR removes, and it removes it under load too.

Two costs in the same table, for the record: held_team/held_pool at width 64 go 1.48 -> 6.12 ms
(4.1x slower, no change at width 48), and a call after a 10 ms gap goes 17 µs -> 100-500 µs in the
first metric, at both widths.

Caveat on the numbers: another user's two Julia jobs ran during part of the window (load 25 in the
5-minute sample), and libgomp's own row shows the contamination (161 ms in the main pass against
5.2 ms in the PR pass). The acquire_probe_107 matrix I started for the same comparison was stopped
for that reason and will be re-run on a quiet host; the direction above is consistent with the
fresh_pool/with_team result from the earlier run, but treat the absolute values as provisional.

team route: some processes run every repetition at the 1-thread time (16T, broccoli) #107

Description

Observation (broccoli, faer#99 re-measurement, 2026-10-29T02:26Z)

What is ruled out by the code

Probe and what it shows so far (primerose, width 4, not 16)

Request: run the probe on broccoli at 16T (via the monitor)

Next

Activity

shinaoka commented on Oct 29, 2026

How OpenMP runtimes and OpenBLAS wake members and wait, compared with threadteam

Waking members, barriers, reuse, affinity

How spinning is kept cheap

What threadteam does differently

Recommendation

Not verified

shinaoka commented on Oct 29, 2026

shinaoka commented on Oct 29, 2026

20 processes of team_mode_probe 16 512 20 on broccoli: 10 fast, 10 slow, and what separates them

What the differences say

shinaoka commented on Oct 29, 2026

Probe update for the next broccoli run (fix/team-serial-mode at df7e803)

One correction to the 17-on-16 reading

What the probe now records (both options are environment variables)

Suggested broccoli runs

How to read the result

shinaoka commented on Oct 29, 2026

A/B/C on broccoli, 20 processes each, arms interleaved (probe at df7e803, sampler on CPU 8)

Reading

shinaoka commented on Oct 29, 2026

Control: OpenBLAS dgeqrf 512 at 16T, 20 processes — the per-process mode does not reproduce

shinaoka commented on Oct 29, 2026

threadteam-rs#10 re-checked against #107's mode (20 processes per arm, 16T, CPUs 48-63)

shinaoka commented on Oct 29, 2026

OpenMP-default control: it does not stack either, and the settings it runs under

The settings it runs under, and the difference to threadteam

shinaoka commented on Oct 29, 2026

Is the slow mode a long-lived in-share stack? Split the stacked samples and measure how long they last

Answer

shinaoka commented on Oct 29, 2026

shinaoka commented on Oct 29, 2026

Reopened: threadteam 0.1.3 still shows the mode in the configuration the harness uses

shinaoka commented on Oct 29, 2026

The spare worker: what it runs, how long it spins, and a probe to measure it on broccoli

What the spare runs (code reading)

Indicative numbers (primerose, not a measurement)

Probe (branch probe/107-acquire-per-call, c1d8e4c, not for merge)

shinaoka commented on Oct 29, 2026

Acquisition modes at 64T: the trigger is a pool per call, not the team acquisition (and not the spare alone)

gemm128 f64 at 64T, 50 calls per process

chol2048 f64 at 64T, 10 calls per process

Placement and the "one free CPU" control (gemm128, one process per arm)

The +1 spare worker

shinaoka commented on Oct 29, 2026

threadteam-rs#14 re-check on the #107 arms: the width-64 collapse is gone in threadteam's own bench

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

20 processes of `team_mode_probe 16 512 20` on broccoli: 10 fast, 10 slow, and what separates them

Probe update for the next broccoli run (`fix/team-serial-mode` at `df7e803`)

A/B/C on broccoli, 20 processes each, arms interleaved (probe at `df7e803`, sampler on CPU 8)

Control: OpenBLAS `dgeqrf` 512 at 16T, 20 processes — the per-process mode does not reproduce

Probe (branch `probe/107-acquire-per-call`, `c1d8e4c`, not for merge)

`gemm128` f64 at 64T, 50 calls per process

`chol2048` f64 at 64T, 10 calls per process

Placement and the "one free CPU" control (`gemm128`, one process per arm)