Execution pool recruits every core regardless of work size: 2.7x the CPU for 1.2x the speed

[vbmithr]

Summary

The execution pool splits a query across every available worker regardless of
how much work there is. Past a handful of workers the coordination cost grows
faster than the saving, so total CPU rises steeply while wall time barely
improves. A service that is not latency-bound pays several times the CPU it
needs and has no way to ask for the cheaper trade.

Reproducer

200 inner-joins of a 3-row key set against a 130,000-row table, the shape a
market-data service runs continuously:

(set N 130000)
(set syms (map (fn [i] (parse (format "INST%" i))) (til 773)))
(set t (table [venue instrument session message_id px]
         (list (take ['BFU] N) (take syms N) (take ['S1] N) (til N) (til N))))
(set ks (select {from: t by: [venue instrument] n: (count px)}))
(set k3 (select {from: ks take: 3}))
(set n 0)
(times 200 (set n (+ n (count (inner-join [venue instrument] t k3)))))

/usr/bin/time on a 16-core machine, 2.6.2.r229.ge7770412:

cores	user CPU	wall	speedup	CPU vs -c 1
-c 1	0.71 s	0.71 s	1.0x	1.00x
-c 2	0.71 s	0.36 s	2.0x	1.00x
-c 4	0.80 s	0.22 s	3.2x	1.13x
-c 16	1.93 s	0.18 s	3.9x	2.72x

-c 2 is free: half the wall time for the same CPU. -c 4 costs 13% for a
3.2x speedup. -c 16 costs 172% more CPU to go 1.2x faster than -c 4.

Small queries are unaffected — 20,000 joins over a 10-row table cost 0.05 s at
every core count, so nothing is dispatched for those. The effect starts where
the pool starts splitting.

Impact in production

A depth-of-book service of ours, default pool on a 16-core host, against an
identical instance on the same live feed with -c 2, in lock-step on every
state counter throughout:

default (16c)   50%  13%  23%  44%  57%  64%
-c 2            25%  10%  15%  23%  27%  31%

Half the CPU, identical throughput, identical progress. A second run comparing
-c 1 with -c 4 was monotonic in the same direction. The service's own batch
takes 2.5-5.7 ms against a ~26 ms inter-batch gap, so the latency the pool buys
is worth nothing to it — and .sys.querylog reports effective parallelism of
~0.0002 for a typical batch, while perf put 73% of samples in worker_loop.

Notably our other Rayforce services on the same host and the same feed sit at
1-2% CPU: they do not run joins large enough to dispatch, so they never meet
this. It is specific to whoever does the table-sized work.

What we are asking for

1. Scale the worker count to the work, not to the machine — a morsel-count
   or row-count heuristic, so a 130k-row join does not recruit 16 threads for a
   1.2x gain over 4. The measurements above suggest the knee is low.
2. Failing that, a throughput mode — a way to say "minimise CPU, I am not
   latency-bound". -c is that knob today, but it is a blunt global: it also
   caps the genuinely parallel work a process may want elsewhere.

We have set -c 2 and consider the immediate problem solved for us; this is
filed because the default costs every non-latency-bound service the same
multiple, silently.

---

Measured on rayforce 2.6.2.r229.ge7770412, 16-core host.

[singaraiona]

Reproduced, and the cause is not what either of us assumed. Thank you for the
CPU-vs-wall table — it is the measurement that made this findable.

Reproduced

Your reproducer on an i7-6700 (4 physical cores, 8 logical), release build:

cores	user CPU	wall	speedup	CPU vs -c 1
-c 1	2.38 s	2.38 s	1.0x	1.00x
-c 2	2.52 s	1.27 s	1.9x	1.06x
-c 4	2.59 s	0.67 s	3.6x	1.09x
-c 8	5.17 s	0.70 s	3.4x	2.17x

Same shape as yours: past a point the CPU climbs steeply and the wall stops
improving. Note where that point is on this machine — -c 8 is where the pool
starts using hyperthread siblings rather than more cores.

It is SMT oversubscription, not coordination overhead

The pool defaults to every online logical CPU. src/core/pool.c calls
ray_thread_count() (sysconf(_SC_NPROCESSORS_ONLN)), commented "Default to
every online logical CPU". Meanwhile the tree already contains
ray_physical_core_count(), whose own comment says why that is wrong — two SMT
siblings share one core's load/store machinery, so the second thread adds
contention rather than throughput. It was being used only to estimate a cache
size, never to size the pool.

Changing that one call, nothing else:

workload (default pool)		user CPU	wall
your 130k-row probe ×200	before	5.19 s	0.70 s
	physical cores	2.58 s (−50%)	0.66 s
50k-row probe ×200	before	2.07 s	0.36 s
	physical cores	1.16 s (−44%)	0.35 s

Half the CPU, and the wall time does not regress. That is the bulk of what you
measured, and on your 16-logical host it is very likely the same story.

What I can tell you about the trade

I have not shipped that default change, because it is not free everywhere. On
the full ClickBench suite (10M rows, 43 queries × 3) on this box:

• sum of per-query hot times: 5690.7 ms → 5845.6 ms (+2.7%), 12 queries
  regressing >10%, worst +36%
• whole-process CPU 55.1 s → 36.4 s (−34%), wall 23.3 s → 18.6 s (−20%)

So it halves CPU and improves end-to-end time while costing a few percent on the
hot per-query numbers that get published. This box is also the least favourable
case for it — 4 physical / 8 logical means giving up half the threads — and the
in-tree note records ClickBench running 11% faster on physical cores on a
5950X (16 physical / 32 logical). Your host is closer to that. It needs a
decision against a multi-machine benchmark run, so it is filed separately rather
than slipped into a patch release.

What this means for you

Your -c 2 is sound, and -c <physical core count> is probably better —
here -c 4 gave 3.6x the throughput of -c 1 for 9% more CPU, while -c 8
bought nothing for another 1.9x. On a 16-logical host, try -c 8 (or whatever
lscpu reports as Core(s) per socket × Socket(s)): you may get most of your
latency back at close to the CPU you have now.

Two corrections to the framing in the issue, for the record:

1. Coordination overhead is not the dominant cost. At 50k rows and -c 4,
   the parallel split costs 24% more CPU for 2.6x the wall — a good trade. I
   tried the sizing heuristic you asked for (request 1), gating the join probe on
   the engine's own RAY_PARALLEL_THRESHOLD, and it made things worse: 2 tasks
   instead of 16 regressed both CPU and wall at -c 4 (2.59 s / 0.67 s →
   3.11 s / 1.58 s). The cost curve is U-shaped, because the main thread's
   completion spin burns a full core waiting on a straggler, so under-splitting is
   expensive too. I have dropped that approach.

2. The join probe does still lack a work-size gate — it dispatches from 8193
   rows up, 8x under the threshold 16 other dispatch sites respect. Real, but
   closing it costs ~2.6x wall on sub-threshold joins for ~44% CPU. That is your
   preferred trade, not a safe default, which is what request 2 was really
   asking for.

On request 2, a throughput mode: I would rather not add a knob for it. If the
pool sizes itself correctly there is much less left for the knob to buy, and -c
already covers the blunt case. Happy to be argued out of that if physical-core
sizing still leaves you paying for latency you do not want.

One small thing landed out of this: #605 makes a dispatch wake only the workers
it can keep busy, instead of the whole pool. It does not help your shape
(your probe makes 16 tasks against 8 workers, so the clamp does not bind —
measured 5.19 s → 5.18 s), but it removes real waste for dispatches narrower
than the machine.

Leaving this open and tracking the default in the new issue.

[vbmithr]

Your diagnosis holds on a 16-logical host, and I have a measurement that removes
a confound both of our tables contain. Short version: ship the physical-core
default. Also, the framing in my issue body was wrong and I withdraw it.

The confound: user CPU is not a safe metric here

My original table and yours both report user CPU. That is thread-seconds, and on
any part that drops clocks as more cores go active, identical work inflates.
I re-ran your sweep on an 8-physical / 16-logical AMD Ryzen AI 7 350 and user CPU
rose smoothly with no knee at 8 — which appears to contradict the SMT
explanation:

-c 1  0.73   -c 2  0.76   -c 4  0.84   -c 6  0.96
-c 8  1.12   -c 12 1.53   -c 16 1.84        (user CPU, mean of 3)

Measuring instructions and cycles instead — both frequency-independent — tells a
different and much sharper story:

	instructions	cycles	IPC	wall
-c 1	21.49 G	3.30 G	6.5	0.72
-c 2	21.50 G	3.33 G	6.4	0.38
-c 4	21.52 G	3.42 G	6.3	0.22
-c 8	21.54 G	4.04 G	5.3	0.16
-c 12	21.57 G	5.71 G	3.8	0.16
-c 16	21.63 G	6.87 G	3.2	0.16

Cycle counts are means of 3; the -c 8 and -c 16 figures varied by under 4%.

Two things fall out of that

1. Instructions are flat: +0.7% from 1 to 16 cores, while cycles double.

The extra CPU is not extra work. It is stall cycles — IPC collapses from 6.5 to
3.2. So the framing in my issue body is wrong and I withdraw it: I wrote
"the coordination cost grows faster than the saving", which implies work being
added. It is not. This is direct evidence for your explanation instead — the
main thread's completion spin burning a core waiting on a straggler — and it is
consistent with your finding that under-splitting made things worse.

2. The split lands almost entirely on the SMT step, as you argued.

-c 1  ->  -c 8    (real cores)      cycles 3.30G -> 4.04G   1.21x   wall 0.72 -> 0.16
-c 8  ->  -c 16   (SMT siblings)    cycles 4.04G -> 6.87G   1.70x   wall 0.16 -> 0.16

1.70x the cycles for exactly zero wall improvement. Your hypothesis, on the
topology you said you lacked, with the throttling confound removed. The residual
sub-SMT overhead is only 1.21x across a 4.5x wall improvement — that is a good
trade and I do not think it is worth chasing.

Worth noting this box is heterogeneous (Zen5 + Zen5c, different clocks), so
straggler variance should be systematically worse here than on either of your
homogeneous test machines. It still shows the effect concentrated at the SMT
boundary rather than smeared across core counts, which I think strengthens the
case rather than weakening it.

On the ClickBench trade

Your 4-physical/8-logical box gives up half its threads and is the least
favourable case; the in-tree note has a 5950X running 11% faster on physical
cores. Ours sits with the 5950X: 41% of cycles recovered for no wall regression
at all. For whatever it is worth as a third data point, I would take a 2.7%
regression on published hot per-query numbers over a 1.7x cycle cost that buys
nothing, and I would ship #606.

Your -c 8 suggestion: right in general, wrong for us

It is well supported by the table above — -c 8 gets all the available wall
benefit (0.16 s, identical to -c 16) at 59% of the cycles. Anyone
latency-sensitive on an SMT host should do this today.

We will stay at -c 2, because the service is not latency-bound: its batch takes
2.5-5.7 ms against a ~26 ms inter-batch gap, so 0.38 s versus 0.16 s buys nothing
we can use, and -c 2 costs 3.33 G cycles against 4.04 G.

On request 2, where I would push back gently

I accept that a "throughput mode" is the wrong shape, and I am not asking for a
mode. But I do not think -c covers the blunt case, for a reason your own
correction 2 identifies.

-c conflates two separate things: how much parallelism a process may use,
and how eagerly work is split. We want the pool to stay available for a rare
whole-table operation (an invalidation over every book) while never splitting the
steady small joins. -c cannot express that — which is exactly why our unit file
carries -c 2 rather than -c 1, with a comment explaining we are keeping one
spare worker for an event that happens a few times a day.

You measured the gate already: the join probe dispatches from 8,193 rows, 8x
under the threshold 16 other sites respect, and closing it costs ~2.6x wall for
~44% CPU — "your preferred trade, not a safe default". I agree it is not a safe
default. That is an argument for making the dispatch floor settable, not for
dropping it: one tunable that decouples split-eagerness from pool size, left at
today's value by default. That is a smaller surface than a mode and it is the
thing -c genuinely cannot do.

One caveat on my own issue body

The production comparison I cited (50% -> 25% across state counters) was measured
on an older revision of that service. It has since been rewritten to avoid the
joins that were doing the dispatching, and on #596 I have just measured that the
join in question is no longer executed in its steady state at all. So please do
not weight that particular table heavily — the reproducer above and the cycle
counts are the part I stand behind.