perf(core): reuse persistent worker subprocesses across translation units by hedgar2017 · Pull Request #524 · NomicFoundation/solx

hedgar2017 · 2026-07-09T11:48:57Z

Replaces the fork/exec-per-translation-unit subprocess model with a pool of persistent worker subprocesses fed over pipes, and slims the per-unit IPC payload. Previously every translation unit fork/exec'd the 55 MB LLVM-linked binary, CBOR-encoded the whole payload, and each child built an unused N-thread rayon pool.

Local results with the raw openzeppelin standard JSON (average of 5 runs):

Case	Before (`solx-old`)	After (`solx-new`)	Δ
dwarf	20.94s	19.55s	−6.6%
no-dwarf	20.02s	18.87s	−5.7%

Copilot

Pull request overview

This PR refactors solx-core’s recursive subprocess compilation into a pool of persistent worker subprocesses using a framed CBOR protocol, aiming to eliminate per-translation-unit fork/exec and reduce IPC overhead while keeping outputs byte-identical.

Changes:

Introduces a framed CBOR channel and splits compilation payload into Session (sent once) + per-unit Job.
Adds a parent-side Pool/Worker for persistent subprocess reuse and a child-side long-lived worker loop.
Updates compilation pipeline call sites and LLVM option handling to avoid cross-job option leakage.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
solx/tests/cli/recursive_process.rs	Updates CLI expectations for the new worker protocol/argument behavior.
solx-core/src/project/mod.rs	Switches multi-pass compilation to dispatch jobs through the persistent worker pool.
solx-core/src/project/contract/mod.rs	Adjusts `compile_to_evm` API to take `output_selection` by reference; removes per-unit stack handler install.
solx-core/src/process/mod.rs	Replaces single-shot subprocess model with new process module structure for persistent workers.
solx-core/src/process/channel.rs	Adds framed CBOR send/recv helpers shared by parent/child.
solx-core/src/process/session.rs	Defines session payload shared across jobs and sent once per worker.
solx-core/src/process/job.rs	Defines per-translation-unit job payload.
solx-core/src/process/pool.rs	Adds worker pooling, reuse policy, and retry-on-reused-worker-death behavior.
solx-core/src/process/worker.rs	Implements parent-side subprocess spawn + framed I/O.
solx-core/src/process/child.rs	Implements child-side long-lived worker loop and stack-too-deep handler response path.
solx-core/src/process/output.rs	Minor documentation cleanup for the output type.
solx-core/src/lib.rs	Updates exports to expose new pool/session/job and subprocess entrypoint.
solx-core/src/compiler.rs	Reorders initialization to avoid building rayon pool in worker subprocess mode.
solx-core/src/arguments.rs	Updates `--recursive-process` semantics/docs and argument validation expectations.
solx-codegen-evm/src/target_machine.rs	Ensures `-evm-metadata-size` is always passed to heal stale process-global LLVM options.
solx-codegen-evm/src/codegen/context/mod.rs	Adjusts size-fallback flag handling and propagates metadata sizing into fallback settings.

github-actions · 2026-07-13T16:21:44Z

📊 solx Tester Report

➡️ Download

github-actions · 2026-07-13T16:39:11Z

📊 Hardhat Projects Report

➡️ Download

github-actions · 2026-07-13T18:30:47Z

Compile-time benchmark (`--standard-json`, EVMLA pipeline)

pr: solx v0.1.4, LLVM-based Solidity compiler for the EVM, Front end: solc, LLVM build: bc069547bc52f9fc49c44592bdf0082e26533219
main: solx v0.1.4, LLVM-based Solidity compiler for the EVM, Front end: solc, LLVM build: 11a67cb325d04a38ea459c9f04f82a33b3bc59a4
release: solx v0.1.4, LLVM-based Solidity compiler for the EVM, Front end: solc, LLVM build: aaa40607301776680f66f4119b5f564a8416cca0

ens-verifiable-factory-solx--solx-legacy-dwarf

binary	mean ± σ	min … max	vs baseline
pr	2.983 s ± 0.020 s	2.961 s … 3.000 s	0.994×
main	3.002 s ± 0.016 s	2.985 s … 3.017 s	1.000×
release	6.353 s ± 0.092 s	6.291 s … 6.459 s	2.116×

openzeppelin-contracts-0.34--solx-legacy-dwarf

binary	mean ± σ	min … max	vs baseline
pr	32.754 s ± 0.097 s	32.645 s … 32.829 s	0.956×
main	34.258 s ± 0.058 s	34.198 s … 34.313 s	1.000×
release	171.965 s ± 18.648 s	159.021 s … 193.340 s	5.020×

uniswap-v4-core-solx--solx-legacy-dwarf

binary	mean ± σ	min … max	vs baseline
pr	17.161 s ± 0.110 s	17.097 s … 17.288 s	0.996×
main	17.223 s ± 0.027 s	17.193 s … 17.247 s	1.000×
release	48.218 s ± 0.811 s	47.574 s … 49.129 s	2.800×

github-actions · 2026-07-13T23:07:30Z

📊 Foundry Projects Report

➡️ Download

nebasuke · 2026-07-14T09:35:04Z

Not my wheelhouse, so feel free to discard if irrelevant, but here's a drive-by Claude comment.

Claude comment

Non-blocking, follow-up material: Pool::execute drops the worker on any Err, including ordinary per-contract compile errors. In that case the child is actually at a clean frame boundary (the worker loop sends the Err frame and keeps looping), so we discard a healthy worker and every failing contract re-spawns the 55 MB binary. A cheap refinement would be to return the worker to the pool on job-level Err results and only drop it on channel/protocol errors — with the caveat that StackTooDeep must keep dropping, since that child exits after replying.

Related, minor: a crashed worker now surfaces as "The worker closed the response channel" without the exit status the old wait_with_output path reported (stderr still reaches the terminal via Stdio::inherit). Calling wait() on that path and including the exit code in the error would recover the old diagnosability.

hedgar2017 · 2026-07-14T14:56:13Z

Not my wheelhouse, so feel free to discard if irrelevant, but here's a drive-by Claude comment.

Thanks, both sound reasonable, fixed!

…xit status Return a worker to the idle pool after a per-unit compile error (Error::StandardJson) instead of dropping it on every Err, retiring it only on a transport failure (Error::Generic) or a StackTooDeep reply, after which the child exits. Reap a crashed worker on the closed-channel path and fold its exit status into the error, restoring the diagnosability of the former wait_with_output path. Add From<std::io::Error> for Error so the wait result converts cleanly. Addresses review feedback on #524.

Copilot

Pull request overview

Copilot reviewed 21 out of 22 changed files in this pull request and generated 1 comment.

The full suite on #524 passed macos-15 in 10m32s (run 29374444399), faster than main's 12-15.5 min baseline on that runner. Switch to the runner where PR #545 observed the 4h26m silent hang, keeping everything else (RelWithDebInfo+assertions LLVM, direct tester invocation) fixed.

The current tip 1ae05b1 is green and faster than main on both arm64 runners (runs 29374444399, 29399344551), but PR #545's 4h26m xlarge hang was observed at 6bd1976 — before b8469e0 (worker retention) and the 1ae05b1 solx-llvm reset-option-occurrences pin. Re-point the harness at the old tip: a watchdog trip here confirms the hang belonged to that intermediate state and is fixed since. Also adds an Intel path for later: solx-tester rebuilt with nightly to dodge the 1.96.1 enum-niche miscompile (#547), solx kept on the pinned stable, watchdog raised to 75 min for the slow Intel minis. free-disk-space-macos is carried over from main (predates #533 here).

Both arm64 legs are green and faster than main at 1ae05b1, and the old-tip hang is confirmed fixed (run 29401155956). The one untested platform is macOS x86-64, where the pinned 1.96.1 miscompiles the tester on any code (#547): rebuild just solx-tester with nightly (fixed LLVM), keep solx on the pinned stable, and run the suite under the 75-minute Intel watchdog.

nebasuke · 2026-07-15T11:50:08Z

macOS validation results

Following up on the concern that persistent-worker changes can regress badly on macOS while looking fine on Linux: I ran this PR's tester suite on macOS runners via a debug harness (#546 — direct solx-tester invocation, symbols kept, hang watchdog with sample, crash reports uploaded). Summary: the current tip is green on macOS arm64 and consistently faster than main; on x86-64 it is functional with no sign of the hang (a toolchain bug unrelated to this PR required a workaround, and the runner is too slow to complete the suite); the hang we initially observed was real, but belonged to the intermediate state 6bd1976a and is fixed by the two commits that followed.

Code	Runner	Result
main (baseline)	macos-15	✅ 15m33s (run)
main (baseline)	macos-15-xlarge	✅ 5m31s (same run)
#524 @ `1ae05b1` (current)	macos-15	✅ 10m32s (run)
#524 @ `1ae05b1` (current)	macos-15-xlarge	✅ 4m31s (run)
#524 @ `1ae05b1` (current)	macos-15-large (x86-64)	✅ functional — no crash, no hang; suite too slow for the runner to finish (run, details below)
#524 @ `6bd1976` (old tip)	macos-15-xlarge	🐌 4h26m silent hang, job timeout (#545 run)
#524 @ `6bd1976` (old tip)	macos-15	🐌 hang reproduced, watchdog kill at 30 min (run, samples in the run artifact)

Anatomy of the (already-fixed) hang

sample stacks from the old-tip reproduction:

every solx-tester thread: Child::wait_with_output → poll, waiting on top-level solx processes that never exit;
every sampled worker subprocess (--recursive-process): idle in FrameRead::recv → read(stdin), waiting for a job frame that never arrives.

So the wedge is inside the top-level solx processes, between receiving translation units and dispatching to their (ready) workers. 6bd1976a resets LLVM cl-options per unit but predates the solx-llvm reset-option-occurrences pin (1ae05b1d), which is consistent with dispatch threads deadlocking in LLVM's option machinery — and with the hang disappearing exactly when b8469e03 + the LLVM pin landed. It reproduces on the 3-core standard runner, so it wasn't an xlarge/VM artifact. I didn't capture the parents' own stacks (harness sampled only tester + workers), so the LLVM-side mechanism is inferred rather than observed; happy to grab parent stacks with one more free-runner run if you want it pinned down exactly.

macOS x86-64

Getting an Intel datapoint required a workaround: solx-tester itself is killed on any Intel mac — including on main — by an unrelated rustc 1.95/1.96 enum-niche miscompilation (#547; fix is the upcoming 1.97.1 toolchain bump). So the harness rebuilt just the tester with nightly rustc while keeping solx on the pinned stable. With that:

smoke test green; the full suite ran ~78 minutes without a crash or a wedge before the watchdog killed it — at kill time the tester was mid-posix_spawn launching new solx invocations, with per-invocation lifetimes so short that pgrep caught zero solx processes in flight. That's the opposite signature of the old-tip hang (frozen solx parents, starved workers).
the suite is simply too slow to complete on the 2012-era Intel minis GitHub provides (the 3-core arm runner needs 10–15 min; this 12-core box couldn't finish even the Ethereum corpus subset in 23 min on main). No wedge, but also no completed run or main-vs-PR timing comparison is attainable on this hardware — a limitation of the runner, not of this PR.

Other notes

Linux was green at both tips all along (this PR's own integration runs), so the old hang was genuinely macOS-specific — the platform concern was justified, just already addressed by the later commits.

#545 served its purpose as the first mac experiment and is superseded by the #546 harness; closing it.

…nits Replace the fork/exec-per-unit model with a pool of persistent workers fed over pipes. The per-unit CBOR payload is split into a Session (project-wide data, sent once per worker) and a slim per-unit Job, and idle workers are held in a std Mutex<Vec> and reused. The process module is one entity per file: channel (the length-prefixed CBOR frame codec, via FrameRead/FrameWrite extension traits), session, job, output, pool (parent-side scheduler), worker (subprocess handle), and child (the subprocess loop). The child runs one long-lived stack-sized thread that owns the deserialized Session and compiles jobs until stdin closes, instead of spawning a thread and re-cloning the session per job. Worker stderr is inherited, so subprocess diagnostics stream straight to the parent. Because LLVM command-line options are process-global and now survive across units in one worker, always emit -evm-metadata-size to heal a stale value, and carry metadata_size into the size-fallback settings; reset IS_SIZE_FALLBACK per job. A worker is retired when its job set a spill-area size (occurrence- gated cl-option) or carried extra llvm_options, and never reused after a stack-too-deep response since its handler has already exited the process. When a reused worker dies mid-job, the job is retried once on a fresh worker so the death is not mis-attributed to the contract. Install the stack-error handler once per worker process instead of per unit, and build the rayon pool only in the parent, after the recursive-process branch, since workers compile a single unit and never use it.

Worker::execute now returns a plain crate::Result<EVMOutput> — a subprocess I/O failure folds into Error::Generic — so the ? drops a dead or errored worker and the pool is just pop-or-spawn, run, and return. This removes the retry, the WorkerError enum, dispatch, and worker_failed. Whether a worker may be reused lives on the data owners: Session::allows_worker_reuse (no extra llvm_options) and Job::allows_worker_reuse (no stack-too-deep spill area), each documenting the process-global cl-option it guards against. The cached reuse_workers field is removed. run_multi_pass_pipeline breaks the loop with its value instead of a declare-then-assign, and no longer threads the contract name.

Reset the process-global LLVM command-line option occurrences before every parse (new LLVMResetAllOptionOccurrences, exposed through llvm-sys and inkwell), so a translation unit never inherits an option a previous one set in the same persistent worker. This removes the whole option-leak layer: the -evm-metadata-size=0 heal is gone, and the pool reuses a worker unconditionally on success — Session/Job::allows_worker_reuse and the spill/llvm_options gating are deleted. Points solx-llvm, llvm-sys, and inkwell at their az-reset-option-occurrences branches. Byte-identical on the EVMLA smoke; CLI suite 397/397.

…xit status Return a worker to the idle pool after a per-unit compile error (Error::StandardJson) instead of dropping it on every Err, retiring it only on a transport failure (Error::Generic) or a StackTooDeep reply, after which the child exits. Reap a crashed worker on the closed-channel path and fold its exit status into the error, restoring the diagnosability of the former wait_with_output path. Add From<std::io::Error> for Error so the wait result converts cleanly. Addresses review feedback on #524.

Bump the solx-llvm submodule to main (bc069547), which merged the LLVMResetAllOptionOccurrences C API, and repoint the inkwell dependency off its feature branch onto the merged rev (c2ac839f); the transitive llvm-sys resolves to its merged rev (1aef1b28). Replaces the orphaned submodule pin (9537dbed) and the branch-tracked inkwell dep. Verified: solx links and compiles a multi-contract unit.

github-actions · 2026-07-15T13:47:04Z

Coverage Summary

Crate	Line Coverage	Function Coverage
solx	🟢 94.7%	🟡 68.0%
solx-benchmark-converter	🔴 0.0%	🔴 0.0%
solx-codegen-evm	🟢 88.9%	🟢 83.4%
solx-compiler-downloader	🔴 0.0%	🔴 0.0%
solx-core	🟢 92.8%	🟢 90.9%
solx-dev	🔴 2.3%	🔴 3.0%
solx-evm-assembly	🟡 70.6%	🟡 66.2%
solx-mlir	🟡 69.3%	🟡 61.5%
solx-slang	🔴 36.0%	🔴 43.4%
solx-solc-test-adapter	🔴 1.7%	🔴 2.1%
solx-standard-json	🟢 94.9%	🟢 93.4%
solx-tester	🔴 37.8%	🔴 35.7%
solx-utils	🟢 87.2%	🟢 82.9%
solx-yul	🟡 78.9%	🟡 71.3%
Total	🔴 25.8%	🔴 27.3%

Codecov Report | HTML Report | Workflow Run

nebasuke

Looks good from my end.

…lation units (#524)" This reverts commit ebc9059.

solx-tester has no macOS CI coverage anywhere; on PR #545 it was signal-killed within seconds on macos-15-large and ran for hours on macos-15-xlarge, but with PR #524 included, so mac breakage and #524 regressions were indistinguishable. This branch is plain main, so its results are the macOS baseline. The workflow mirrors test.yaml's build-llvm/build-solc parameters to hit the finished-install caches saved on main, invokes solx-tester directly so a signal death is named instead of collapsing to "subprocess failed without exit code", and samples hung processes before killing them. Crash reports and samples are uploaded as artifacts.

hedgar2017 force-pushed the az-evm-assembly-worker-pool branch 4 times, most recently from f9371c4 to 2f00823 Compare July 9, 2026 16:19

hedgar2017 commented Jul 12, 2026

View reviewed changes

hedgar2017 requested a review from Copilot July 12, 2026 09:17

hedgar2017 self-assigned this Jul 12, 2026

Copilot started reviewing on behalf of hedgar2017 July 12, 2026 09:18 View session

Copilot AI reviewed Jul 12, 2026

View reviewed changes

Comment thread solx-core/src/process/channel.rs

Comment thread solx-core/src/process/channel.rs Outdated

nebasuke added the ci:compile-benchmark Run hyperfine compilation benchmark for this PR label Jul 13, 2026

hedgar2017 force-pushed the az-evm-assembly-worker-pool branch from 2f00823 to 8276ba7 Compare July 13, 2026 14:35

hedgar2017 added ci:integration Trigger integration tests workflow on PR ci:slang Trigger slang unit tests on PR ci:integration-benchmark-full Run the full benchmark matrix (all compilers + comparisons) in integration tests labels Jul 13, 2026

hedgar2017 marked this pull request as ready for review July 13, 2026 14:45

hedgar2017 requested review from a team and nebasuke July 13, 2026 17:32

nebasuke requested changes Jul 14, 2026

View reviewed changes

Comment thread Cargo.toml Outdated

Comment thread solx-llvm

hedgar2017 requested a review from nebasuke July 14, 2026 16:35

hedgar2017 requested a review from Copilot July 14, 2026 16:36

Copilot started reviewing on behalf of hedgar2017 July 14, 2026 16:36 View session

hedgar2017 added the ci:coverage label Jul 14, 2026

hedgar2017 added the ci:sanitizer Trigger sanitizer workflow on PR label Jul 14, 2026

Copilot AI reviewed Jul 14, 2026

View reviewed changes

Comment thread solx-core/src/process/channel.rs

This was referenced Jul 14, 2026

ci: debug solx-tester on macOS runners (main baseline for #524 mac validation) #546

Draft

solx-tester SIGSEGV on macOS x86-64: rustc 1.95/1.96 enum-niche miscompilation (rust-lang/rust#159035) — fix by bumping to 1.97.1 #547

Closed

hedgar2017 added 6 commits July 15, 2026 16:46

style: satisfy clippy collapsible-conditional lints on Rust 1.96

5c9e448

hedgar2017 force-pushed the az-evm-assembly-worker-pool branch from 1ae05b1 to 7655308 Compare July 15, 2026 12:49

nebasuke approved these changes Jul 15, 2026

View reviewed changes

hedgar2017 added this pull request to the merge queue Jul 15, 2026

Merged via the queue into main with commit ebc9059 Jul 15, 2026
54 checks passed

hedgar2017 deleted the az-evm-assembly-worker-pool branch July 15, 2026 16:51

nebasuke mentioned this pull request Jul 15, 2026

ci: cross-OS solx-tester suite on free macOS/Windows runners (ci:cross-os label + weekly cron) #553

Draft

hedgar2017 added a commit that referenced this pull request Jul 18, 2026

Revert "perf(core): reuse persistent worker subprocesses across trans…

7acbfa9

…lation units (#524)" This reverts commit ebc9059.

hedgar2017 mentioned this pull request Jul 18, 2026

DO NOT MERGE: revert 6 perf PRs to test the summary on real regressions #567

Closed

hedgar2017 mentioned this pull request Sep 2, 2026

Integration macos large hack #545

Draft

Uh oh!

Conversation

hedgar2017 commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Compile-time benchmark (--standard-json, EVMLA pipeline)

ens-verifiable-factory-solx--solx-legacy-dwarf

openzeppelin-contracts-0.34--solx-legacy-dwarf

uniswap-v4-core-solx--solx-legacy-dwarf

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nebasuke commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Claude comment

Uh oh!

hedgar2017 commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

nebasuke commented Jul 15, 2026

macOS validation results

Anatomy of the (already-fixed) hang

macOS x86-64

Other notes

Uh oh!

github-actions Bot commented Jul 15, 2026

Coverage Summary

Uh oh!

nebasuke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hedgar2017 commented Jul 9, 2026 •

edited

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading

Compile-time benchmark (`--standard-json`, EVMLA pipeline)

github-actions Bot commented Jul 13, 2026 •

edited

Loading

nebasuke commented Jul 14, 2026 •

edited

Loading

hedgar2017 commented Jul 14, 2026 •

edited

Loading