Integration macos large hack - #545
Draft
nebasuke wants to merge 5 commits into
Draft
Conversation
…nits Replace the fork/exec-per-unit model with a pool of persistent workers fed over pipes. The per-unit CBOR payload is split into a Session (project-wide data, sent once per worker) and a slim per-unit Job, and idle workers are held in a std Mutex<Vec> and reused. The process module is one entity per file: channel (the length-prefixed CBOR frame codec, via FrameRead/FrameWrite extension traits), session, job, output, pool (parent-side scheduler), worker (subprocess handle), and child (the subprocess loop). The child runs one long-lived stack-sized thread that owns the deserialized Session and compiles jobs until stdin closes, instead of spawning a thread and re-cloning the session per job. Worker stderr is inherited, so subprocess diagnostics stream straight to the parent. Because LLVM command-line options are process-global and now survive across units in one worker, always emit -evm-metadata-size to heal a stale value, and carry metadata_size into the size-fallback settings; reset IS_SIZE_FALLBACK per job. A worker is retired when its job set a spill-area size (occurrence- gated cl-option) or carried extra llvm_options, and never reused after a stack-too-deep response since its handler has already exited the process. When a reused worker dies mid-job, the job is retried once on a fresh worker so the death is not mis-attributed to the contract. Install the stack-error handler once per worker process instead of per unit, and build the rayon pool only in the parent, after the recursive-process branch, since workers compile a single unit and never use it.
Worker::execute now returns a plain crate::Result<EVMOutput> — a subprocess I/O failure folds into Error::Generic — so the ? drops a dead or errored worker and the pool is just pop-or-spawn, run, and return. This removes the retry, the WorkerError enum, dispatch, and worker_failed. Whether a worker may be reused lives on the data owners: Session::allows_worker_reuse (no extra llvm_options) and Job::allows_worker_reuse (no stack-too-deep spill area), each documenting the process-global cl-option it guards against. The cached reuse_workers field is removed. run_multi_pass_pipeline breaks the loop with its value instead of a declare-then-assign, and no longer threads the contract name.
Reset the process-global LLVM command-line option occurrences before every parse (new LLVMResetAllOptionOccurrences, exposed through llvm-sys and inkwell), so a translation unit never inherits an option a previous one set in the same persistent worker. This removes the whole option-leak layer: the -evm-metadata-size=0 heal is gone, and the pool reuses a worker unconditionally on success — Session/Job::allows_worker_reuse and the spill/llvm_options gating are deleted. Points solx-llvm, llvm-sys, and inkwell at their az-reset-option-occurrences branches. Byte-identical on the EVMLA smoke; CLI suite 397/397.
Temporary hack to exercise PR #524 (persistent worker subprocess pool) on GitHub's large macOS runners instead of the self-hosted Linux container. - Matrix the integration job over macos-15-large (Intel) and macos-15-xlarge (Apple Silicon); drop the `container:` block since macOS runners can't run job containers. LLVM/solc build from source via the same build-llvm/build-solc actions test.yaml uses on macOS. - Add test.yaml's macOS free-disk-space step (LLVM builds twice). - Suffix artifact names and PR-comment message-ids with the runner label so the two matrix legs don't clobber each other's uploads and comments. Not for merge.
nebasuke
added
ci:integration
|
📊 Hardhat Projects Report ( ➡️ Download |
nebasuke
added a commit
that referenced
this pull request
Jul 15, 2026
The current tip 1ae05b1 is green and faster than main on both arm64 runners (runs 29374444399, 29399344551), but PR #545's 4h26m xlarge hang was observed at 6bd1976 — before b8469e0 (worker retention) and the 1ae05b1 solx-llvm reset-option-occurrences pin. Re-point the harness at the old tip: a watchdog trip here confirms the hang belonged to that intermediate state and is fixed since. Also adds an Intel path for later: solx-tester rebuilt with nightly to dodge the 1.96.1 enum-niche miscompile (#547), solx kept on the pinned stable, watchdog raised to 75 min for the slow Intel minis. free-disk-space-macos is carried over from main (predates #533 here).
hedgar2017
force-pushed
the
az-evm-assembly-worker-pool
branch
from
July 15, 2026 12:49
1ae05b1 to
7655308
Compare
hedgar2017
pushed a commit
that referenced
this pull request
Sep 1, 2026
solx-tester has no macOS CI coverage anywhere; on PR #545 it was signal-killed within seconds on macos-15-large and ran for hours on macos-15-xlarge, but with PR #524 included, so mac breakage and #524 regressions were indistinguishable. This branch is plain main, so its results are the macOS baseline. The workflow mirrors test.yaml's build-llvm/build-solc parameters to hit the finished-install caches saved on main, invokes solx-tester directly so a signal death is named instead of collapsing to "subprocess failed without exit code", and samples hung processes before killing them. Crash reports and samples are uploaded as artifacts.
Contributor
|
Left this on |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.