Repository navigation
ci: run twice as many test files at once as cores, and drop the type-check step - #936
Merged
nissa-seru merged 2 commits intoOct 30, 2026
Merged
Conversation
added 2 commits
October 30, 2026 08:34
…check step The suite's files spend most of their time waiting on timers and child processes, not the CPU. Measured on a Linux host, Node 24.22.2, with each file's process timed from its start to its exit: - at four files at once, as now, the suite took 314 and 325 s with the host's twelve cores available, and 317 s on four pinned CPUs. In the 325 s run its 390 files took 1,239 s between them, the four slots were 95% busy, and the files' own CPU was 33% of their time and their children's 17%. On the four pinned CPUs it used 1.85 of them. - on the same four pinned CPUs: at eight files at once, 208 s, using 3.05 of the cores, and 216 s in a second run; at six, 246 s; at twelve, 186 s, using 3.48. Build & Test and publish.yml's Test step now set AF_TEST_CONCURRENCY to twice the core count, the same expression in each, so a release's gate runs the suite as a pull request's run did. ci-workflow-pins.test.ts now checks that the two compute it the same way; it fails with publish.yml as it was. The Type-check step goes from both workflows: Build runs tsc over the same tsconfig.json and fails on the same errors. CONTRIBUTING.md's account of CI's concurrency, and its command for running the suite as CI schedules it, follow.
The comments over both workflows' Test step gave the reason for the width and the measurements behind it, which the commit that set the width carries. The check that ci.yml and publish.yml compute AF_TEST_CONCURRENCY the same way read only a bare `AF_TEST_CONCURRENCY=` line in each npm test step, so a later `export AF_TEST_CONCURRENCY=4` in one of them left the two widths different with every test passing. It now reads the `export` form too, so a second assignment fails its check that the step sets the width once. With such a line added before npm test in publish.yml, the test fails; with the previous commit's test, it passed. Eight files at once rather than twelve: on the same four pinned CPUs, twelve took 186 s against eight's 208 and 216 s, but used 3.48 of the CPUs against eight's 3.05 and 2.91, leaving less for timing-sensitive tests, and its one failure was such a test, workspace-three-way's re-attached watcher, which had also failed once at four files at once without pinning.
tel-0s
added a commit
that referenced
this pull request
Oct 30, 2026
…s eight at once (#936) No conflict; CI workflows, CONTRIBUTING and the ci-workflow-pins test only. Version 0.31.0; floors this branch's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This runs twice as many test files at once as the runner has cores in
ci.yml's Build & Test andpublish.yml's Test step, and drops both workflows' Type-check step, since Build compiles the same files withtsc.In a run at
48f26ce56cat four files at once on Linux, summed over the 390 test files, their own CPU came to 33% of the time their processes ran. On four pinned CPUs of a Linux host, at48f26ce56c, the suite took 317 s at four files at once, using 1.85 of the CPUs, and 208 s at eight, using 3.05; a second run at eight took 216 s (Tests). On GitHub's 4-core runner, in six attempts of this pull request's Build & Test on its first commit's tree, the Test step took a median 249 s, against a median 348 s in 30 earlier Build & Test jobs without it (Tests, Problem).Problem
At
main(c5306693e0), Build & Test setsAF_TEST_CONCURRENCYto the runner's core count (.github/workflows/ci.yml:104), andnpm testpasses it tonode --testas--test-concurrency. On ubuntu-latest that's 4: a Build & Test log at48f26ce56cprints "cores: 4 (Node's availableParallelism: 4); test files at once: 4".In the 30 successful Build & Test jobs from October 29 at 21:40 UTC to October 30 at 10:58 UTC, 25 on pull requests and 5 on pushes, the job took a median 378 s (310 to 443 s) and its Test step a median 348 s (286 to 410 s), by GitHub's step times. Its Type-check step (
.github/workflows/ci.yml:92–:93),npx tsc --noEmit, took a median 5 s. Its Build step runsnpm run build, which istsc(package.json:28), over the sametsconfig.json.Changes
ci.yml's Build & Test (.github/workflows/ci.yml:97) andpublish.yml's Test step (.github/workflows/publish.yml:51) each setAF_TEST_CONCURRENCY=$(node -p "2 * require('node:os').cpus().length"), the same line in both. The step'sechostill prints the core count, then the number of test files at once.taskset. Eight keeps the suite further from using all four CPUs.npx tsc --noEmit. Build'snpm run buildrunstscover the sametsconfig.json, whoseincludeissrc/**/*andtest/**/*. In Tests, with a type error added to a source file and one to a test file, it exits non-zero with the same two errors asnpx tsc --noEmit.CONTRIBUTING.mdsays CI runs twice as many test files at once as the runner has cores, and its command for running the suite as CI schedules it setsAF_TEST_CONCURRENCYthe same way (CONTRIBUTING.md:215–:220).test/ci-workflow-pins.test.tsadds "ci.yml and publish.yml compute AF_TEST_CONCURRENCY the same way" (:54). It compares the line that setsAF_TEST_CONCURRENCY, with or withoutexport, in each workflow'snpm teststep, and fails unless each workflow has exactly one. Its header says why: if the two differ, a release's gate runs the suite in a way no pull request's run checked.By
git merge-treeon October 30 at 14:20 UTC, each of the 43 other pull requests open then merges with this head as it does withc5306693e0, the commit this head is on, with the same conflicted files and conflict blocks.Tests
Build & Test on GitHub's runner. This pull request's CI run 41063740020 tested GitHub's merge of its first commit,
58e2db34a6, intomainas it was then,c5306693e0. Since58e2db34a6is onc5306693e0, the merge has58e2db34a6's tree. The run's Build & Test ran in six attempts one after another, the first and five re-runs, on ubuntu-latest with Node 22.12.0. Each attempt's log prints "cores: 4 (Node's availableParallelism: 4); test files at once: 8", and each attempt's Test step ran 9,513 tests, of which 9,450 passed, 63 were skipped and none failed. By GitHub's step times:The Test step's median is 249 s (197 to 265 s), and the job's 281 s. In the 30 jobs in Problem, without this change, they were 348 s (286 to 410 s) and 378 s.
CI also ran at this head,
0a147a1b48, on GitHub's merge of it intomaina687fa61c1, which isc5306693e0with #934 merged. Its one Build & Test attempt, in run 41070691660, printed "test files at once: 8", and its Test step ran 9,469 tests, of which 9,406 passed, 63 were skipped and none failed. Its Test step took 239 s, and the job 270 s.The suite's time, by files at once. Each of these runs is at
48f26ce56c, an earliermain, on Linux x86_64 with 12 CPUs, Node 24.22.2 and npm 11.19.1, afternpm installwithnode_modulesandpackage-lock.jsonremoved, andnpm run build. Each rannode --testoverdist/test/*.test.jswith--test-concurrencyset, undertaskset -c 0-3, with a preload recording each file's process time and CPU. The host ran other work at the same time. Over the run of 4, 8 and 12 files at once, the rest of the host used 6.4 of its 12 CPUs on average, and 8.0 over the run of 8 and then 6.tasksetpins the suite's processes to CPUs 0 to 3, and other work could still run there.duration_ms.test/mcpl-no-reconnect-after-stop.test.ts's "once stop() has begun, a server with reconnect launches no child again, …", withspawnSync ps ENOBUFS. Since test: the no-reconnect-after-stop test asks pgrep for its launches, not ps for every process #929, whichmainhas atc5306693e0, that test runspgrepin place ofps(test/mcpl-no-reconnect-after-stop.test.ts:43).test/workspace-three-way.test.ts's "a re-attached watcher runs its pass at its own ready, so a change after that pass reaches the store", at "some of chokidar's directory reads waited at the closed gate". The same test failed once in a run at four files at once on all 12 CPUs, withouttaskset.The pin test. At a clean checkout of this head,
0a147a1b48, on Linux with Node 24.22.2, npm 11.19.1,typescript7.0.2 and@types/node22.20.5, afternpm installandnpm run build,test/ci-workflow-pins.test.tspasses 6 of 6. With.github/workflows/publish.ymlas it is atc5306693e0, 5 pass and "ci.yml and publish.yml compute AF_TEST_CONCURRENCY the same way" fails. With.github/workflows/ci.ymlas it is atc5306693e0instead, the same test fails and the other 5 pass. With a lineexport AF_TEST_CONCURRENCY=4added beforenpm testinpublish.yml's Test step, the same test fails, with ".github/workflows/publish.yml's npm test step sets AF_TEST_CONCURRENCY once". With that line and the test file as it is at58e2db34a6, rebuilt, all 6 pass.Build in place of Type-check. Same checkout and versions.
npx tsc --noEmitandnpm run buildeach exit 0 with no errors. With one type error added tosrc/index.tsand one totest/ci-workflow-pins.test.ts,npx tsc --noEmitexits 1 andnpm run buildexits 2, and both report the same two errors:Not verified
48f26ce56c, and this head is onc5306693e0.publish.ymlitself, which runs at a release.src/changes (.github/workflows/changelog.yml:32–:36).Paloma-2644 measured the suite, made the change and re-ran its Build & Test on GitHub, Ada-1017 chose it from Paloma-2644's measurements, and Ambrose-2591 read both commits word for word. An internal review found that the pin test missed a later
export AF_TEST_CONCURRENCY=line, and the second commit makes it count one. Daniel-2612 wrote this description.