Skip to content

ci: run twice as many test files at once as cores, and drop the type-check step - #936

Merged
nissa-seru merged 2 commits into
anima-research:mainfrom
nissa-seru:paloma/ci-tests-8-at-once
Oct 30, 2026
Merged

nissa-seru merged 2 commits into
anima-research:mainfrom
nissa-seru:paloma/ci-tests-8-at-once

Conversation

@nissa-seru

@nissa-seru nissa-seru commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

This runs twice as many test files at once as the runner has cores in ci.yml's Build & Test and publish.yml's Test step, and drops both workflows' Type-check step, since Build compiles the same files with tsc.

In a run at 48f26ce56c at four files at once on Linux, summed over the 390 test files, their own CPU came to 33% of the time their processes ran. On four pinned CPUs of a Linux host, at 48f26ce56c, the suite took 317 s at four files at once, using 1.85 of the CPUs, and 208 s at eight, using 3.05; a second run at eight took 216 s (Tests). On GitHub's 4-core runner, in six attempts of this pull request's Build & Test on its first commit's tree, the Test step took a median 249 s, against a median 348 s in 30 earlier Build & Test jobs without it (Tests, Problem).

Problem

At main (c5306693e0), Build & Test sets AF_TEST_CONCURRENCY to the runner's core count (.github/workflows/ci.yml:104), and npm test passes it to node --test as --test-concurrency. On ubuntu-latest that's 4: a Build & Test log at 48f26ce56c prints "cores: 4 (Node's availableParallelism: 4); test files at once: 4".

In the 30 successful Build & Test jobs from October 29 at 21:40 UTC to October 30 at 10:58 UTC, 25 on pull requests and 5 on pushes, the job took a median 378 s (310 to 443 s) and its Test step a median 348 s (286 to 410 s), by GitHub's step times. Its Type-check step (.github/workflows/ci.yml:92–:93), npx tsc --noEmit, took a median 5 s. Its Build step runs npm run build, which is tsc (package.json:28), over the same tsconfig.json.

Changes

  • Twice the cores. ci.yml's Build & Test (.github/workflows/ci.yml:97) and publish.yml's Test step (.github/workflows/publish.yml:51) each set AF_TEST_CONCURRENCY=$(node -p "2 * require('node:os').cpus().length"), the same line in both. The step's echo still prints the core count, then the number of test files at once.
  • Eight, not twelve. Twelve files at once was the fastest width in Tests' runs on the host, 186 s against 208 s and 216 s at eight. It also used 3.48 of the four CPUs, against 3.05 and 2.91 at eight, and its run had one failure, of a test that also failed once at four files at once without taskset. Eight keeps the suite further from using all four CPUs.
  • No Type-check step. Both workflows drop npx tsc --noEmit. Build's npm run build runs tsc over the same tsconfig.json, whose include is src/**/* and test/**/*. In Tests, with a type error added to a source file and one to a test file, it exits non-zero with the same two errors as npx tsc --noEmit.
  • CONTRIBUTING.md says CI runs twice as many test files at once as the runner has cores, and its command for running the suite as CI schedules it sets AF_TEST_CONCURRENCY the same way (CONTRIBUTING.md:215–:220).
  • test/ci-workflow-pins.test.ts adds "ci.yml and publish.yml compute AF_TEST_CONCURRENCY the same way" (:54). It compares the line that sets AF_TEST_CONCURRENCY, with or without export, in each workflow's npm test step, and fails unless each workflow has exactly one. Its header says why: if the two differ, a release's gate runs the suite in a way no pull request's run checked.

By git merge-tree on October 30 at 14:20 UTC, each of the 43 other pull requests open then merges with this head as it does with c5306693e0, the commit this head is on, with the same conflicted files and conflict blocks.

Tests

Build & Test on GitHub's runner. This pull request's CI run 41063740020 tested GitHub's merge of its first commit, 58e2db34a6, into main as it was then, c5306693e0. Since 58e2db34a6 is on c5306693e0, the merge has 58e2db34a6's tree. The run's Build & Test ran in six attempts one after another, the first and five re-runs, on ubuntu-latest with Node 22.12.0. Each attempt's log prints "cores: 4 (Node's availableParallelism: 4); test files at once: 8", and each attempt's Test step ran 9,513 tests, of which 9,450 passed, 63 were skipped and none failed. By GitHub's step times:

Attempt Test step Job
1 265 s 295 s
2 250 s 281 s
3 248 s 281 s
4 197 s 226 s
5 245 s 277 s
6 251 s 286 s

The Test step's median is 249 s (197 to 265 s), and the job's 281 s. In the 30 jobs in Problem, without this change, they were 348 s (286 to 410 s) and 378 s.

CI also ran at this head, 0a147a1b48, on GitHub's merge of it into main a687fa61c1, which is c5306693e0 with #934 merged. Its one Build & Test attempt, in run 41070691660, printed "test files at once: 8", and its Test step ran 9,469 tests, of which 9,406 passed, 63 were skipped and none failed. Its Test step took 239 s, and the job 270 s.

The suite's time, by files at once. Each of these runs is at 48f26ce56c, an earlier main, on Linux x86_64 with 12 CPUs, Node 24.22.2 and npm 11.19.1, after npm install with node_modules and package-lock.json removed, and npm run build. Each ran node --test over dist/test/*.test.js with --test-concurrency set, under taskset -c 0-3, with a preload recording each file's process time and CPU. The host ran other work at the same time. Over the run of 4, 8 and 12 files at once, the rest of the host used 6.4 of its 12 CPUs on average, and 8.0 over the run of 8 and then 6. taskset pins the suite's processes to CPUs 0 to 3, and other work could still run there.

Files at once Suite CPUs used, of 4 Failures
4 317 s 1.85 0
6 246 s 2.48 0
8 208 s 3.05 0
8 216 s 2.91 1
12 186 s 3.48 1
  • "CPUs used" is the files' own CPU, and the CPU of the child processes each file had waited for by the time it exited, divided by the run's duration_ms.
  • The failure at 8 is test/mcpl-no-reconnect-after-stop.test.ts's "once stop() has begun, a server with reconnect launches no child again, …", with spawnSync ps ENOBUFS. Since test: the no-reconnect-after-stop test asks pgrep for its launches, not ps for every process #929, which main has at c5306693e0, that test runs pgrep in place of ps (test/mcpl-no-reconnect-after-stop.test.ts:43).
  • The failure at 12 is test/workspace-three-way.test.ts's "a re-attached watcher runs its pass at its own ready, so a change after that pass reaches the store", at "some of chokidar's directory reads waited at the closed gate". The same test failed once in a run at four files at once on all 12 CPUs, without taskset.

The pin test. At a clean checkout of this head, 0a147a1b48, on Linux with Node 24.22.2, npm 11.19.1, typescript 7.0.2 and @types/node 22.20.5, after npm install and npm run build, test/ci-workflow-pins.test.ts passes 6 of 6. With .github/workflows/publish.yml as it is at c5306693e0, 5 pass and "ci.yml and publish.yml compute AF_TEST_CONCURRENCY the same way" fails. With .github/workflows/ci.yml as it is at c5306693e0 instead, the same test fails and the other 5 pass. With a line export AF_TEST_CONCURRENCY=4 added before npm test in publish.yml's Test step, the same test fails, with ".github/workflows/publish.yml's npm test step sets AF_TEST_CONCURRENCY once". With that line and the test file as it is at 58e2db34a6, rebuilt, all 6 pass.

Build in place of Type-check. Same checkout and versions. npx tsc --noEmit and npm run build each exit 0 with no errors. With one type error added to src/index.ts and one to test/ci-workflow-pins.test.ts, npx tsc --noEmit exits 1 and npm run build exits 2, and both report the same two errors:

src/index.ts(220,14): error TS2322: Type 'string' is not assignable to type 'number'.
test/ci-workflow-pins.test.ts(133,7): error TS2322: Type 'number' is not assignable to type 'string'.

Not verified

  • The host's timings at this head's base. They ran at 48f26ce56c, and this head is on c5306693e0.
  • More runs at each width on the host. Six and twelve files at once ran once each there, and eight ran twice.
  • publish.yml itself, which runs at a release.

  • This change is internal-only, CI and a test, with no changelog fragment. The changelog check asks for one only when src/ changes (.github/workflows/changelog.yml:32–:36).

Paloma-2644 measured the suite, made the change and re-ran its Build & Test on GitHub, Ada-1017 chose it from Paloma-2644's measurements, and Ambrose-2591 read both commits word for word. An internal review found that the pin test missed a later export AF_TEST_CONCURRENCY= line, and the second commit makes it count one. Daniel-2612 wrote this description.

Paloma-2644 added 2 commits October 30, 2026 08:34
…check step

The suite's files spend most of their time waiting on timers and child
processes, not the CPU. Measured on a Linux host, Node 24.22.2, with
each file's process timed from its start to its exit:

- at four files at once, as now, the suite took 314 and 325 s with the
  host's twelve cores available, and 317 s on four pinned CPUs. In the
  325 s run its 390 files took 1,239 s between them, the four slots
  were 95% busy, and the files' own CPU was 33% of their time and their
  children's 17%. On the four pinned CPUs it used 1.85 of them.
- on the same four pinned CPUs: at eight files at once, 208 s, using
  3.05 of the cores, and 216 s in a second run; at six, 246 s; at
  twelve, 186 s, using 3.48.

Build & Test and publish.yml's Test step now set AF_TEST_CONCURRENCY to
twice the core count, the same expression in each, so a release's gate
runs the suite as a pull request's run did. ci-workflow-pins.test.ts
now checks that the two compute it the same way; it fails with
publish.yml as it was.

The Type-check step goes from both workflows: Build runs tsc over the
same tsconfig.json and fails on the same errors.

CONTRIBUTING.md's account of CI's concurrency, and its command for
running the suite as CI schedules it, follow.
The comments over both workflows' Test step gave the reason for the
width and the measurements behind it, which the commit that set the
width carries.

The check that ci.yml and publish.yml compute AF_TEST_CONCURRENCY the
same way read only a bare `AF_TEST_CONCURRENCY=` line in each npm test
step, so a later `export AF_TEST_CONCURRENCY=4` in one of them left the
two widths different with every test passing. It now reads the
`export` form too, so a second assignment fails its check that the step
sets the width once. With such a line added before npm test in
publish.yml, the test fails; with the previous commit's test, it
passed.

Eight files at once rather than twelve: on the same four pinned CPUs,
twelve took 186 s against eight's 208 and 216 s, but used 3.48 of the
CPUs against eight's 3.05 and 2.91, leaving less for timing-sensitive
tests, and its one failure was such a test, workspace-three-way's
re-attached watcher, which had also failed once at four files at once
without pinning.
@nissa-seru
nissa-seru merged commit 71193a9 into anima-research:main Oct 30, 2026
12 checks passed
tel-0s added a commit that referenced this pull request Oct 30, 2026
…s eight at once (#936)

No conflict; CI workflows, CONTRIBUTING and the ci-workflow-pins test only.
Version 0.31.0; floors this branch's.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant