Repository navigation
Shard the full E2E run and make CI-sensitive probes reproducible - #877
Conversation
The nightly full run failed the same timing-sensitive tests every night: on the 4-vCPU runner, four software-rendered browsers space frames far enough apart that presses miss their windows or a short chart ends before the test acts (#792). Run the full scope on two workers with a 150-minute limit, keep four for the smoke run, and let a manual run narrow the tests with a grep input.
On the CI runner Synesthesia's software rendering still starved four tests on two workers: the course maker's input never resolved, and the auto-timing and fixed-display presses missed their windows (#792). Specs that name no built-in skin now open Plain; the two option- semantics loops that kept failing run on Plain and Native only. Synesthesia stays covered where a spec names it.
…ndering" This reverts commit f2e14e1.
Two workers still starved 31 tests on the full run (#792): presses on the page missed their windows, text and audio probes fell under their thresholds. Split the full run over four runners with one browser each, so each software-rendered browser has the runner to itself; the smoke run and grep runs stay on one runner.
CI traces showed the features working while the probes failed (#792): Linux draws the chart text thin, so few pixels pass a near-white test; a 50 ms frame can land a page-timed press in GOOD; and the course maker spends seconds per click. Count bright pixels and compare the cleared text with the shown one, count GOOD as a hit where the test checks key binding or gauge policy, and give the course maker a longer timeout.
Exports Benchmark🔴 4 regressions · 5 improvements · 16 within noise · 885 compared
Regressions (4)
Improvements (5)
Changed packages
All 885 cases
Removed exports (1)
Within noise (16): past the threshold, but the runs overlap
…and 4 more. I/O-bound, not judged (19)These cases touch the filesystem, so their time follows the OS cache and the disk more than the code.
Excluded: too fast to time (77)A timed sample lasted 1 µs or less on at least one side (usually a base commit from before calls were batched), so its time is timer noise and no change is shown.
…and 65 more. Run details
|
Closes #792.
Problem
The nightly full E2E run failed every night. It ran
E2E_WORKERS=4on one 4-vCPU ARM runner that renders in software, which spaced frames far apart. As a result:Change
Workflow (
.github/workflows/e2e.yml)--shard N/4,E2E_WORKERS=1), with a 90-minute limit per shard. Each shard uploads its own report artifact.grepinput onworkflow_dispatchruns selected tests on one runner.Probes, adjusted where CI traces showed the feature working but the measurement failing
chart-text: count bright rather than near-white pixels, because Linux draws the thin text with few fully white pixels. A cleared text now compares against the midpoint between the empty and shown counts.bmson-zero-total,bmson-generic: count GOOD as a hit, because these tests check key binding and gauge policy rather than accuracy, and a 50 ms CI frame can land a page-timed press in GOOD. Thebmson-zero-totalgauge assertion (no recovery) is unchanged.course-maker: timeout raised to 360 s, because it takes several hundred UI steps at about 2.6 s per click on CI.Verification on CI (this branch)
🤖 Generated with Claude Code