Skip to content

Reduce complete hosted test execution to five minutes - #19

Draft
david-hoffman wants to merge 37 commits into
codex-mainfrom
codex/test-suite-under-five-minutes
Draft

david-hoffman wants to merge 37 commits into
codex-mainfrom
codex/test-suite-under-five-minutes

Conversation

@david-hoffman

@david-hoffman david-hoffman commented Oct 6, 2026 •

Copy link
Copy Markdown
Owner

The reported complete test suite took about 24 minutes. This draft preserves every original test/assertion, real clean installation, fresh runtime fingerprint and verification gate. One reviewed hosted run completed the all-platform assigned-test window in 299.669 seconds. The automatic PR repeat passed every correctness gate, but its slowest verifier receipt measured 300.160 seconds. Performance acceptance for that repeat is unresolved; further balancing is being verified on a separate trial branch before any source update to this draft.

Depends on #18 (99ae885806e17181631b914ce58e83fbffc373d9); targets codex-main. This stacked draft carries that prerequisite until it is integrated. The performance work preserves numerical and release implementations, the hashed dependency lock and original finite deadlines, except the previously owner-approved real-coverage harness allowance.

The verifier isolates worker scratch and pytest caches, keeps shared-fixture consumers together, and balances both scheduling levels using retained platform timings. Black and Flake8 each use one worker inside the already parallel test layer. Windows disposable environments use runner temporary storage and still perform fresh real installations. Explicit coverage roots measure owned modules and copied CLI helpers without tracing unrelated dependencies. Missing, duplicate, skipped or failed execution and incomplete coverage still fail verification.

Exact candidate: aa2650bba6055ad0c87d76ab92534141aea1c8ae.

Validation: run 37523179737 passed all 28 jobs and ci-required. Each platform executed 2,210 unique tests with 6,630 passing phases, all 19 real installation/probe cases, and exact global/package/file coverage of 3,510 statements and 900 branches across 21 files. All 67 worker records and raw parent/child coverage are retained. No new numerical tests were added.

Measurement Result
All twenty complete assigned-test steps, fractional workflow-log enclosure 299.669 s
Linux / macOS / Windows step windows 299.669 / 233.256 / 277.611 s
Local canonical full command, eight workers 304.23 s
Whole hosted workflow 757 s
Summed runner time, including provisioning wait 8,740 s
Summed idle wait / shard setup excluding idle 2,479 / 946 s
Collection commands, Linux / macOS / Windows 66 / 71 / 79 s
Aggregation commands, Linux / macOS / Windows 22 / 31 / 21 s

The hosted result uses twenty machines: seven Linux with four local workers, five macOS with three, and eight Windows with three. GitHub's second-resolution step metadata records 300 s; fractional logs conservatively enclose observer creation, complete verifier execution and observer shutdown within 299.6689119 s. Fractional group-start spread is 0.1507095 s. Exact subprocess lifecycle times and cross-host clock accuracy were not measured; queue-only time is unavailable. The measured margin is only 0.331 s, so future host variation can exceed the target. Collection, setup, waiting, artifacts and aggregation remain in the separately reported whole-workflow time.

Fresh independent ROOT review #11 accepted structure, correctness and the observed timing target with no material findings. It verified preservation of all 645 original test functions, 1,250 function assertions, 1,535 module assertions, signatures and decorators; the 540-source seed/partitions; all worker seals/hashes and 12,761 raw coverage files; and 9,615 diagnostic snapshot hashes. Review and exact-candidate receipts are retained outside the tracked tree at /private/tmp/dphtools-performance-review-aa2650b-report.md and /private/tmp/dphtools-performance-final-aa2650b.json. Repeatability remains unproven. Generated workflow diagnostic scripts and Git shell hooks lack owned-runtime statement/branch measurement; diagnostic snapshots do not replace gate evidence.

The automatic PR repeat, run 37525821216, passed all 28 jobs and ci-required. Linux shard 4 recorded an internal monotonic verifier interval of 300.159587758 s; diagnostic shutdown cannot explain that interval. Its synthetic merge checkout 736f35c632fa8167e8604cda846e9ae3f7e0d17a has the identical Git tree a0d09836b68b55f6389932b8cc51186ffe2d3e13 as the reviewed head. This repeat misses the 300-second performance contract by at least 0.1596 s. Retained logs and receipts are under /private/tmp/dphtools-performance-ci-37525821216/. The existing draft head remains unchanged while rebalancing is evaluated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant