Repository navigation
Reduce complete hosted test execution to five minutes - #19
Draft
david-hoffman wants to merge 37 commits into
Draft
david-hoffman wants to merge 37 commits into
david-hoffman wants to merge 37 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reported complete test suite took about 24 minutes. This draft preserves every original test/assertion, real clean installation, fresh runtime fingerprint and verification gate. One reviewed hosted run completed the all-platform assigned-test window in 299.669 seconds. The automatic PR repeat passed every correctness gate, but its slowest verifier receipt measured 300.160 seconds. Performance acceptance for that repeat is unresolved; further balancing is being verified on a separate trial branch before any source update to this draft.
Depends on #18 (
99ae885806e17181631b914ce58e83fbffc373d9); targetscodex-main. This stacked draft carries that prerequisite until it is integrated. The performance work preserves numerical and release implementations, the hashed dependency lock and original finite deadlines, except the previously owner-approved real-coverage harness allowance.The verifier isolates worker scratch and pytest caches, keeps shared-fixture consumers together, and balances both scheduling levels using retained platform timings. Black and Flake8 each use one worker inside the already parallel test layer. Windows disposable environments use runner temporary storage and still perform fresh real installations. Explicit coverage roots measure owned modules and copied CLI helpers without tracing unrelated dependencies. Missing, duplicate, skipped or failed execution and incomplete coverage still fail verification.
Exact candidate:
aa2650bba6055ad0c87d76ab92534141aea1c8ae.Validation: run 37523179737 passed all 28 jobs and
ci-required. Each platform executed 2,210 unique tests with 6,630 passing phases, all 19 real installation/probe cases, and exact global/package/file coverage of 3,510 statements and 900 branches across 21 files. All 67 worker records and raw parent/child coverage are retained. No new numerical tests were added.The hosted result uses twenty machines: seven Linux with four local workers, five macOS with three, and eight Windows with three. GitHub's second-resolution step metadata records 300 s; fractional logs conservatively enclose observer creation, complete verifier execution and observer shutdown within 299.6689119 s. Fractional group-start spread is 0.1507095 s. Exact subprocess lifecycle times and cross-host clock accuracy were not measured; queue-only time is unavailable. The measured margin is only 0.331 s, so future host variation can exceed the target. Collection, setup, waiting, artifacts and aggregation remain in the separately reported whole-workflow time.
Fresh independent ROOT review #11 accepted structure, correctness and the observed timing target with no material findings. It verified preservation of all 645 original test functions, 1,250 function assertions, 1,535 module assertions, signatures and decorators; the 540-source seed/partitions; all worker seals/hashes and 12,761 raw coverage files; and 9,615 diagnostic snapshot hashes. Review and exact-candidate receipts are retained outside the tracked tree at
/private/tmp/dphtools-performance-review-aa2650b-report.mdand/private/tmp/dphtools-performance-final-aa2650b.json. Repeatability remains unproven. Generated workflow diagnostic scripts and Git shell hooks lack owned-runtime statement/branch measurement; diagnostic snapshots do not replace gate evidence.The automatic PR repeat, run 37525821216, passed all 28 jobs and
ci-required. Linux shard 4 recorded an internal monotonic verifier interval of 300.159587758 s; diagnostic shutdown cannot explain that interval. Its synthetic merge checkout736f35c632fa8167e8604cda846e9ae3f7e0d17ahas the identical Git treea0d09836b68b55f6389932b8cc51186ffe2d3e13as the reviewed head. This repeat misses the 300-second performance contract by at least 0.1596 s. Retained logs and receipts are under/private/tmp/dphtools-performance-ci-37525821216/. The existing draft head remains unchanged while rebalancing is evaluated.