Skip to content

Consolidate redundant verifier fixture preflight - #21

Closed
david-hoffman wants to merge 1 commit into
codex-mainfrom
codex/local-verification-opt-pr
Closed

david-hoffman wants to merge 1 commit into
codex-mainfrom
codex/local-verification-opt-pr

Conversation

@david-hoffman

Copy link
Copy Markdown
Owner

Consolidate the invariant Black stand-in preflight from 75 launches to one per verifier test module. All 75 cases keep their own private VerificationCommand fixture. No cases, assertions, numerical tolerances, deadlines, coverage settings, dependencies, workflows or verification gates change. The original 48-case rationale documents the same setup purpose; 27 cases were inherited from the newer base.

The complete-suite 300-second goal remains unmet. Current warm median is 1,293.802 s, 993.802 s over the goal; cold is 1,311.454 s, 1,011.454 s over. The two warm pairs do not establish a consistent whole-suite speedup (candidate median 8.566 s / 0.667% slower). This PR removes demonstrated duplicate setup; it does not claim the overall performance objective is achieved.

Validation on exact unchanged candidate 445432eeb15298168cbadc5f26c8951677e84023, frozen integration baseline a835a490373835fa01e4460ce2159365150d1946:

Condition Baseline wall time (s) Candidate wall time (s)
cold 1355.494931375 1311.453771084
warm-1 1253.927822333 1306.662380042
warm-2 1316.542752208 1280.941088541

All six complete local runs exited 0: 1,831 passed, zero failures/errors/skips, 185 warnings. All 24 canonical stages passed, including quality, audit, fresh builds/installations, tests, subprocess coverage combination and final report validation. Receipt/input/log seals and retained artifact hashes were checked. Both trees and dependency inventories stayed unchanged.

Owned package Statements Branches Coverage
dphtools 328/328 116/116 100%
dphtools.utils 1645/1645 408/408 100%
tools 1179/1179 254/254 100%
Global 3152/3152 778/778 100%

All 19 owned Python runtime files individually have exact 100% statement/branch coverage and zero exclusions. Relevant subprocess/copied-helper data is combined and never-imported owned files remain in the gate. Shell-hook statement/branch coverage is unsupported by coverage.py; real local bare-Git hook tests remain included. Evidence is local macOS only. No remote runner or hosted verification evidence is used.

Measured command, launched in each isolated arm:

/private/tmp/dphtools-python13-isolated/bin/python tools/verification.py full

Hardware: Mac16,12, macOS 27.0 arm64, 10 physical/logical cores, 16 GiB RAM. Python 3.13.12. One pytest worker, no xdist workers; BLACK_NUM_WORKERS=1, PYTEST_XDIST_AUTO_NUM_WORKERS=1, and OMP_NUM_THREADS, OPENBLAS_NUM_THREADS, MKL_NUM_THREADS, VECLIB_MAXIMUM_THREADS, NUMEXPR_NUM_THREADS, BLIS_NUM_THREADS all 1. PYTEST_ADDOPTS=--durations=30, PYTHONHASHSEED=0, MPLBACKEND=Agg. No CPU/worker/thread allocation increased. Runs were sequential in order candidate cold, baseline cold, baseline warm1, candidate warm1, baseline warm2, candidate warm2.

External monotonic timing includes declared cold preparation before command launch and every canonical step through final exit. Cold clears separate arm XDG/pip/Matplotlib and checkout mypy/pytest/build/egg-info/bytecode caches inside the timer. Warm retains each arm caches. OS/shared platform caches are not claimed cold. The same locked outer runtime is a command precondition for both arms; required nested environments, installation checks and report validation stay inside the command. Independent post-run evidence archival is outside the canonical timer. Audit/install operations access network data; execution stays local.

Interpreter executable SHA256 4e999a1436fc1ff0c1d6a6ecc8e93566ecfc6706157de6232096238c420cf357; unchanged requirements-dev.lock SHA256 4ad780fe68daa3e413bf7156cd158768450c26c7c243013ceac304c33fd47eca. All locked installed versions matched before/after and across arms (except the freshly rebuilt distribution under test). Key versions: NumPy 2.5.3, SciPy 1.18.1, pandas 3.0.6, Matplotlib 3.11.2, scikit-image 0.26.0, pytest 9.1.1, coverage 7.16.1, Black 26.5.1, pip 26.2.1, mypy 2.3.1.

Measured bottlenecks: candidate cold tests 1,280.289 s (97.62% of total); release workflow module 483.593 s, release smoke 212.428 s, reuse 148.419 s, verifier 109.319 s. Five installation-retargeting cases (PYTHONPATH, PYTHONHOME, PIP_TARGET, PIP_PREFIX, PIP_USER) alone total 399.890 s. Their distinct behaviors and real clean installations are retained. Verifier module times were 111.816/112.067/112.442 s baseline versus 109.319/107.301/106.758 s candidate; this smaller metric does not substitute for the unmet full-command target.

All earlier full attempts are disclosed:

Attempt Wall time (s) Result
Original baseline initial cold profile 133.938941875 PASS (historical older base)
Original baseline cold priming 128.463393083 PASS (historical older base)
Original candidate cold 120.668876292 PASS (historical older base)
Original baseline warm-1 94.738758000 PASS (historical older base)
Original candidate warm-1 92.176546250 PASS (historical older base)
Original baseline warm-2 93.783640458 PASS (historical older base)
Original candidate warm-2 91.251515750 PASS (historical older base)
Original baseline warm-3 95.029757875 PASS (historical older base)
Original candidate warm-3 93.280996709 PASS (historical older base)
Earlier integration audit 27.269317125 FAIL: disk exhaustion; downstream blocked
Earlier integration full 852.344722375 FAIL: disk exhaustion; 1608 pass/36 fail/187 errors
Recreated wrapper full 1337.319159791 FAIL: 1813 pass/18 fail; customized base and nested Black 23.3.0
Restricted-network audit bootstrap 29.859626875 FAIL: downstream tests/coverage blocked; absent reports explicitly retained

Historical baseline e557c7d5822be5d8413d039454782eb7772c9074, candidate/reviewed checkpoint d66ea14e0e402f137aba6dba7aeb71232396d43c, command /private/tmp/dphtools-verification-env/bin/python tools/verification.py full: all nine old passes had 1,232 tests, 2,094/2,094 statements and 544/544 branches. Their vanished temporary raw evidence cannot be independently reread and is not reused for current verification. Exact executable paths for the first two historical failed integration attempts are unavailable in retained raw data; both ran canonical tools/verification.py full. A historical independent reviewer also failed ENOSPC and supplied no acceptance.

Environment failures were independently classified as tooling. Primary Miniconda remains intact; a private base copies identical Python/stdlib/native-library bytes and installs the unchanged lock directly so nested environments inherit locked tools. Fresh preflight and all 59 affected reuse tests passed without candidate/test changes. The external archival helper initially failed on missing reports after the restricted audit attempt; it was corrected to retain partial failed receipts before the matched plan froze. No verifier gate changed.

Current full measurements: 7,825.023 s; all 19 full attempts: 11,015.148 s, 15 pass/4 fail. The original two-scenario specialist slice had B acceptance at round 1 and zero C repairs; all 35 nonfixture syntax trees on the current base are preserved. Fresh independent root reviewer accepted this exact candidate with no blocking defects; no edits or repairs. Session 01a11583-fffb-7ca0-997d-c6899f3e0fea, external review wall time 385.481 s of the 900 s budget. It directly validated all six seals, reports, raw coverage and all 890 runtime-copy file hashes. Performance benefit remains unproven. No merge/release is requested.

@david-hoffman
david-hoffman deleted the codex/local-verification-opt-pr branch October 7, 2026 20:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant