Reduce complete hosted test execution to five minutes

LESSONS/20261005T175124Z-SPECTRUM-FITTING-001-coordinator-prominence-interval.md

@@ -0,0 +1,18 @@
+# Sequence prominence can hide a scalar-validation defect
+
+- ID: 20261005T175124Z-SPECTRUM-FITTING-001-coordinator-prominence-interval
+- Date: 2026-10-05T17:51:24Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: SciPy accepts a two-value prominence sequence as an interval. A
+  nominal invalid-scalar test using an interval that removes every peak can pass
+  through the separate no-peaks ValueError even if the fitter ignores scalar
+  validation.
+- Evidence: B round 1 found this in S23; the historical A report records its
+  correction. A source-free SciPy 1.18.1 probe with
+  `x=linspace(-5,5,101); y=3*exp(-x*x/2)` returned zero peaks for `[1,2]` and one
+  for `[0,10]`. The [primary definition](https://docs.scipy.org/doc/scipy-1.15.3/reference/generated/scipy.signal.find_peaks.html)
+  documents interval prominence. No product mutation was needed.
+- Lesson: For scalar-only wrapper validation, choose a sequence that the
+  underlying API would accept and that preserves an otherwise valid peak. Keep
+  unrelated sample, shape, ordering, and discovery preconditions valid.

LESSONS/20261005T183608Z-SPECTRUM-FITTING-001-coordinator-nested-base-tools.md

@@ -0,0 +1,9 @@
+# A copied virtual environment can retain an unsuitable base interpreter
+
+- ID: 20261005T183608Z-SPECTRUM-FITTING-001-coordinator-nested-base-tools
+- Date: 2026-10-05T18:36:08Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: A copied locked Python 3.13 environment passed dependency and copied-nested preflight checks, but real system-site nested environments inherited the original Conda base's Black 23.3.0 and unaudited startup inputs. The required Black version was 26.5.1.
+- Evidence: Candidate dd31c2c237ac119d1ba1a3008f6b16e7e48b41f8 full receipt full-w9crt1ji reported 18 verification-reuse failures, including 12 Black version mismatches. The unchanged verification-reuse tests passed all 59 cases on a disposable non-Conda CPython 3.12.14 base with the same applicable hashed lock; tools/verification_reuse.py measured 78/78 statements and 46/46 branches, without exclusions. A real nested system-site environment inherited pinned tools and installed/imported a wheel.
+- Lesson: Before an expensive full run, verify the actual base interpreter, its startup inputs, and the standard nested-environment creation path used by tests. Matching the selected environment's pins and a copied-nested probe does not establish those properties. Preserve the original environment and fix a disposable verification environment rather than weakening cache eligibility or version checks.

LESSONS/20261005T185328Z-SPECTRUM-FITTING-001-coordinator-positive-width-difference-step.md

@@ -0,0 +1,9 @@
+# A positive physical width can yield an unrepresentable difference step
+
+- ID: 20261005T185328Z-SPECTRUM-FITTING-001-coordinator-positive-width-difference-step
+- Date: 2026-10-05T18:53:28Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: The smallest positive float is an approved positive finite width, but halving it for a central-difference step rounds to zero. Gaussian and Lorentzian public fits leaked LinAlgError during uncertainty calculation, even after every new-helper statement and branch had been measured by ordinary fixtures.
+- Evidence: Candidate c7ee4099e5276c9218a2d5bd90891ba7f017c6c2; clean Python 3.12.14. A 309-case focused diagnostic measured the helper's 155/155 statements and 66/66 branches. A separate public call with x=linspace(-1,1,41), y[20]=2 and other samples zero, full guess [2,0,nextafter(0,1)], and no background returned LinAlgError for both profiles. Inputs stayed unchanged. R3 accepts a verified result or RuntimeError for numerical failure, so the leaked ValueError subclass is a product defect. Raw public probe and interrupted full evidence are retained in the task conversation's external evidence directory.
+- Lesson: Check representability of numerical perturbations as well as physical parameter positivity. Exercise valid domain limits independently, preserve numerical warnings, and honor the public failure-type boundary. Complete measured branch coverage does not establish behavior at every floating-point magnitude.

LESSONS/20261005T200454Z-SPECTRUM-FITTING-001-coordinator-numerical-observer-boundaries.md

@@ -0,0 +1,19 @@
+# Numerical observers must preserve permitted implementations
+
+- ID: 20261005T200454Z-SPECTRUM-FITTING-001-coordinator-numerical-observer-boundaries
+- Date: 2026-10-05T20:04:54Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: Dependency-failure observers rejected valid stacked/complex SVD
+  operands, and a covariance observer assumed float64 despite no output-dtype
+  contract. Real physical outputs do not require real-only internal operands.
+- Evidence: B6 reproduced valid stacked `(1,201,3)` rejection; the coordinator's
+  complex128 decomposition matched analytic real covariance at rtol1e-12 while
+  the old observer rejected it. B7 accepted corrected tests SHA-256
+  50bb168312355b5c3035f534195260f4a0081a6f171cdd6a6de52235774837dd
+  after46 independent checks. Native wider-range execution was unavailable on
+  this ARM host; higher-range reasoning remains explicitly distinguished from
+  native execution in the historical test report.
+- Lesson: Fault observers should constrain the public contract rather than
+  numerical backend representation. Determine representability from the actual
+  output dtype. Check legitimate alternate algorithms as well as wrong outputs.

LESSONS/20261005T205314Z-SPECTRUM-FITTING-001-coordinator-background-covariance-rounding.md

@@ -0,0 +1,9 @@
+# Large background offsets can corrupt finite covariance
+
+- ID: 20261005T205314Z-SPECTRUM-FITTING-001-coordinator-background-covariance-rounding
+- Date: 2026-10-05T20:53:14Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: A well-conditioned Gaussian fit with a constant background of 1e12 data units returned finite covariance with sigma variance about 15.3 times too small and no warning, despite complete measured coverage.
+- Evidence: Independent D public reproduction on commit820e17dc31d9121b7b1ff2f5c839c0e77d8ad70f, /private/tmp/dphtools-spectrum-D-review-bxbulkvc/covariance_probe.py; coordinator reproduction exit1. Expected sigma variance5.77897e-7 x-unit squared; returned3.77959e-8. Normalized Jacobian condition3.257. Zero-background control agrees. Evidence applies to local CPython3.12.14/macOS ARM; hosted results are not established.
+- Lesson: For covariance regressions, vary the background magnitude and compare the complete physical covariance with independently derived derivatives and actual RSS/(N-P). Finiteness and rank alone do not establish derivative accuracy after subtraction of large offsets. Preserve permitted numerical failure rather than accepting misleading finite uncertainty.

LESSONS/20261005T214425Z-SPECTRUM-FITTING-001-coordinator-physical-acceptance-scope.md

@@ -0,0 +1,9 @@
+# Numerical stress validity does not establish physical acceptance scope
+
+- ID: 20261005T214425Z-SPECTRUM-FITTING-001-coordinator-physical-acceptance-scope
+- Date: 2026-10-05T21:44:25Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: A covariance regression using a 1e12-data-unit offset, a height-three peak and noise around 0.01 data units was mathematically resolved but did not establish a physically representative measurement. The owner excluded this extreme offset from required acceptance.
+- Evidence: Historical checkpoint e5bc47e1448c5466f981cfb4dd29150e8539552f has two failing extreme-offset cases; public contract R4 records the owner's physical-plausibility clarification and retains ordinary fit/covariance requirements. The original numerical finding remains recorded in the 20261005T205314Z background-covariance lesson.
+- Lesson: Establish units, background provenance and a plausible noise model before making a stress fixture a required scientific acceptance case. Keep robustness checks distinct from representative fit and uncertainty checks; a local covariance oracle does not validate repeated-measurement confidence intervals. An owner scope correction must retain the original evidence and receive fresh test review.

LESSONS/20261005T222923Z-SPECTRUM-FITTING-001-coordinator-physical-unit-default-fit.md

@@ -0,0 +1,9 @@
+# Physical unit conversion can expose an unfinished default fit
+
+- ID: 20261005T222923Z-SPECTRUM-FITTING-001-coordinator-physical-unit-default-fit
+- Date: 2026-10-05T22:29:23Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: The resolved R4 spectrum fitted correctly in nanometres, but default LM returned the initial peak guesses as successful when the same coordinates and widths were converted to metres. Its covariance still matched the local recipe at the unfinished parameters.
+- Evidence: Candidate f63ebf2de3fc65c047b00a3631f84b8bf0c68437 passed 2118 tests and full owned coverage. Independent D2 public reproduction found RSS 1.19075 V² in metres versus 0.0170722 V² for nanometre/default and metre/realcustom fits; a feasible generating model had RSS 0.0172294 V². The observed center perturbation was 14.9 nm against a 0.85 nm starting width. The narrow public unit-gap packet and complete independent receipts remain in the task conversation.
+- Lesson: Include physically identical unit-converted inputs in scientific acceptance. Verify fit quality and stationarity alongside covariance; a solver success flag and mathematically consistent local covariance do not establish that optimization finished.

LESSONS/20261006T054452Z-SPECTRUM-FITTING-001-coordinator-measured-gap-witness.md

@@ -0,0 +1,9 @@
+# Confirm that a public witness reaches the measured coverage gap
+
+- ID: 20261006T054452Z-SPECTRUM-FITTING-001-coordinator-measured-gap-witness
+- Date: 2026-10-06T05:44:52Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: Two public numerical-covariance failures had different outcomes. The first supplied witness exercised a nonfinite Jacobian but left the originally missing statement and branch unmeasured.
+- Evidence: C3 full receipt full-hq3ox68t at candidate 1c72cadf8f31a7a5925b06eb7183aa5319e5fb52: 3338/3339 statements and 857/858 branches; retained focused2 coverage still missed _spectrum_fit.py line67/branch66→67, while steps-boundary-private-measurement.json and focused3 confirmed that outcome was exercised.
+- Lesson: Before handing a blind test author a semantic witness, privately confirm that the public reproduction exercises the actual missing outcome. Keep implementation and raw measurement outside the blind packet.

LESSONS/20261006T054452Z-SPECTRUM-FITTING-001-coordinator-warning-provenance.md

@@ -0,0 +1,9 @@
+# Keyword matching does not establish numerical warning relevance
+
+- ID: 20261006T054452Z-SPECTRUM-FITTING-001-coordinator-warning-provenance
+- Date: 2026-10-06T05:44:52Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: A test diagnostic classifier accepted display-only warnings containing fit/optimizer words and rejected conventional informative linear-algebra failures.
+- Evidence: B13 public review /private/tmp/dphtools-spectrum-B17-sqf9i07c/review.md and independent-diagnosis-diagnostics.json: Fit legend unavailable and Optimizer progress display failed were accepted, while Singular matrix and SVD did not converge were rejected. Tests checkpoint 173f8958ff2d2fc35508226143c83876197a80b755254bc0924dc4fc1c56261d was not accepted.
+- Lesson: Test warning preservation through known caller-controlled numerical provenance when the contract permits it. Do not claim a generic English substring classifier establishes arbitrary diagnostic relevance; review the actual numerical context separately.

LESSONS/20261006T144344Z-TEST-SUITE-PERFORMANCE-001-author-coverage-ancestor-glob.md

@@ -0,0 +1,9 @@
+# Coverage globs can capture a virtual environment through an ancestor name
+
+- ID: 20261006T144344Z-TEST-SUITE-PERFORMANCE-001-author-coverage-ancestor-glob
+- Date: 2026-10-06T14:43:44Z
+- Task/role: TEST-SUITE-PERFORMANCE-001 / routine author
+- Status: confirmed
+- Observation: `*/dphtools/**/*.py` also matched unrelated dependencies inside a checkout named `dphtools`, including coverage.py itself in a repository-local virtual environment.
+- Evidence: `tests/test_verification_coverage_scope.py::test_repo_named_dphtools_does_not_instrument_environment_dependencies` failed on the original configuration and passed with anchored owned-source instrumentation. It also verifies nested source, installed package, and copied CLI measurement from outside the checkout.
+- Lesson: Anchor owned source roots explicitly and test the raw measured-file set; report filtering alone can conceal costly instrumentation of unrelated code.

LESSONS/20261006T145753Z-TEST-SUITE-PERFORMANCE-001-author-worker-scratch.md

@@ -0,0 +1,9 @@
+# Keep worker scratch outside the checkout
+
+- ID: 20261006T145753Z-TEST-SUITE-PERFORMANCE-001-author-worker-scratch
+- Date: 2026-10-06T14:57:53Z
+- Task/role: TEST-SUITE-PERFORMANCE-001 / author
+- Status: confirmed
+- Observation: A private worker temporary directory inside its checkout still violates a genuine clean-install isolation assertion. On this host, the checkout also uses overlay storage while the operating-system temporary directory uses tmpfs.
+- Evidence: Candidate `d28bcd5` failed `test_actual_probe_measurement_and_real_failure_controls` with “Disposable environment must be outside checkout”; its interrupted full receipt is retained under `reports/verification/performance-candidate/`.
+- Lesson: Keep worker pytest and operating-system scratch directories outside the checkout, clean them after execution, and retain logs, receipts and raw coverage separately. Measure storage and base-interpreter scope when assessing suite performance.

LESSONS/20261006T152542Z-TEST-SUITE-PERFORMANCE-001-author-nested-environment-tools.md

@@ -0,0 +1,9 @@
+# Nested environments inherit the base interpreter's packages
+
+- ID: 20261006T152542Z-TEST-SUITE-PERFORMANCE-001-author-nested-environment-tools
+- Date: 2026-10-06T15:25:42Z
+- Task/role: TEST-SUITE-PERFORMANCE-001 / author
+- Status: confirmed
+- Observation: Locked tools installed only in an outer virtual environment were unavailable to real nested environments created with `system_site_packages=True`.
+- Evidence: Candidate `45373d7` failed real-venv reuse tests with missing Black/Flake8/pydocstyle. Installing the unchanged hashed lock directly in the private standalone base interpreter restored both `test_real_venv_directory_aliases_allow_unchanged_command_reuse` and `test_real_venv_reuses_exact_windows_hook_and_rechecks_changed_bytes`; the five focused correction cases passed in `/tmp/dphtools-performance-corrections.log`.
+- Lesson: Mirror CI's directly provisioned base interpreter for these full-suite fixtures; an outer virtual environment alone does not establish their dependency boundary.

LESSONS/20261006T155643Z-SPECTRUM-FITTING-001-coordinator-python-patch-skew.md

@@ -0,0 +1,9 @@
+# Floating Python patches can split one CI workflow
+
+- ID: 20261006T155643Z-SPECTRUM-FITTING-001-coordinator-python-patch-skew
+- Date: 2026-10-06T15:56:43Z
+- Task/role: SPECTRUM-FITTING-001 / coordinator
+- Status: confirmed
+- Observation: Separate Linux jobs requesting Python `3.10` resolved to 3.10.21 and 3.10.22 in the same workflow, causing exact manifest validation to reject a worker before tests ran.
+- Evidence: [Run 37483079692](https://github.com/david-hoffman/dphtools/actions/runs/37483079692), candidate `0634499296e143bab6a16daba3f504b5c6e4391d`: collection manifest and failed worker/aggregation receipts differ only at `environment/version`. `tools/verification_shards.py` also seals the workflow run and attempt.
+- Lesson: Pin one supported exact interpreter patch per platform consistently across collection, workers and aggregation. Preserve identity checks. Run a fresh complete workflow after repair; failed-job-only retries cannot reuse a collection sealed to an earlier attempt.

LESSONS/20261006T161805Z-TEST-SUITE-PERFORMANCE-001-author-measured-platform-weights.md

@@ -0,0 +1,9 @@
+# New tests need measured scheduling costs
+
+- ID: 20261006T161805Z-TEST-SUITE-PERFORMANCE-001-author-measured-platform-weights
+- Date: 2026-10-06T16:18:05Z
+- Task/role: TEST-SUITE-PERFORMANCE-001 / author
+- Status: confirmed
+- Observation: Equal one-second weights for new infrastructure cases substantially understated their complete setup/call/teardown costs. The hosted trial exceeded 300 seconds despite balanced historical weights.
+- Evidence: [Run 37491893447](https://github.com/david-hoffman/dphtools/actions/runs/37491893447) measured new distributed cases around 14–38 seconds on Linux; complete test windows were 337 seconds on macOS and 525 seconds on Windows. All original assertions were preserved.
+- Lesson: Refresh advisory scheduling weights from actual platform phase receipts and use the same selected weights at both partition levels. Weights never substitute for fresh test or coverage evidence.

LESSONS/20261006T163753Z-TEST-SUITE-PERFORMANCE-001-author-fixed-interpreter-patch.md

@@ -0,0 +1,9 @@
+# Floating interpreter selectors can violate a sealed distributed runtime
+
+- ID: 20261006T163753Z-TEST-SUITE-PERFORMANCE-001-author-fixed-interpreter-patch
+- Date: 2026-10-06T16:37:53Z
+- Task/role: TEST-SUITE-PERFORMANCE-001 / author
+- Status: confirmed
+- Observation: One Linux shard selected Python 3.10.22 after collection selected 3.10.21; other platforms used 3.10.11. The portable identity gate rejected the Linux mismatch before tests.
+- Evidence: CI run 37494868812, Linux shard 3 job 112378218979, retained collection/shard checks and shard-inputs.log. https://github.com/david-hoffman/dphtools/actions/runs/37494868812
+- Lesson: Pin one exact interpreter patch across collection, execution and aggregation. Rerun complete platform proofs after changing that pin; never weaken the identity comparison.

LESSONS/20261006T163753Z-TEST-SUITE-PERFORMANCE-001-author-shared-fixture-affinity.md

@@ -0,0 +1,9 @@
+# Measured test setup charges move between consumers
+
+- ID: 20261006T163753Z-TEST-SUITE-PERFORMANCE-001-author-shared-fixture-affinity
+- Date: 2026-10-06T16:37:53Z
+- Task/role: TEST-SUITE-PERFORMANCE-001 / author
+- Status: confirmed
+- Observation: After duration repartitioning, unchanged module fixtures added cold setup to nodes whose prior weights described warm consumers. The same three fixture groups were repeatedly rebuilt across workers.
+- Evidence: CI runs 37491893447 and 37494868812; test_verification_shards interface_shards and completed_shards groups and the imported completed_shards group in test_verification_distributed. The latter trial retains 108 combined source records for the advisory model. https://github.com/david-hoffman/dphtools/actions/runs/37494868812
+- Lesson: Keep known expensive shared-fixture consumers cohesive at both scheduling levels and charge setup once in the advisory model. Verify actual complete timing; a modeled worker maximum does not account for cross-job setup or launch skew.

LESSONS/20261006T164612Z-TEST-SUITE-PERFORMANCE-001-author-platform-patch-availability.md

@@ -0,0 +1,9 @@
+# Exact interpreter pins also need platform and architecture availability
+
+- ID: 20261006T164612Z-TEST-SUITE-PERFORMANCE-001-author-platform-patch-availability
+- Date: 2026-10-06T16:46:12Z
+- Task/role: TEST-SUITE-PERFORMANCE-001 / author
+- Status: confirmed
+- Observation: Python 3.10.21 existed for Linux x64 but the action had no build for macOS arm64 or Windows x64. A common pin stopped those platforms before collection.
+- Evidence: CI run https://github.com/david-hoffman/dphtools/actions/runs/37497382992, setup jobs 112385313467 and 112385313663; actions/python-versions versions-manifest.json SHA-256 183f5392fbc06cd7ee039f040d132500f28b305554790aeeebea242da4b98ea6 confirms 3.10.11 availability for those platforms/architectures.
+- Lesson: Verify exact version, platform and architecture together before pinning. Use the same available patch across each platform's collection, shards and aggregation; separate platform proofs need not share a patch version.

LESSONS/20261006T170831Z-TEST-SUITE-PERFORMANCE-001-author-pytest-cache-ownership.md

@@ -0,0 +1,9 @@
+# Pytest cache ownership belongs to each invocation
+
+- ID: 20261006T170831Z-TEST-SUITE-PERFORMANCE-001-author-pytest-cache-ownership
+- Date: 2026-10-06T17:08:31Z
+- Task/role: TEST-SUITE-PERFORMANCE-001 / author
+- Status: confirmed
+- Observation: Private worker basetemp and coverage did not isolate pytest's shared cache. A sibling run removed a transient pytest-cache-files directory during nested Windows collection.
+- Evidence: CI run 37498756376, Windows shard7 worker3 failed test_merge_rejects_changed_worker_execution[missing] before its intended mutation; the fresh independent review of58812a2 confirmed the ownership defect. Two real overlapping cache regressions pass locally after private cache_dir is supplied.
+- Lesson: Give collection and execution invocations absolute cache paths inside their private report directories; retain the cache provider and unchanged collection/error gates.