Skip to content

Add reproducible scientific and performance reports - #121

Merged
mveril merged 3 commits into
mainfrom
feature/scientific-validation-reports
Oct 12, 2026
Merged

mveril merged 3 commits into
mainfrom
feature/scientific-validation-reports

Conversation

@mveril

@mveril mveril commented Oct 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Add reproducible scientific and Criterion performance reports for RustiQ.
  • Require explicit convergence from both RustiQ and PySCF, enforce RustiQ JSON schema version 1 and required calculation sections, and compute each case's final status once after all validations. Any recorded error forces failure.
  • Share RustiQ JSON contract validation with the canonical PySCF pytest suite.
  • Verify frozen-core MP2 runfiles apply the requested frozen orbitals and memory budget under the MP2 configuration.
  • Reject stale Criterion estimates and clear old results before workflow benchmarks.
  • Keep performance scope and provenance explicit: configured workspace is not peak RSS, MP2 timings are synthetic, measurements are machine-dependent, and no cross-package performance comparison is made.

Regression coverage

Added focused cases for reference convergence failures and unknown status, missing/unsupported schema versions, malformed output sections, non-finite energies and spin values, recorded execution errors, combined HF/MP2/spin status, and failed-case retention in Markdown and JSON. Criterion tests cover report parsing and stale result rejection.

Verification on commit 0f75639abc77802b46e1db088413e46deb5bb7be

  • uv run --locked pytest tools/reference tools/benchmarks — 55 passed.
  • uv run --locked ruff check tools/reference tools/benchmarks — passed.
  • cargo fmt --all -- --check — passed.
  • cargo clippy --workspace --all-targets --all-features -- -D warnings — passed.
  • uv run --locked python tools/reference/report.py --output-dir target/scientific-report — generated a report with 16/16 cases passing; report metadata records this commit and a clean working tree.
  • Ran actual small Criterion suites after removing prior results:
    • RUSTIQ_BENCH_FILTER=h2-sto3g cargo bench --features bench-support --bench eri_timings
    • RUSTIQ_MP2_SIZES=12 RUSTIQ_MP2_MEMORY='32 MiB' cargo bench --features bench-support --bench mp2_timings
    • uv run --locked python tools/benchmarks/report.py --criterion-dir target/criterion --output-dir target/performance-report
    • The generated report parsed one ERI case (eri/h2-sto3g/sto-3g) and MP2 dimension 12 from Criterion 0.8.2. The means were approximately 0.985 ms for ERI, 2.581 ms for dense MP2, and 1.075 ms for blocked MP2. The configured MP2 workspace budget was 32 MiB. Both report metadata sections record this commit and a clean working tree; peak RSS was not measured.

Remaining limitations

The scientific suite covers selected geometries, basis sets, methods, and tolerances; agreement does not establish general chemical accuracy. Only small representative benchmarks were run locally; larger benchmark cases remain machine-dependent and were not measured here. The MP2 performance inputs are synthetic and do not represent complete molecular calculations. The workflow is opt-in via workflow_dispatch; its dispatched benchmark job was not run in this review.

Merge readiness

Pending: CI run 629 for this commit is still in progress. The requested local checks and report generation passed.

@mveril
mveril force-pushed the feature/scientific-validation-reports branch 2 times, most recently from 6db5bd8 to a2c4391 Compare October 12, 2026 11:08
@mveril
mveril force-pushed the feature/scientific-validation-reports branch from a2c4391 to 79e94ec Compare October 12, 2026 16:13
@mveril
mveril merged commit 0333174 into main Oct 12, 2026
17 checks passed
@mveril
mveril deleted the feature/scientific-validation-reports branch October 12, 2026 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant