Skip to content

Add full-atom NMR baselines and leakage-safe evaluation - #1

Closed
jgwy wants to merge 1 commit into
BojarLab:mainfrom
jgwy:codex/nmr-fixed-test-pilot
Closed

jgwy wants to merge 1 commit into
BojarLab:mainfrom
jgwy:codex/nmr-fixed-test-pilot

Conversation

@jgwy

@jgwy jgwy commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a full-atom, target-masked NMR data contract for the updated bundled dataset
  • add atom-level GCN, SchNet, DimeNet, DimeNet++, and clean-room ComENet baselines
  • use covalent connectivity for GCN and full-atom 3D context for geometric models
  • add fixed-split evaluation plus leakage-safe, exact-IUPAC-grouped experimental CV
  • add resumable CPU/CUDA runners, strict run identities, frozen manifests, and audited result tables
  • cache fixed SchNet and ComENet geometry after verifying output and parameter-gradient equivalence

Evaluation protocol

The updated bundle contains 376,397 input atoms and 200,308 measured C/H targets
over 2,497 structures and 2,073 glycans. All atoms are model inputs; loss,
normalization, validation selection, and RMSE are restricted to is_target.

  • Combined and simulated-only results use the bundled fixed test split.
  • Experimental-only results are pooled OOF predictions from five folds over all
    189 experimental structures (147 exact-IUPAC groups; 5,522 targets).
  • For test fold k, fold (k + 1) mod 5 is validation.
  • Simulated counterparts of validation and test glycans are removed from training.
  • Target normalization is fitted on training target atoms only.
  • Fixed-test and OOF predictions are frozen; no post-test retuning was performed.

Results (atom-micro RMSE, ppm)

Model 1H Comb. 1H Exp. 1H Sim. 13C Comb. 13C Exp. 13C Sim.
Mean 0.573326 0.557806 0.573745 20.215230 22.086568 20.133200
Median 0.573261 0.579421 0.572993 20.542083 22.201387 20.429581
GCN 0.228425 0.384843 0.225961 2.411853 5.116724 2.380072
SchNet 0.139652 0.183659 0.139899 1.725902 3.541742 1.689782
DimeNet 0.131970 0.200244 0.132370 1.758265 4.536187 1.734080
DimeNet++ 0.137637 0.187826 0.137963 1.655883 3.462201 1.620200
ComENet 0.572932 0.510402 0.573282 20.196897 20.052519 20.113191

Validation

  • independent final audit: status=pass
  • exact identities for 20,160 fixed-test targets and 5,522 experimental OOF targets
  • no duplicate/missing target identities; all predictions finite
  • zero exact-IUPAC overlap across fold train/validation/test partitions
  • held-out simulated counterparts excluded; train-only normalization reconstructed
  • committed result CSV/JSON files match the frozen remote artifacts byte-for-byte
  • full remote repository suite: 245 passed
  • NMR-focused remote suite: 233 passed
  • modified Python files: Ruff check and format pass; remote compileall passes

Large checkpoints and per-atom predictions are intentionally not committed.

@jgwy
jgwy force-pushed the codex/nmr-fixed-test-pilot branch from 6a8eaa6 to fb43b3d Compare August 11, 2026 08:29
@jgwy jgwy closed this Aug 11, 2026
@jgwy
jgwy deleted the codex/nmr-fixed-test-pilot branch August 11, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant