diff --git a/research/delayed_evidence_invalidation/AMENDMENT_0.1.1.md b/research/delayed_evidence_invalidation/AMENDMENT_0.1.1.md new file mode 100644 index 00000000..2e69b2f3 --- /dev/null +++ b/research/delayed_evidence_invalidation/AMENDMENT_0.1.1.md @@ -0,0 +1,95 @@ +# Protocol Amendment v0.1.1 — Drift onset versus scientific invalidity + +**Date:** 2026-10-08 +**Timing:** before implementation of the campaign simulator and before inspection of any primary comparative outcome +**Supersedes:** specific validity semantics in PREREGISTRATION v0.1 Sections 8, 9, 15, and 17 where “drift onset” could be read as equivalent to “scientific invalidity onset” + +## Reason for amendment + +Gradual physical drift and scientific invalidity are distinct events. + +A measurement process may begin drifting while remaining inside an application-validity tolerance and while scheduled check standards still satisfy the registered control rule. Treating every observation after the first nonzero physical drift as invalid would make the registered suspect interval logically inconsistent: an “in-control” check could occur after physical drift began. + +The correction below separates: + +1. **physical drift onset** — the first experiment whose simulated instrument parameter differs from its baseline; and +2. **scientific invalidity onset** — the first experiment whose systematic measurement bias exceeds a predeclared application-validity tolerance. + +This amendment changes no observed results and was made before the campaign implementation existed. + +## Effective validity rule + +Let the systematic pH reporting bias at experiment (t) be + +[ +b_{mathrm{sys}}(t)= +E[hat{pH}_tmid pH_t^*,M_t]-pH_t^*, +] + +where the expectation removes zero-mean measurement noise while retaining the current metrological state. + +For the primary v0.1.1 benchmark, define the application-validity tolerance + +[ +delta_{mathrm{app}} = 1.0,hatsigma_0. +] + +The hidden oracle defines the **scientific invalidity onset** + +[ +k_{mathrm{invalid}} += +min{tge k_{mathrm{drift}}: +|b_{mathrm{sys}}(t)|>delta_{mathrm{app}}}. +] + +Evidence before (k_{mathrm{invalid}}) is not labelled invalid solely because nonzero drift has begun. + +## Detection and suspect interval + +The registered check-standard rule remains unchanged: + +[ +|ar y_c-7.00|>3hatsigma_0/sqrt{5}. +] + +The operational suspect interval supplied to non-oracle recovery methods remains + +[ +(t_{mathrm{last in control}},t_{mathrm{detect}}]. +] + +It should be described as an **operational bracketing interval**, not as proof of the true physical drift onset. + +Under the primary parameters, the application-validity tolerance is lower than the check-mean alarm threshold, so delayed invalidity is possible: measurements may become scientifically invalid before the next scheduled check declares the process out of control. + +## Evaluation changes + +Primary hidden-oracle validity labels use (k_{mathrm{invalid}}), not (k_{mathrm{drift}}). + +Report both: + +- physical-drift detection latency: (t_{mathrm{detect}}-k_{mathrm{drift}}); +- scientific-invalidity detection latency: (t_{mathrm{detect}}-k_{mathrm{invalid}}). + +The primary count of “invalid observations admitted before alarm” begins at (k_{mathrm{invalid}}). + +Validity-classification accuracy is scored against invalidity labels derived from (delta_{mathrm{app}}). + +Selection-influence tracing may begin from evidence labelled invalid under this rule; physical drift alone does not initiate invalidation propagation. + +## Required sensitivity analysis + +Before broad claims, repeat the primary comparison for: + +[ +delta_{mathrm{app}}/hatsigma_0 in {0.5,1.0,2.0}. +] + +The 1.0 condition remains primary. + +If the main conclusion depends qualitatively on the 1.0 choice, the paper must narrow the claim accordingly. + +## Claim discipline + +This benchmark tolerance is a study parameter, not a universal pH-metrology criterion and not a claim about acceptable error in any production laboratory. diff --git a/research/delayed_evidence_invalidation/AMENDMENT_0.1.2.md b/research/delayed_evidence_invalidation/AMENDMENT_0.1.2.md new file mode 100644 index 00000000..10e1d470 --- /dev/null +++ b/research/delayed_evidence_invalidation/AMENDMENT_0.1.2.md @@ -0,0 +1,72 @@ +# Protocol Amendment v0.1.2 — Indexing, event ordering, drift sign, and false alarms + +**Date:** 2026-10-08 +**Timing:** before implementation of the campaign simulator and before inspection of primary comparative outcomes +**Builds on:** v0.1.1 + +This amendment freezes implementation degrees of freedom that were left implicit in v0.1/v0.1.1. + +## Experiment indexing + +Scientific experiments use **one-based** indices: + +[ +tin{1,ldots,240}. +] + +The initial design occupies experiments 1–16. Adaptive selection begins at experiment 17. + +The hidden physical-drift onset is sampled uniformly from the integer set + +[ +{60,ldots,120}. +] + +The onset experiment itself receives the first nonzero drift increment. + +## Check-standard ordering + +A scheduled check-standard event occurs immediately **after** scientific experiments + +[ +20,40,60,ldots,240. +] + +The check uses the same current instrument/metrological state as the scientific measurement just completed at that experiment index. It receives a separate measurement-noise draw. + +Thus a check at experiment 80 summarizes instrument state after the 80th scientific experiment; it does not alter the scientific-experiment count. + +## Primary drift direction + +The primary offset-drift family uses positive physical offset drift: + +[ +Delta_{mathrm{offset}}(t)>0. +] + +Under the LADS-aligned stale-calibration algebra this produces a negative systematic reported-pH bias. + +A sign-reversal ablation with otherwise matched standardized magnitude is required before making sign-invariant claims. + +## False-positive control alarms + +A check-standard alarm before the hidden scientific-invalidity onset is a **false-positive control alarm** for purposes of this benchmark. + +Such runs must: + +1. remain in the released campaign corpus; +2. be counted in the false-alarm outcome; +3. never be silently regenerated with a different seed. + +The primary delayed-invalidation recovery estimand is evaluated on campaigns in which scientific invalidity occurs before the first control alarm. This conditioning rule is fixed before results and must be reported with the unconditional alarm rate and the fraction of campaigns entering the primary recovery estimand. + +A sensitivity analysis must also report all-seed operational cost so that a method is not rewarded for expensive reactions to false alarms. + +## Additional required sensitivity analysis + +Add: + +- drift-sign reversal; +- false-positive alarm cost across all seeds. + +No production-metrology claim follows from these benchmark choices. diff --git a/research/delayed_evidence_invalidation/PREREGISTRATION.md b/research/delayed_evidence_invalidation/PREREGISTRATION.md new file mode 100644 index 00000000..cf0c07b2 --- /dev/null +++ b/research/delayed_evidence_invalidation/PREREGISTRATION.md @@ -0,0 +1,495 @@ +# Preregistration v0.1 — Delayed Evidence Invalidation and Scientific-State Recovery + +**Registration status:** frozen before implementation of the new benchmark task and before inspection of primary comparative outcomes +**Frozen on:** 2026-10-08 +**Repository base:** `fraware/LabTrust-Gym@7e361583fb3e8ca4f482b072f4ac190e30cc69a5` +**Study branch:** `research/delayed-evidence-invalidation-v0` + +## 1. Research question + +A continuously adaptive laboratory repeatedly executes the loop + +[ +B_t ightarrow a_t ightarrow e_t ightarrow B_{t+1}, +] + +where (B_t) is scientific state, (a_t) is the next experiment, and (e_t) is newly generated evidence. + +This study asks: + +> When evidence (e_k), previously admitted and used to update scientific state, is invalidated only at a later time ( au>k), which downstream scientific objects remain valid, which require recomputation or qualification, and which recovery strategy best restores a trustworthy scientific state while preserving useful work? + +The first experimental family uses delayed discovery of calibration drift in a simulated pH measurement process. Detection of loss of measurement control is treated as an external metrology event. The contribution under test begins after that event: propagation, classification, and recovery. + +## 2. Prior-art boundary and non-novel components + +The study does **not** claim novelty for: + +- laboratory measurement assurance, check standards, control charts, or calibration-drift detection; +- scientific workflow provenance or generic dependency graphs; +- claim/evidence lineage; +- machine unlearning or the general observation that deleting stored data does not erase learned influence; +- downstream influence in sequential decision systems; +- robust Bayesian optimization under corrupted observations; +- dynamic safety cases or runtime assurance; +- laboratory device interoperability. + +Relevant adjacent work includes NIST measurement-assurance practice; ASTM D6299/E3500 statistical quality-control practice; W3C PROV and scientific-workflow provenance; robust Bayesian optimisation under corruptions; machine unlearning; offline-bandit unlearning; 2026 work on downstream “influence echoes” in self-improving agent networks; dynamic safety assurance; and OPC UA LADS. + +The prospective contribution is narrower: + +> Scientific invalidation in an adaptive physical experiment creates semantically distinct dependency classes—measurement validity, physical/sample lineage, model update, claim support, and experiment-selection dependence. Recovery should reason over those classes separately instead of treating every descendant as equally invalid. + +This statement is a hypothesis to test, not a priority claim. + +## 3. Formal state + +At experiment (t), define + +[ +S_t=(P_t,M_t,E_t,B_t,Pi_t), +] + +where: + +- (P_t): physical/sample state and retained-material inventory; +- (M_t): metrological state, including instrument identity, calibration state, and validity envelope; +- (E_t): admitted evidence and its validity status; +- (B_t): scientific model/belief state; +- (Pi_t): experiment-selection state/policy. + +A campaign also maintains a typed directed influence graph + +[ +G_t=(V,mathcal E). +] + +Nodes may represent samples, measurements, raw signals, calibration states, derived values, model versions, experiment decisions, executed experiments, and claims. + +Frozen edge types for v0.1: + +- `MEASURED_UNDER`: measurement depends on instrument/calibration state; +- `DERIVED_FROM`: derived scientific object depends computationally or analytically on another object; +- `UPDATES`: evidence contributed to a model/belief-state update; +- `SELECTED_BY`: experiment choice was produced by a model/policy state; +- `SUPPORTS`: evidence/model supports a scientific claim; +- `EXECUTED_ON`: action or measurement applies to a specific physical sample/material lineage. + +No edge type may be added to improve the primary result after primary outcomes are inspected. New types require a versioned protocol revision or secondary analysis. + +## 4. Evidence validity states + +Each evidence object carries a time-dependent state + +[ +v_i(t)in +{ + exttt{provisional}, + exttt{admitted}, + exttt{quarantined}, + exttt{invalidated}, + exttt{superseded} +}. +] + +An invalidation record is + +[ +I=(e_i, au,r,s,mathcal D), +] + +with discovery time ( au), reason (r), affected validity scope (s), and supporting invalidation evidence (mathcal D). + +The primary implementation must distinguish: + +### Validity closure + +(C_V(e)): descendants whose warrant or numerical value depends on invalid evidence through `MEASURED_UNDER`, `DERIVED_FROM`, `UPDATES`, or `SUPPORTS`. + +### Selection-influence closure + +(C_Pi(e)): experiments or artifacts whose existence/selection depends on contaminated policy/model states through `SELECTED_BY`. + +The two closures are evaluated separately. + +## 5. Descendant classification + +Every downstream object receives exactly one primary recovery disposition: + +- `INVALID`: its own scientific warrant directly depends on invalid evidence and no registered repair is available; +- `RECOMPUTE`: its source evidence remains available in a form that supports deterministic or registered recomputation; +- `SELECTION_TAINTED`: its creation/selection was influenced by invalid scientific state, while its own physical execution or measurement remains independently admissible; +- `UNAFFECTED`: no material dependency on the invalidated evidence exists. + +If more than one label appears applicable, severity order is: + +[ + exttt{INVALID} > + exttt{RECOMPUTE} > + exttt{SELECTION_TAINTED} > + exttt{UNAFFECTED}. +] + +This rule is frozen for v0.1. + +## 6. Experimental substrate + +### 6.1 Instrument layer + +Primary instrument semantics follow the open OPC UA LADS reference pH-meter simulator. The simulator exposes: + +- a Nernst-equation signal model; +- temperature compensation; +- pH slope and offset; +- raw signal; +- calibration values and calibration methods; +- simulated deliberate slope/offset detuning. + +The benchmark integration may reproduce these semantics in Python for deterministic offline runs. If the TypeScript LADS server is used directly, the study must record its commit and preserve the same benchmark-level state variables. + +### 6.2 Scientific response surface + +The first study uses a synthetic, deterministic formulation-response surface. This surface is intentionally a benchmark function, not a claim of chemical realism. + +Input: + +[ +xin[0,1]. +] + +True property: + +[ +pH^*(x)=7.0+1.20sin(2pi x)+0.35cos(6pi x)+0.25x. +] + +Scientific objective: + +[ +u(x)=-(pH^*(x)-7.40)^2. +] + +Measurement noise is introduced only through the instrument model. + +The benchmark must expose the latent (pH^*(x)) only to the hidden adjudicator, never to the optimizer or recovery policy. + +### 6.3 Adaptive policy + +Primary policy: Gaussian-process upper-confidence-bound Bayesian optimization. + +If the repository lacks an acceptable GP implementation without adding a heavy core dependency, the benchmark may place the BO implementation behind a study-only optional dependency. The exact package/version and kernel are frozen before primary campaign execution. + +A robust-BO comparator must be included. Preferred comparator: RCGP-UCB or the closest reproducible public implementation from Ezzerg, Bogunovic, and Knoblauch (ICML 2026). If exact reuse proves impractical, the deviation must be documented before results are inspected. + +### 6.4 Campaign horizon + +- total experiments: (T=240); +- initial non-adaptive design: 16 space-filling points; +- adaptive experiments: 224; +- one retained physical sample record per experiment; +- check-standard event every 20 experiment slots; +- primary reference check: pH 7.00 at fixed temperature. + +The check-standard operations are instrumentation events and do not count toward the 240 scientific experiments. + +## 7. Calibration-only pre-study + +Before primary campaign generation, run a calibration-only noise characterization at pH 7.00 with no drift. + +- readings: 1,000; +- fixed calibration-study seed: 20261008; +- result: (hatsigma_0), the baseline standard deviation of the reported pH measurement. + +No adaptive scientific campaign data are included in this estimate. + +All primary drift magnitudes are expressed relative to (hatsigma_0), so the protocol remains portable across simulator implementations. + +## 8. Drift process + +Primary failure: gradual offset drift. + +For each campaign seed: + +1. drift onset (k) is sampled uniformly from experiment indices 60 through 120 using a hidden failure RNG stream; +2. after onset, offset grows linearly; +3. drift rate is drawn from the frozen set + [ + {0.075,0.125,0.200}hatsigma_0 + ] + per scientific experiment; +4. the optimizer does not observe onset or drift parameters. + +Secondary ablation: slope drift with matched standardized effect size. + +No combined slope+offset result may replace the offset-drift primary result. + +## 9. Measurement-control rule + +At each scheduled check-standard event, obtain five repeated readings. + +Let (ar y_c) be the mean check reading. Trigger an out-of-control event if + +[ +|ar y_c-7.00| > 3hatsigma_0/sqrt{5}. +] + +The immediately preceding check event that satisfied the control rule is the last-known-in-control checkpoint. + +Primary suspect interval: + +[ +(t_{mathrm{last good}},t_{mathrm{detect}}]. +] + +This deliberately conservative interval is supplied to recovery methods that do not estimate a more precise onset. + +The hidden oracle retains the true onset for evaluation only. + +## 10. Stored-sample assumption + +Primary study assumes physical samples remain available and stable for remeasurement through the campaign horizon. + +This assumption is explicit because it creates a scientifically meaningful recovery action: remeasure a previously generated sample after restoring instrument control. + +Secondary ablation will impose sample expiry windows. No general claim about recoverability is permitted without reporting this ablation. + +## 11. Recovery conditions + +### B0 — no retroactive recovery + +The control alarm affects future operation only. Previously admitted evidence and model state remain unchanged. + +### B1 — provenance flagging + +The suspect interval is marked in the execution record. No model reconstruction occurs. + +### B2 — QC + discard + retrain + +All pH observations in the conservative suspect interval are removed. The scientific model is retrained on retained observations. No stored sample is remeasured. + +### B3 — QC + blanket remeasurement + retrain + +All retained samples from the conservative suspect interval are remeasured after instrument recovery, then the model is rebuilt. + +This is a deliberately strong recovery baseline. + +### B4 — typed dependency-aware recovery + +The influence graph classifies affected artifacts, invalidates or recomputes dependent objects, and selects stored samples for remeasurement under the same recovery budget used by the matched B3 budgeted condition. + +B4 may use graph structure and model-state information available before hidden adjudication. It may not use true drift onset, true latent pH, or hidden labels. + +### B5 — robust BO + B2 recovery + +Robust BO operates during acquisition, followed by the same conservative discard/retrain procedure as B2 after control failure is detected. + +This tests whether robustness during optimization makes the recovery layer unnecessary. + +## 12. Recovery budgets + +Two analyses are preregistered. + +### Exact/upper-bound recovery + +B3 remeasures the full suspect interval. B4 may also use the full interval. This establishes the best attainable recovery under retained-sample access. + +### Budgeted recovery + +Remeasurement budgets are: + +[ +bin{0.10,0.25,0.50} +] + +of samples in the conservative suspect interval, rounded up. + +For B3 budgeted recovery, sample selection is uniform random with a dedicated RNG stream. + +For B4, the selection rule must be frozen before primary outcomes. The intended rule is greedy expected reduction in model-state discrepancy estimated from the observed model and influence graph. If implementation requires a different criterion, it must be versioned before outcome inspection. + +## 13. Hidden clean counterfactual + +For each campaign seed, run a clean campaign with identical initial design, policy hyperparameters, non-failure RNG stream, and no drift. + +Because adaptive action sequences diverge after corruption, the clean campaign is not treated as a pointwise paired trajectory after divergence. It is used as an adjudication reference. + +The latent response surface also permits direct evaluation against physical ground truth independently of the clean policy trajectory. + +## 14. Primary outcomes + +### O1 — post-recovery scientific-model error + +On a fixed dense evaluation grid (mathcal X_{eval}), measure normalized integrated squared prediction error against (pH^*(x)). + +Primary comparison: B4 versus B2 and B3. + +### O2 — retained scientifically valid work + +Fraction of valid physical experiments / revalidated measurements retained in the recovered campaign state. + +### O3 — recovery cost + +Count of remeasurements plus normalized computational reconstruction cost. Physical remeasurement count is reported separately and receives priority over compute in interpretation. + +### O4 — validity-classification accuracy + +Against hidden oracle dependencies, macro-F1 over `INVALID`, `RECOMPUTE`, `SELECTION_TAINTED`, and `UNAFFECTED`. + +No model-performance result substitutes for failure to classify dependencies correctly. + +## 15. Secondary outcomes + +- detection latency from true drift onset to control alarm; +- cumulative optimization regret; +- number of invalid observations admitted before alarm; +- validity-closure size; +- selection-influence-closure size; +- policy-path divergence from clean reference; +- number of scientific claims requiring withdrawal/recomputation; +- wall-clock recovery time; +- sample remeasurement efficiency; +- false invalidation of unaffected artifacts. + +The following programme constructs are exploratory until validated: + +- validity contamination depth; +- selection influence depth; +- scientific-state recovery fidelity; +- scientific waste. + +They must not be presented as established community metrics. + +## 16. Primary hypotheses + +### H1 — delayed invalidation has consequences beyond a QC alarm + +B0/B1 retain materially higher post-recovery model error or unsupported downstream state than B2–B4. + +### H2 — dependency semantics preserve useful science + +At matched model-error tolerance, B4 retains more scientifically valid work or requires fewer remeasurements than full rollback / blanket recovery. + +### H3 — robust acquisition does not eliminate retroactive recovery + +B5 reduces sensitivity to corrupted observations but does not, by itself, resolve provenance/claim invalidation or recover the correct dependency state after a late control failure. + +### H4 — selection dependence differs from validity dependence + +There exist campaigns containing artifacts correctly classified as `SELECTION_TAINTED` but not `INVALID`. + +Failure to produce such cases in the primary family weakens the central semantic claim and requires either a second failure family or narrowing of the paper. + +## 17. Statistical design + +Primary Monte Carlo unit: campaign seed. + +Initial target: 200 seeds per drift-rate condition, for 600 campaigns per method before recovery-budget expansion. + +All methods operate on the same campaign seed and hidden failure realization where their acquisition semantics permit paired comparison. + +Primary uncertainty reporting: + +- bootstrap 95% confidence intervals over campaign seeds; +- paired bootstrap differences where trajectories share the same pre-divergence seed; +- effect sizes alongside p-values; +- full empirical distributions for heavy-tailed outcomes. + +No claim rests solely on statistical significance. + +A calibration run with 20 seeds per condition is permitted solely to verify code, runtime, nondegenerate drift detection, and absence of ceiling/floor effects. Those seeds are permanently excluded from primary analysis. + +Any sample-size change after the 20-seed calibration must be justified from runtime/variance estimates and frozen before primary results are inspected. + +## 18. Ablations required before broad claims + +1. slope drift instead of offset drift; +2. different check intervals: 10, 20, 40; +3. stored-sample expiry / unavailable sample recovery; +4. different BO acquisition policy; +5. different latent response surface; +6. graph-edge removal, one type at a time; +7. incorrect or incomplete dependency metadata; +8. false-positive QC alarm; +9. multiple separate invalidation events. + +A paper may report fewer ablations only if its claims are narrowed accordingly. + +## 19. Kill criteria + +The richer architecture is not supported if any of the following survives adequate testing: + +1. B2 (QC + discard + retrain) matches B4 on model error, retained valid work, and recovery cost within practically negligible differences. +2. B3 blanket recovery dominates B4 across the recovery Pareto frontier. +3. validity and selection closures collapse to the same set in realistic benchmark variants. +4. B4's advantage depends on hidden information unavailable to a deployed system. +5. the result disappears under a second latent response surface or slope-drift family. +6. results depend on benchmark-specific identifiers or hand-coded knowledge of the injected failure. +7. a simpler generic provenance traversal reproduces B4 behavior with equivalent effort and outputs. +8. the LADS-aligned benchmark semantics materially differ from the reference implementation in ways that drive the result. + +A null result must be reported as a programme-narrowing result, not repaired by weakening baselines after inspection. + +## 20. Reproducibility requirements + +Every released run must record: + +- repository commit; +- study-contract digest; +- dependency versions; +- Python version; +- seed streams; +- latent-surface version; +- instrument-model version; +- drift realization; +- check-standard observations; +- recovery method configuration; +- influence graph before and after invalidation; +- raw and derived metrics. + +Primary figures and tables must rebuild from released run artifacts. + +The existing LabTrust paper-claims regression and release-pack machinery should be reused where practical. + +## 21. External-validity ladder + +Claims are tiered. + +### Tier S0 — benchmark semantics + +The implementation behaves as registered inside the deterministic benchmark. + +### Tier S1 — standards-aligned simulation + +Results reproduce through the OPC UA LADS-compatible instrument semantics. + +### Tier S2 — second failure family / second substrate + +The recovery semantics transfer to a non-metrological invalidation mechanism or another open autonomous-lab stack. + +### Tier S3 — real laboratory shadow evaluation + +The system processes authentic run artifacts without controlling live scientific actions. + +### Tier S4 — prospective bounded pilot + +Recovery outputs influence a real laboratory workflow under an independently approved operating envelope. + +The first paper should claim no higher tier than the evidence actually achieved. + +## 22. Implementation ownership + +- **LabTrust-Gym:** campaign simulator, hidden oracle, failures, stored samples, benchmark runner, metrics. +- **EnvAssure:** metrological/environment validity semantics and validity-envelope transitions. +- **Scientific Memory:** persistent claim/evidence/model dependency records and typed influence graph. +- **AKTA:** consumes scientific-state disposition when deciding whether future action is admissible. +- **VerifierLab:** adversarial/repair campaign methodology and comparative qualification. + +No new repository is authorized by this preregistration. + +## 23. Planned first paper claim + +The aspirational claim to test is: + +> Delayed invalidation in adaptive autonomous experimentation creates scientifically distinct validity and selection dependencies. A typed, dependency-aware recovery process can restore scientific state while preserving more legitimate experimental work than coarse rollback or provenance/QC alone. + +This sentence is not a supported result until the registered comparisons succeed. diff --git a/research/delayed_evidence_invalidation/README.md b/research/delayed_evidence_invalidation/README.md new file mode 100644 index 00000000..29a5140f --- /dev/null +++ b/research/delayed_evidence_invalidation/README.md @@ -0,0 +1,38 @@ +# Delayed Evidence Invalidation Study + +**Status:** preregistration v0.1.2 effective / no primary results inspected +**Branch:** `research/delayed-evidence-invalidation-v0` +**Base commit:** `7e361583fb3e8ca4f482b072f4ac190e30cc69a5` + +This directory freezes the first study in the long-horizon scientific-state assurance programme. + +## Research object + +The study asks what a continuously adaptive laboratory should do when evidence previously admitted into its scientific state is later found invalid. + +The central distinction is between: + +- **validity dependence**: an artifact, model, or claim depends on evidence whose own validity has failed; and +- **selection dependence**: an experiment was selected by a scientific state influenced by invalid evidence, although the experiment or later measurement may remain scientifically usable. + +The first benchmark uses delayed calibration-drift discovery in an OPC UA LADS-aligned pH-measurement loop. Metrology owns detection that the measurement process has left control. The study begins at that detection event and evaluates propagation and recovery. + +## Frozen artifacts + +- `PREREGISTRATION.md` — original v0.1 hypotheses, prior-art boundary, design, baselines, outcomes, analysis, ablations, and kill criteria. +- `AMENDMENT_0.1.1.md` — pre-outcome amendment separating physical drift onset from scientific invalidity onset. +- `AMENDMENT_0.1.2.md` — pre-outcome amendment freezing indexing, event ordering, drift sign, and false-alarm handling. +- `study_contract.v0.1.yaml` — historical machine-readable v0.1 contract. +- `study_contract.v0.1.1.yaml` — historical amended contract. +- `study_contract.v0.1.2.yaml` — **effective** machine-readable contract for implementation and analysis. +- `REFERENCES.md` — literature and standards that define the prior-art boundary. + +## Claim discipline + +This study is a simulation / reference-integration study. It does not establish production laboratory safety, clinical validity, general real-world failure rates, or a universal theory of scientific recovery. Those claims require external laboratory data and independent replication. + +No implementation decision may silently weaken a frozen baseline, outcome definition, or kill criterion. Any protocol change after primary outcomes are inspected must be versioned and reported as post-registration. + +## Effective protocol + +Implementation and primary analysis must follow `study_contract.v0.1.2.yaml` together with `AMENDMENT_0.1.1.md` and `AMENDMENT_0.1.2.md`. Earlier versions remain immutable historical records. diff --git a/research/delayed_evidence_invalidation/REFERENCES.md b/research/delayed_evidence_invalidation/REFERENCES.md new file mode 100644 index 00000000..a27a27f2 --- /dev/null +++ b/research/delayed_evidence_invalidation/REFERENCES.md @@ -0,0 +1,83 @@ +# Prior-art and standards boundary + +This file records sources that constrain novelty and benchmark design. It is not a claim that every source directly studies delayed evidence invalidation in autonomous laboratories. + +## Autonomous laboratories and scientific-agent assurance + +1. Canty, R. B.; Abolhasani, M. **The past, present and future of self-driving laboratories.** *Nature Reviews Chemistry* 10, 523–537 (2026). DOI: 10.1038/s41570-026-00847-2. + Relevance: scalability, generalizability, and provenance-complete experimentation as interdependent SDL requirements. + +2. Chen, L.; Li, X.; Lin, Q.; et al. **Self-driving laboratories need an autonomy safety harness.** *Nature Synthesis* (2026). DOI: 10.1038/s44160-026-01120-6. + Relevance: AI intent, executable experiments, monitored actions, and trustworthy downstream evidence. + +3. Yin, X.; Du, M.; Prince, M. H.; Cherukara, M. J. **Artifact-centered Claim-aware Observability for Autonomous Scientific Agents.** arXiv:2608.18312 (2026). + Relevance: claim-aware artifact lineage; prevents this study from claiming generic claim/evidence provenance as novel. + +4. Bi, J.; Pinilla, S.; Zhu, C. **Certifying when decision-time information justifies adaptive experimentation.** arXiv:2607.27651 (2026). + Relevance: authorization is a distinct layer in adaptive science; prevents conflation of recovery with initial authorization. + +5. Alnasir, J. J. **AI-Driven Scientific Computing Workflows: A Systems Review of Orchestration, Execution, Reproducibility and Provenance.** arXiv:2609.21162 (2026). + Relevance: identifies state-aware recovery, model-mediated decision provenance, and reproducible adaptive execution as open systems problems. + +## Metrology and quality control + +6. National Institute of Standards and Technology. **Measurement Assurance Programs / Calibration Policies** and **Measurement Assurance Basics**. + Relevance: periodic check standards, uncertainty estimation, statistical control, and continuous monitoring of measurement-process quality. + +7. ASTM D6299-26. **Standard Practice for Applying Statistical Quality Assurance and Control Charting Techniques to Evaluate Analytical Measurement System Performance.** + Relevance: ongoing monitoring of precision/bias and analytical measurement system state of control. + +8. ASTM E3500-25. **Standard Practice for Quality Control of Routine Testing in a Laboratory.** + Relevance: control samples, drift detection, out-of-control action plans, and acceptance/evaluation of sample results. + +## Robust adaptive optimization and inference + +9. Ezzerg, A.; Bogunovic, I.; Knoblauch, J. **Robust Bayesian Optimisation with Unbounded Corruptions.** ICML 2026, PMLR 306:28528–28565. + Relevance: strong comparator for acquisition robustness under corrupted observations; robust acquisition does not itself specify scientific invalidation semantics. + +10. Adaptive-experiment and sequential-inference literature reviewed in: + **The Blessings and Curses of Adaptivity in Sequential Experiments** (Annual Review of Statistics and Its Application). + Relevance: adaptively collected data require explicit inferential treatment; later observations are not automatically equivalent to a fixed-design sample. + +## Unlearning and downstream influence + +11. Bourtoule, L.; et al. **Machine Unlearning.** arXiv:1912.03817. + Relevance: data influence persists in trained models after storage deletion; retraining/structured recovery baselines. + +12. Ye, Z.; Wang, R.; Wang, X.; Liu, X.; Li, S.; Hajiesmaili, M. **Unlearning Offline Stochastic Multi-Armed Bandits.** arXiv:2605.00638 (2026). + Relevance: unlearning in sequential decision-making and rollback/utility trade-offs. The present study must not claim that unlearning in sequential decision systems is new. + +13. Ding, Z.; Huang, J.; Dong, L. **When Unlearning Fails: Reliable Data Deletion under Post-Training in Agent Networks.** arXiv:2607.28829 (2026). + Relevance: especially close adjacent work. It identifies downstream “influence echoes” when target data have already shaped later retained trajectories. This prevents any priority claim for the generic idea that corrupted/deleted data influence later collected trajectories. + +14. Nguyen, T. T.; et al. **A Survey of Machine Unlearning.** *ACM Transactions on Intelligent Systems and Technology* (2025). DOI: 10.1145/3749987. + Relevance: unlearning scope, streaming removal, retained utility, and exact-versus-approximate removal. + +## Runtime/dynamic assurance + +15. **Safe autonomous systems in a changing world: Operationalising dynamic safety cases.** *Safety Science* 191, 106965 (2025). DOI: 10.1016/j.ssci.2025.106965. + Relevance: monitorability, assessability, and updateability of assurance arguments at runtime. + +16. Herd, B.; Kelly, J.; Heinemann, C.; Zacchi, J.-V. **A Subjective Logic-based method for runtime confidence updates in safety arguments.** arXiv:2605.22530 (2026). + Relevance: runtime evidence updates confidence in assurance claims; the present work must distinguish scientific validity propagation from generic assurance-case updating. + +## Laboratory interoperability substrate + +17. OPC Foundation / SPECTARIS. **OPC UA Laboratory and Analytical Device Standard (LADS), Companion Specification 30500.** + Relevance: standards-aligned laboratory device state, measurements, alarms/events, methods, and result semantics. + +18. `opcua-lads/lads-server-collection`, pH-meter reference simulator (2025–2026). + Relevance: Nernst-equation signal model, temperature compensation, slope/offset detuning, raw signal, calibration methods, AFO metadata, and ASM result payloads. The repository explicitly describes the servers as proof-of-concept simulators for demos/integration/reference use. + +## Novelty discipline + +The study should avoid priority claims until a systematic literature review and external expert review are complete. + +The most defensible provisional distinction is scientific-semantic: + +- metrological invalidity is not the same relation as computational derivation; +- claim invalidity is not the same relation as experiment-selection influence; +- an experiment selected by a contaminated model may still produce a physically valid artifact or later measurement; +- recovery should preserve these distinctions and be evaluated against strong rollback, retraining, robust-optimization, and blanket-remeasurement baselines. + +The July 2026 “influence echo” work in agent networks is the closest conceptual warning found so far. Any paper must cite and distinguish it explicitly. diff --git a/research/delayed_evidence_invalidation/study_contract.v0.1.1.yaml b/research/delayed_evidence_invalidation/study_contract.v0.1.1.yaml new file mode 100644 index 00000000..b34d1fa9 --- /dev/null +++ b/research/delayed_evidence_invalidation/study_contract.v0.1.1.yaml @@ -0,0 +1,284 @@ +schema_version: delayed_evidence_invalidation.v0.1.1 +status: preregistered_amended +frozen_date: "2026-10-08" +repository: + name: fraware/LabTrust-Gym + base_commit: 7e361583fb3e8ca4f482b072f4ac190e30cc69a5 + branch: research/delayed-evidence-invalidation-v0 + +study: + id: dei_ph_drift_v0_1_1 + title: Delayed Evidence Invalidation and Scientific-State Recovery + mode: simulation + primary_failure_family: metrological_delayed_invalidation + no_new_repository: true + +state_model: + components: + - physical_state + - metrological_state + - evidence_state + - scientific_model_state + - selection_policy_state + influence_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SELECTED_BY + - SUPPORTS + - EXECUTED_ON + validity_states: + - provisional + - admitted + - quarantined + - invalidated + - superseded + descendant_dispositions: + severity_order: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +substrate: + instrument_semantics: opcua_lads_ph_meter_reference + reference_repository: opcua-lads/lads-server-collection + reference_component: servers/lads-ph-meter + required_features: + - nernst_signal_model + - temperature_compensation + - slope + - offset + - raw_signal + - calibration_values + - calibration_methods + note: > + Python reimplementation is permitted for deterministic offline campaigns only if + benchmark-level semantics are checked against the reference implementation. + +scientific_surface: + domain: + x_min: 0.0 + x_max: 1.0 + true_property: + expression: "7.0 + 1.20*sin(2*pi*x) + 0.35*cos(6*pi*x) + 0.25*x" + hidden_from_policy: true + target_ph: 7.40 + utility: + expression: "-(true_ph - 7.40)^2" + realism_claim: false + +campaign: + scientific_experiments: 240 + initial_design_points: 16 + adaptive_experiments: 224 + retained_sample_per_experiment: true + check_interval_experiments: 20 + check_replicates: 5 + check_reference_ph: 7.00 + +noise_calibration: + seed: 20261008 + readings: 1000 + reference_ph: 7.00 + output: sigma0 + excluded_from_primary_analysis: true + +failure: + type: gradual_offset_drift + onset: + distribution: discrete_uniform + min_experiment: 60 + max_experiment: 120 + hidden_from_methods: true + standardized_rates_per_experiment: + - 0.075 + - 0.125 + - 0.200 + rate_units: sigma0 + secondary_failure: + type: slope_drift + match_standardized_effect: true + +application_validity: + primary_tolerance_sigma0: 1.0 + systematic_bias_definition: "E[reported_ph | true_ph, metrological_state] - true_ph" + scientific_invalidity_onset: "first t >= physical_drift_onset with abs(systematic_bias(t)) > tolerance" + physical_drift_alone_implies_invalidity: false + oracle_only: true + sensitivity_tolerance_sigma0: + - 0.5 + - 1.0 + - 2.0 + production_metrology_claim: false + +qc: + rule: shewhart_mean_threshold + threshold_expression: "abs(mean_check - 7.00) > 3*sigma0/sqrt(5)" + suspect_interval: "(last_in_control_check, detection_check]" + true_physical_drift_onset_available_to_hidden_adjudicator_only: true + true_scientific_invalidity_onset_available_to_hidden_adjudicator_only: true + suspect_interval_semantics: operational_bracketing_not_true_onset + +adaptive_policy: + primary: gp_ucb + exact_implementation: freeze_before_primary_runs + robust_comparator: + preferred: rcgp_ucb + reference: "Ezzerg, Bogunovic, Knoblauch, ICML 2026" + deviation_requires_preoutcome_protocol_update: true + +recovery_methods: + B0: + name: no_retroactive_recovery + retroactive_change: false + B1: + name: provenance_flagging + rebuild_model: false + B2: + name: qc_discard_retrain + discard_suspect_ph: true + remeasure_samples: false + B3: + name: qc_blanket_remeasure_retrain + exact_mode: + remeasure_fraction: 1.0 + budgeted_selection: uniform_random + B4: + name: typed_dependency_aware_recovery + hidden_oracle_access: false + exact_mode: + max_remeasure_fraction: 1.0 + budgeted_selection: + intended_rule: greedy_expected_model_discrepancy_reduction + freeze_before_primary_runs: true + B5: + name: robust_bo_plus_B2 + acquisition_robustness: true + recovery_semantics: B2 + +recovery_budgets: + fractions: + - 0.10 + - 0.25 + - 0.50 + rounding: ceil + +primary_outcomes: + O1: + name: post_recovery_model_error + measure: normalized_integrated_squared_error + reference: latent_true_surface + O2: + name: retained_valid_work + measure: fraction_valid_physical_experiments_or_revalidated_measurements_retained + O3: + name: recovery_cost + measures: + - remeasurement_count + - normalized_compute_cost + interpretation_priority: remeasurement_count + O4: + name: validity_classification_accuracy + measure: macro_f1 + classes: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +secondary_outcomes: + - physical_drift_detection_latency + - scientific_invalidity_detection_latency + - cumulative_optimization_regret + - invalid_observations_admitted_from_scientific_invalidity_onset + - validity_closure_size + - selection_influence_closure_size + - policy_path_divergence + - claims_withdrawn_or_recomputed + - wall_clock_recovery_time + - remeasurement_efficiency + - false_invalidation_count + +hypotheses: + H1: delayed_invalidity_requires_retroactive_recovery + H2: dependency_semantics_preserve_valid_science + H3: robust_acquisition_does_not_replace_recovery + H4: selection_dependence_differs_from_validity_dependence + +statistics: + monte_carlo_unit: campaign_seed + target_seeds_per_drift_rate_per_method: 200 + drift_rate_conditions: 3 + initial_total_campaigns_per_method_before_budget_expansion: 600 + calibration_seeds_per_condition: 20 + calibration_seeds_excluded_from_primary: true + confidence_intervals: + method: bootstrap + level: 0.95 + paired_bootstrap_when_valid: true + report_effect_sizes: true + significance_alone_sufficient: false + sample_size_change: + allowed_before_primary_outcome_inspection: true + requires_variance_or_runtime_justification: true + +required_ablations: + - slope_drift + - check_interval_10_20_40 + - sample_expiry + - alternate_acquisition_policy + - alternate_latent_surface + - edge_type_removal + - incomplete_dependency_metadata + - false_positive_qc_alarm + - multiple_invalidation_events + +kill_criteria: + - "B2 matches B4 on model error, retained valid work, and recovery cost within practically negligible differences." + - "B3 dominates B4 across the recovery Pareto frontier." + - "Validity and selection closures collapse to the same set in realistic variants." + - "B4 advantage depends on hidden information unavailable to deployed systems." + - "Effect disappears under a second response surface or slope-drift family." + - "Effect depends on benchmark-specific identifiers or hand-coded injected-failure knowledge." + - "A simpler generic provenance traversal reproduces B4 with equivalent outputs and effort." + - "LADS-aligned semantics differ from the reference implementation in result-driving ways." + +claim_tiers: + S0: benchmark_semantics + S1: standards_aligned_simulation + S2: second_failure_family_or_substrate + S3: real_lab_shadow_evaluation + S4: prospective_bounded_pilot + +ownership: + LabTrust-Gym: + - campaign_simulator + - hidden_oracle + - failure_injection + - stored_samples + - benchmark_runner + - metrics + EnvAssure: + - metrological_validity_semantics + - environment_validity_transitions + Scientific-Memory: + - typed_influence_graph + - evidence_invalidation_records + - claim_model_dependencies + AKTA: + - consume_scientific_state_disposition_for_action_admissibility + VerifierLab: + - comparative_campaign_methodology + - repair_and_recovery_evaluation + +amendments: + effective: + - file: AMENDMENT_0.1.1.md + reason: distinguish physical drift onset from scientific invalidity onset + timing: before_campaign_implementation_and_before_primary_outcomes + +result_blinding: + implementation_before_primary_results: required + protocol_changes_after_primary_results: + permitted_only_as: post_registration_analysis diff --git a/research/delayed_evidence_invalidation/study_contract.v0.1.2.yaml b/research/delayed_evidence_invalidation/study_contract.v0.1.2.yaml new file mode 100644 index 00000000..3dd6dfd4 --- /dev/null +++ b/research/delayed_evidence_invalidation/study_contract.v0.1.2.yaml @@ -0,0 +1,299 @@ +schema_version: delayed_evidence_invalidation.v0.1.2 +status: preregistered_amended +frozen_date: "2026-10-08" +repository: + name: fraware/LabTrust-Gym + base_commit: 7e361583fb3e8ca4f482b072f4ac190e30cc69a5 + branch: research/delayed-evidence-invalidation-v0 + +study: + id: dei_ph_drift_v0_1_2 + title: Delayed Evidence Invalidation and Scientific-State Recovery + mode: simulation + primary_failure_family: metrological_delayed_invalidation + no_new_repository: true + +state_model: + components: + - physical_state + - metrological_state + - evidence_state + - scientific_model_state + - selection_policy_state + influence_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SELECTED_BY + - SUPPORTS + - EXECUTED_ON + validity_states: + - provisional + - admitted + - quarantined + - invalidated + - superseded + descendant_dispositions: + severity_order: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +substrate: + instrument_semantics: opcua_lads_ph_meter_reference + reference_repository: opcua-lads/lads-server-collection + reference_component: servers/lads-ph-meter + required_features: + - nernst_signal_model + - temperature_compensation + - slope + - offset + - raw_signal + - calibration_values + - calibration_methods + note: > + Python reimplementation is permitted for deterministic offline campaigns only if + benchmark-level semantics are checked against the reference implementation. + +scientific_surface: + domain: + x_min: 0.0 + x_max: 1.0 + true_property: + expression: "7.0 + 1.20*sin(2*pi*x) + 0.35*cos(6*pi*x) + 0.25*x" + hidden_from_policy: true + target_ph: 7.40 + utility: + expression: "-(true_ph - 7.40)^2" + realism_claim: false + +campaign: + experiment_indexing: one_based + scientific_experiments: 240 + initial_design_points: 16 + adaptive_experiments: 224 + retained_sample_per_experiment: true + check_interval_experiments: 20 + check_replicates: 5 + check_reference_ph: 7.00 + check_ordering: immediately_after_scientific_experiment + check_uses_same_current_metrological_state: true + +noise_calibration: + seed: 20261008 + readings: 1000 + reference_ph: 7.00 + output: sigma0 + excluded_from_primary_analysis: true + +failure: + type: gradual_offset_drift + onset: + distribution: discrete_uniform + min_experiment: 60 + max_experiment: 120 + hidden_from_methods: true + primary_direction: positive_physical_offset + primary_reported_bias_direction: negative + standardized_rates_per_experiment: + - 0.075 + - 0.125 + - 0.200 + rate_units: sigma0 + secondary_failure: + type: slope_drift + match_standardized_effect: true + +application_validity: + primary_tolerance_sigma0: 1.0 + systematic_bias_definition: "E[reported_ph | true_ph, metrological_state] - true_ph" + scientific_invalidity_onset: "first t >= physical_drift_onset with abs(systematic_bias(t)) > tolerance" + physical_drift_alone_implies_invalidity: false + oracle_only: true + sensitivity_tolerance_sigma0: + - 0.5 + - 1.0 + - 2.0 + production_metrology_claim: false + +qc: + rule: shewhart_mean_threshold + threshold_expression: "abs(mean_check - 7.00) > 3*sigma0/sqrt(5)" + suspect_interval: "(last_in_control_check, detection_check]" + true_physical_drift_onset_available_to_hidden_adjudicator_only: true + true_scientific_invalidity_onset_available_to_hidden_adjudicator_only: true + suspect_interval_semantics: operational_bracketing_not_true_onset + false_positive_alarm_definition: alarm_before_scientific_invalidity_onset + false_positive_runs_retained: true + primary_recovery_estimand_condition: scientific_invalidity_precedes_first_alarm + report_unconditional_alarm_rate: true + report_fraction_entering_primary_estimand: true + +adaptive_policy: + primary: gp_ucb + exact_implementation: freeze_before_primary_runs + robust_comparator: + preferred: rcgp_ucb + reference: "Ezzerg, Bogunovic, Knoblauch, ICML 2026" + deviation_requires_preoutcome_protocol_update: true + +recovery_methods: + B0: + name: no_retroactive_recovery + retroactive_change: false + B1: + name: provenance_flagging + rebuild_model: false + B2: + name: qc_discard_retrain + discard_suspect_ph: true + remeasure_samples: false + B3: + name: qc_blanket_remeasure_retrain + exact_mode: + remeasure_fraction: 1.0 + budgeted_selection: uniform_random + B4: + name: typed_dependency_aware_recovery + hidden_oracle_access: false + exact_mode: + max_remeasure_fraction: 1.0 + budgeted_selection: + intended_rule: greedy_expected_model_discrepancy_reduction + freeze_before_primary_runs: true + B5: + name: robust_bo_plus_B2 + acquisition_robustness: true + recovery_semantics: B2 + +recovery_budgets: + fractions: + - 0.10 + - 0.25 + - 0.50 + rounding: ceil + +primary_outcomes: + O1: + name: post_recovery_model_error + measure: normalized_integrated_squared_error + reference: latent_true_surface + O2: + name: retained_valid_work + measure: fraction_valid_physical_experiments_or_revalidated_measurements_retained + O3: + name: recovery_cost + measures: + - remeasurement_count + - normalized_compute_cost + interpretation_priority: remeasurement_count + O4: + name: validity_classification_accuracy + measure: macro_f1 + classes: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +secondary_outcomes: + - physical_drift_detection_latency + - scientific_invalidity_detection_latency + - cumulative_optimization_regret + - invalid_observations_admitted_from_scientific_invalidity_onset + - validity_closure_size + - selection_influence_closure_size + - policy_path_divergence + - claims_withdrawn_or_recomputed + - wall_clock_recovery_time + - remeasurement_efficiency + - false_invalidation_count + +hypotheses: + H1: delayed_invalidity_requires_retroactive_recovery + H2: dependency_semantics_preserve_valid_science + H3: robust_acquisition_does_not_replace_recovery + H4: selection_dependence_differs_from_validity_dependence + +statistics: + monte_carlo_unit: campaign_seed + target_seeds_per_drift_rate_per_method: 200 + drift_rate_conditions: 3 + initial_total_campaigns_per_method_before_budget_expansion: 600 + calibration_seeds_per_condition: 20 + calibration_seeds_excluded_from_primary: true + confidence_intervals: + method: bootstrap + level: 0.95 + paired_bootstrap_when_valid: true + report_effect_sizes: true + significance_alone_sufficient: false + sample_size_change: + allowed_before_primary_outcome_inspection: true + requires_variance_or_runtime_justification: true + +required_ablations: + - slope_drift + - check_interval_10_20_40 + - sample_expiry + - alternate_acquisition_policy + - alternate_latent_surface + - edge_type_removal + - incomplete_dependency_metadata + - false_positive_qc_alarm + - multiple_invalidation_events + - drift_sign_reversal + - all_seed_false_alarm_operational_cost + +kill_criteria: + - "B2 matches B4 on model error, retained valid work, and recovery cost within practically negligible differences." + - "B3 dominates B4 across the recovery Pareto frontier." + - "Validity and selection closures collapse to the same set in realistic variants." + - "B4 advantage depends on hidden information unavailable to deployed systems." + - "Effect disappears under a second response surface or slope-drift family." + - "Effect depends on benchmark-specific identifiers or hand-coded injected-failure knowledge." + - "A simpler generic provenance traversal reproduces B4 with equivalent outputs and effort." + - "LADS-aligned semantics differ from the reference implementation in result-driving ways." + +claim_tiers: + S0: benchmark_semantics + S1: standards_aligned_simulation + S2: second_failure_family_or_substrate + S3: real_lab_shadow_evaluation + S4: prospective_bounded_pilot + +ownership: + LabTrust-Gym: + - campaign_simulator + - hidden_oracle + - failure_injection + - stored_samples + - benchmark_runner + - metrics + EnvAssure: + - metrological_validity_semantics + - environment_validity_transitions + Scientific-Memory: + - typed_influence_graph + - evidence_invalidation_records + - claim_model_dependencies + AKTA: + - consume_scientific_state_disposition_for_action_admissibility + VerifierLab: + - comparative_campaign_methodology + - repair_and_recovery_evaluation + +amendments: + effective: + - file: AMENDMENT_0.1.1.md + reason: distinguish physical drift onset from scientific invalidity onset + timing: before_campaign_implementation_and_before_primary_outcomes + - file: AMENDMENT_0.1.2.md + reason: freeze one-based indexing, check ordering, primary drift sign, and false-alarm handling + timing: before_campaign_implementation_and_before_primary_outcomes + +result_blinding: + implementation_before_primary_results: required + protocol_changes_after_primary_results: + permitted_only_as: post_registration_analysis diff --git a/research/delayed_evidence_invalidation/study_contract.v0.1.yaml b/research/delayed_evidence_invalidation/study_contract.v0.1.yaml new file mode 100644 index 00000000..cad87432 --- /dev/null +++ b/research/delayed_evidence_invalidation/study_contract.v0.1.yaml @@ -0,0 +1,263 @@ +schema_version: delayed_evidence_invalidation.v0.1 +status: preregistered +frozen_date: "2026-10-08" +repository: + name: fraware/LabTrust-Gym + base_commit: 7e361583fb3e8ca4f482b072f4ac190e30cc69a5 + branch: research/delayed-evidence-invalidation-v0 + +study: + id: dei_ph_drift_v0_1 + title: Delayed Evidence Invalidation and Scientific-State Recovery + mode: simulation + primary_failure_family: metrological_delayed_invalidation + no_new_repository: true + +state_model: + components: + - physical_state + - metrological_state + - evidence_state + - scientific_model_state + - selection_policy_state + influence_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SELECTED_BY + - SUPPORTS + - EXECUTED_ON + validity_states: + - provisional + - admitted + - quarantined + - invalidated + - superseded + descendant_dispositions: + severity_order: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +substrate: + instrument_semantics: opcua_lads_ph_meter_reference + reference_repository: opcua-lads/lads-server-collection + reference_component: servers/lads-ph-meter + required_features: + - nernst_signal_model + - temperature_compensation + - slope + - offset + - raw_signal + - calibration_values + - calibration_methods + note: > + Python reimplementation is permitted for deterministic offline campaigns only if + benchmark-level semantics are checked against the reference implementation. + +scientific_surface: + domain: + x_min: 0.0 + x_max: 1.0 + true_property: + expression: "7.0 + 1.20*sin(2*pi*x) + 0.35*cos(6*pi*x) + 0.25*x" + hidden_from_policy: true + target_ph: 7.40 + utility: + expression: "-(true_ph - 7.40)^2" + realism_claim: false + +campaign: + scientific_experiments: 240 + initial_design_points: 16 + adaptive_experiments: 224 + retained_sample_per_experiment: true + check_interval_experiments: 20 + check_replicates: 5 + check_reference_ph: 7.00 + +noise_calibration: + seed: 20261008 + readings: 1000 + reference_ph: 7.00 + output: sigma0 + excluded_from_primary_analysis: true + +failure: + type: gradual_offset_drift + onset: + distribution: discrete_uniform + min_experiment: 60 + max_experiment: 120 + hidden_from_methods: true + standardized_rates_per_experiment: + - 0.075 + - 0.125 + - 0.200 + rate_units: sigma0 + secondary_failure: + type: slope_drift + match_standardized_effect: true + +qc: + rule: shewhart_mean_threshold + threshold_expression: "abs(mean_check - 7.00) > 3*sigma0/sqrt(5)" + suspect_interval: "(last_in_control_check, detection_check]" + true_onset_available_to_hidden_adjudicator_only: true + +adaptive_policy: + primary: gp_ucb + exact_implementation: freeze_before_primary_runs + robust_comparator: + preferred: rcgp_ucb + reference: "Ezzerg, Bogunovic, Knoblauch, ICML 2026" + deviation_requires_preoutcome_protocol_update: true + +recovery_methods: + B0: + name: no_retroactive_recovery + retroactive_change: false + B1: + name: provenance_flagging + rebuild_model: false + B2: + name: qc_discard_retrain + discard_suspect_ph: true + remeasure_samples: false + B3: + name: qc_blanket_remeasure_retrain + exact_mode: + remeasure_fraction: 1.0 + budgeted_selection: uniform_random + B4: + name: typed_dependency_aware_recovery + hidden_oracle_access: false + exact_mode: + max_remeasure_fraction: 1.0 + budgeted_selection: + intended_rule: greedy_expected_model_discrepancy_reduction + freeze_before_primary_runs: true + B5: + name: robust_bo_plus_B2 + acquisition_robustness: true + recovery_semantics: B2 + +recovery_budgets: + fractions: + - 0.10 + - 0.25 + - 0.50 + rounding: ceil + +primary_outcomes: + O1: + name: post_recovery_model_error + measure: normalized_integrated_squared_error + reference: latent_true_surface + O2: + name: retained_valid_work + measure: fraction_valid_physical_experiments_or_revalidated_measurements_retained + O3: + name: recovery_cost + measures: + - remeasurement_count + - normalized_compute_cost + interpretation_priority: remeasurement_count + O4: + name: validity_classification_accuracy + measure: macro_f1 + classes: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +secondary_outcomes: + - detection_latency + - cumulative_optimization_regret + - invalid_observations_admitted + - validity_closure_size + - selection_influence_closure_size + - policy_path_divergence + - claims_withdrawn_or_recomputed + - wall_clock_recovery_time + - remeasurement_efficiency + - false_invalidation_count + +hypotheses: + H1: delayed_invalidity_requires_retroactive_recovery + H2: dependency_semantics_preserve_valid_science + H3: robust_acquisition_does_not_replace_recovery + H4: selection_dependence_differs_from_validity_dependence + +statistics: + monte_carlo_unit: campaign_seed + target_seeds_per_drift_rate_per_method: 200 + drift_rate_conditions: 3 + initial_total_campaigns_per_method_before_budget_expansion: 600 + calibration_seeds_per_condition: 20 + calibration_seeds_excluded_from_primary: true + confidence_intervals: + method: bootstrap + level: 0.95 + paired_bootstrap_when_valid: true + report_effect_sizes: true + significance_alone_sufficient: false + sample_size_change: + allowed_before_primary_outcome_inspection: true + requires_variance_or_runtime_justification: true + +required_ablations: + - slope_drift + - check_interval_10_20_40 + - sample_expiry + - alternate_acquisition_policy + - alternate_latent_surface + - edge_type_removal + - incomplete_dependency_metadata + - false_positive_qc_alarm + - multiple_invalidation_events + +kill_criteria: + - "B2 matches B4 on model error, retained valid work, and recovery cost within practically negligible differences." + - "B3 dominates B4 across the recovery Pareto frontier." + - "Validity and selection closures collapse to the same set in realistic variants." + - "B4 advantage depends on hidden information unavailable to deployed systems." + - "Effect disappears under a second response surface or slope-drift family." + - "Effect depends on benchmark-specific identifiers or hand-coded injected-failure knowledge." + - "A simpler generic provenance traversal reproduces B4 with equivalent outputs and effort." + - "LADS-aligned semantics differ from the reference implementation in result-driving ways." + +claim_tiers: + S0: benchmark_semantics + S1: standards_aligned_simulation + S2: second_failure_family_or_substrate + S3: real_lab_shadow_evaluation + S4: prospective_bounded_pilot + +ownership: + LabTrust-Gym: + - campaign_simulator + - hidden_oracle + - failure_injection + - stored_samples + - benchmark_runner + - metrics + EnvAssure: + - metrological_validity_semantics + - environment_validity_transitions + Scientific-Memory: + - typed_influence_graph + - evidence_invalidation_records + - claim_model_dependencies + AKTA: + - consume_scientific_state_disposition_for_action_admissibility + VerifierLab: + - comparative_campaign_methodology + - repair_and_recovery_evaluation + +result_blinding: + implementation_before_primary_results: required + protocol_changes_after_primary_results: + permitted_only_as: post_registration_analysis diff --git a/src/labtrust_gym/research/__init__.py b/src/labtrust_gym/research/__init__.py new file mode 100644 index 00000000..56356708 --- /dev/null +++ b/src/labtrust_gym/research/__init__.py @@ -0,0 +1,4 @@ +"""Research-only modules for preregistered experimental studies. + +These modules are not part of the stable public API unless promoted explicitly. +""" diff --git a/src/labtrust_gym/research/delayed_invalidation/__init__.py b/src/labtrust_gym/research/delayed_invalidation/__init__.py new file mode 100644 index 00000000..68420912 --- /dev/null +++ b/src/labtrust_gym/research/delayed_invalidation/__init__.py @@ -0,0 +1,41 @@ +"""Delayed evidence invalidation study primitives. + +The effective study protocol is frozen under: +research/delayed_evidence_invalidation/study_contract.v0.1.2.yaml +""" + +from .campaign import ( + CampaignConfig, + CampaignRun, + ExperimentPolicy, + ExperimentRecord, + PolicyObservation, + SeedBundle, + derive_seed_bundle, + run_campaign, + run_clean_counterfactual, +) +from .control import CheckStandardResult, SuspectInterval +from .failure import DriftKind, DriftSchedule +from .instrument import LadsAlignedPhMeter, PhMeasurement +from .surface import benchmark_ph_surface, benchmark_utility + +__all__ = [ + "CampaignConfig", + "CampaignRun", + "CheckStandardResult", + "DriftKind", + "DriftSchedule", + "ExperimentPolicy", + "ExperimentRecord", + "LadsAlignedPhMeter", + "PhMeasurement", + "PolicyObservation", + "SeedBundle", + "SuspectInterval", + "benchmark_ph_surface", + "benchmark_utility", + "derive_seed_bundle", + "run_campaign", + "run_clean_counterfactual", +] diff --git a/src/labtrust_gym/research/delayed_invalidation/campaign.py b/src/labtrust_gym/research/delayed_invalidation/campaign.py new file mode 100644 index 00000000..263b497b --- /dev/null +++ b/src/labtrust_gym/research/delayed_invalidation/campaign.py @@ -0,0 +1,332 @@ +"""Campaign generator for delayed evidence invalidation studies. + +The generator separates the policy-visible trace from hidden adjudication state. +It implements the campaign physics and measurement-control boundary only; no +recovery method is implemented here. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from typing import Protocol, Sequence + +import numpy as np + +from .control import CheckStandardResult, SuspectInterval, evaluate_check_standard, suspect_interval_from_history +from .failure import DriftKind, DriftSchedule +from .instrument import LadsAlignedPhMeter, PhMeasurement +from .surface import benchmark_ph_surface + + +@dataclass(frozen=True) +class PolicyObservation: + """Information available to the adaptive scientific policy.""" + + experiment_index: int + x: float + reported_ph: float + + +class ExperimentPolicy(Protocol): + """Stateless-from-history policy interface used by the study harness.""" + + def choose_x( + self, + observations: Sequence[PolicyObservation], + *, + experiment_index: int, + rng: np.random.Generator, + ) -> float: + """Choose the next x in [0, 1] using policy-visible information only.""" + ... + + +@dataclass(frozen=True) +class SeedBundle: + """Independent deterministic RNG streams.""" + + measurement_seed: int + check_seed: int + failure_seed: int + policy_seed: int + + +@dataclass(frozen=True) +class CampaignConfig: + """Frozen primary campaign parameters from protocol v0.1.2.""" + + total_experiments: int = 240 + initial_design_points: int = 16 + check_interval: int = 20 + check_replicates: int = 5 + check_reference_ph: float = 7.0 + temperature_c: float = 25.0 + drift_onset_min: int = 60 + drift_onset_max: int = 120 + drift_rate_sigma_per_experiment: float = 0.125 + drift_direction: int = 1 + application_validity_tolerance_sigma0: float = 1.0 + + def __post_init__(self) -> None: + if self.total_experiments <= 0: + raise ValueError("total_experiments must be positive") + if not 0 < self.initial_design_points < self.total_experiments: + raise ValueError("initial_design_points must lie inside the campaign") + if self.check_interval <= 0 or self.check_replicates <= 0: + raise ValueError("check interval and replicates must be positive") + if not 1 <= self.drift_onset_min <= self.drift_onset_max <= self.total_experiments: + raise ValueError("drift onset range must lie inside the one-based campaign") + if self.drift_rate_sigma_per_experiment < 0: + raise ValueError("drift rate must be non-negative") + if self.drift_direction not in (-1, 1): + raise ValueError("drift_direction must be -1 or 1") + if self.application_validity_tolerance_sigma0 <= 0: + raise ValueError("application validity tolerance must be positive") + + +@dataclass(frozen=True) +class ExperimentRecord: + """Full hidden adjudication record for one scientific experiment.""" + + experiment_index: int + sample_id: str + x: float + true_ph: float + measurement: PhMeasurement + systematic_bias_ph: float + scientifically_valid: bool + + +@dataclass(frozen=True) +class CampaignRun: + """One pre-recovery or full campaign realization.""" + + master_seed: int + seed_bundle: SeedBundle + sigma0: float + config: CampaignConfig + drift_enabled: bool + physical_drift_onset: int | None + scientific_invalidity_onset: int | None + first_alarm_index: int | None + false_positive_alarm: bool + eligible_for_primary_recovery_estimand: bool + suspect_interval: SuspectInterval | None + experiments: tuple[ExperimentRecord, ...] + policy_observations: tuple[PolicyObservation, ...] + checks: tuple[CheckStandardResult, ...] + + +def _derived_seed(master_seed: int, stream_id: int) -> int: + """Derive a stable uint64 seed without shared mutable RNG state.""" + seq = np.random.SeedSequence([int(master_seed), int(stream_id)]) + return int(seq.generate_state(1, dtype=np.uint64)[0]) + + +def derive_seed_bundle(master_seed: int) -> SeedBundle: + """Create independent streams for measurement, QC, failure, and policy.""" + return SeedBundle( + measurement_seed=_derived_seed(master_seed, 1), + check_seed=_derived_seed(master_seed, 2), + failure_seed=_derived_seed(master_seed, 3), + policy_seed=_derived_seed(master_seed, 4), + ) + + +def initial_design_x(index_one_based: int, n_points: int) -> float: + """Deterministic midpoint space-filling initial design on [0, 1].""" + if not 1 <= index_one_based <= n_points: + raise ValueError("initial-design index out of range") + return (index_one_based - 0.5) / n_points + + +def _validate_policy_x(x: float) -> float: + value = float(x) + if not 0.0 <= value <= 1.0: + raise ValueError(f"policy produced x outside [0, 1]: {value}") + return value + + +def _sample_drift_onset(config: CampaignConfig, failure_rng: np.random.Generator) -> int: + """Sample the registered inclusive one-based onset interval.""" + return int(failure_rng.integers(config.drift_onset_min, config.drift_onset_max + 1)) + + +def _scientific_invalidity_onset( + experiments: Sequence[ExperimentRecord], +) -> int | None: + for record in experiments: + if not record.scientifically_valid: + return record.experiment_index + return None + + +def run_campaign( + policy: ExperimentPolicy, + *, + master_seed: int, + sigma0: float, + config: CampaignConfig | None = None, + meter: LadsAlignedPhMeter | None = None, + drift_enabled: bool = True, + stop_at_first_alarm: bool = True, +) -> CampaignRun: + """Run one registered campaign realization. + + Event ordering is scientific experiment, then any scheduled check under the + same current metrological state, then optional stop after the first alarm. + + The adaptive policy sees only PolicyObservation objects. Hidden true pH, + drift parameters, systematic bias, validity labels, and QC readings are + never passed to choose_x. + """ + if sigma0 <= 0: + raise ValueError("sigma0 must be positive") + cfg = config or CampaignConfig() + ph_meter = meter or LadsAlignedPhMeter() + seeds = derive_seed_bundle(master_seed) + + measurement_rng = np.random.default_rng(seeds.measurement_seed) + check_rng = np.random.default_rng(seeds.check_seed) + failure_rng = np.random.default_rng(seeds.failure_seed) + policy_rng = np.random.default_rng(seeds.policy_seed) + + onset = _sample_drift_onset(cfg, failure_rng) if drift_enabled else None + drift = ( + DriftSchedule( + onset_index=onset, + rate_sigma_per_experiment=cfg.drift_rate_sigma_per_experiment, + sigma0=sigma0, + kind=DriftKind.OFFSET, + direction=cfg.drift_direction, + ) + if onset is not None + else None + ) + + experiments: list[ExperimentRecord] = [] + visible: list[PolicyObservation] = [] + checks: list[CheckStandardResult] = [] + first_alarm_index: int | None = None + + validity_tolerance = cfg.application_validity_tolerance_sigma0 * sigma0 + + for t in range(1, cfg.total_experiments + 1): + if t <= cfg.initial_design_points: + x = initial_design_x(t, cfg.initial_design_points) + else: + x = _validate_policy_x( + policy.choose_x( + tuple(visible), + experiment_index=t, + rng=policy_rng, + ) + ) + + true_ph = benchmark_ph_surface(x) + if drift is None: + physical_offset, physical_slope = 0.0, 100.0 + else: + physical_offset, physical_slope = drift.instrument_parameters_at(t) + + systematic_bias = ph_meter.systematic_bias_ph( + true_ph, + temperature_c=cfg.temperature_c, + physical_offset_ph=physical_offset, + physical_slope_pct=physical_slope, + ) + measurement = ph_meter.measure( + true_ph, + rng=measurement_rng, + temperature_c=cfg.temperature_c, + physical_offset_ph=physical_offset, + physical_slope_pct=physical_slope, + ) + record = ExperimentRecord( + experiment_index=t, + sample_id=f"SAMPLE_{master_seed}_{t:04d}", + x=x, + true_ph=true_ph, + measurement=measurement, + systematic_bias_ph=systematic_bias, + scientifically_valid=abs(systematic_bias) <= validity_tolerance, + ) + experiments.append(record) + visible.append( + PolicyObservation( + experiment_index=t, + x=x, + reported_ph=measurement.reported_ph, + ) + ) + + if t % cfg.check_interval == 0: + check_values = tuple( + ph_meter.measure( + cfg.check_reference_ph, + rng=check_rng, + temperature_c=cfg.temperature_c, + physical_offset_ph=physical_offset, + physical_slope_pct=physical_slope, + ).reported_ph + for _ in range(cfg.check_replicates) + ) + check = evaluate_check_standard( + check_values, + experiment_index=t, + sigma0=sigma0, + reference_ph=cfg.check_reference_ph, + ) + checks.append(check) + if not check.in_control and first_alarm_index is None: + first_alarm_index = t + if stop_at_first_alarm: + break + + invalidity_onset = _scientific_invalidity_onset(experiments) + false_positive = ( + first_alarm_index is not None + and (invalidity_onset is None or first_alarm_index < invalidity_onset) + ) + eligible = ( + first_alarm_index is not None + and invalidity_onset is not None + and invalidity_onset <= first_alarm_index + ) + + return CampaignRun( + master_seed=int(master_seed), + seed_bundle=seeds, + sigma0=float(sigma0), + config=cfg, + drift_enabled=bool(drift_enabled), + physical_drift_onset=onset, + scientific_invalidity_onset=invalidity_onset, + first_alarm_index=first_alarm_index, + false_positive_alarm=false_positive, + eligible_for_primary_recovery_estimand=eligible, + suspect_interval=suspect_interval_from_history(checks), + experiments=tuple(experiments), + policy_observations=tuple(visible), + checks=tuple(checks), + ) + + +def run_clean_counterfactual( + policy: ExperimentPolicy, + *, + master_seed: int, + sigma0: float, + config: CampaignConfig | None = None, + meter: LadsAlignedPhMeter | None = None, +) -> CampaignRun: + """Run the no-drift hidden clean reference for the full registered horizon.""" + return run_campaign( + policy, + master_seed=master_seed, + sigma0=sigma0, + config=config, + meter=meter, + drift_enabled=False, + stop_at_first_alarm=False, + ) diff --git a/src/labtrust_gym/research/delayed_invalidation/control.py b/src/labtrust_gym/research/delayed_invalidation/control.py new file mode 100644 index 00000000..3384b73f --- /dev/null +++ b/src/labtrust_gym/research/delayed_invalidation/control.py @@ -0,0 +1,85 @@ +"""Measurement-control primitives for delayed invalidation studies. + +This module owns only the registered check-standard rule and suspect interval. +It does not perform scientific-state recovery. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from math import sqrt +from typing import Iterable + +import numpy as np + + +@dataclass(frozen=True) +class CheckStandardResult: + """One scheduled check-standard decision.""" + + experiment_index: int + reference_ph: float + readings: tuple[float, ...] + mean_reported_ph: float + threshold_ph: float + in_control: bool + + +@dataclass(frozen=True) +class SuspectInterval: + """Conservative interval supplied to non-oracle recovery methods.""" + + last_in_control_index: int + detection_index: int + + def contains(self, experiment_index: int) -> bool: + return self.last_in_control_index < experiment_index <= self.detection_index + + +def evaluate_check_standard( + readings: Iterable[float], + *, + experiment_index: int, + sigma0: float, + reference_ph: float = 7.0, +) -> CheckStandardResult: + """Apply the preregistered 3-sigma mean check rule.""" + values = tuple(float(v) for v in readings) + if not values: + raise ValueError("readings must be non-empty") + if sigma0 <= 0: + raise ValueError("sigma0 must be positive") + mean_value = float(np.mean(np.asarray(values, dtype=float))) + threshold = 3.0 * sigma0 / sqrt(len(values)) + return CheckStandardResult( + experiment_index=int(experiment_index), + reference_ph=float(reference_ph), + readings=values, + mean_reported_ph=mean_value, + threshold_ph=float(threshold), + in_control=abs(mean_value - reference_ph) <= threshold, + ) + + +def suspect_interval_from_history( + history: Iterable[CheckStandardResult], +) -> SuspectInterval | None: + """Return the first detected out-of-control interval from ordered checks. + + A detection requires at least one previous in-control check. If the first + check is already out of control, the conservative onset is unresolved and + this function returns None rather than inventing a prior boundary. + """ + ordered = sorted(history, key=lambda item: item.experiment_index) + last_good: CheckStandardResult | None = None + for result in ordered: + if result.in_control: + last_good = result + continue + if last_good is None: + return None + return SuspectInterval( + last_in_control_index=last_good.experiment_index, + detection_index=result.experiment_index, + ) + return None diff --git a/src/labtrust_gym/research/delayed_invalidation/failure.py b/src/labtrust_gym/research/delayed_invalidation/failure.py new file mode 100644 index 00000000..5b27fdd6 --- /dev/null +++ b/src/labtrust_gym/research/delayed_invalidation/failure.py @@ -0,0 +1,67 @@ +"""Failure schedules for the delayed-evidence-invalidation benchmark.""" + +from __future__ import annotations + +from dataclasses import dataclass +from enum import StrEnum + + +class DriftKind(StrEnum): + """Registered drift families.""" + + OFFSET = "offset" + SLOPE = "slope" + + +@dataclass(frozen=True) +class DriftSchedule: + """Primary gradual drift schedule. + + v0.1 primary analysis uses OFFSET drift. SLOPE is reserved for the + preregistered secondary ablation and requires an explicitly frozen + standardized mapping before slope-drift results are run. + """ + + onset_index: int + rate_sigma_per_experiment: float + sigma0: float + kind: DriftKind = DriftKind.OFFSET + direction: int = 1 + base_offset_ph: float = 0.0 + base_slope_pct: float = 100.0 + + def __post_init__(self) -> None: + if self.onset_index < 0: + raise ValueError("onset_index must be non-negative") + if self.rate_sigma_per_experiment < 0: + raise ValueError("rate_sigma_per_experiment must be non-negative") + if self.sigma0 <= 0: + raise ValueError("sigma0 must be positive") + if self.direction not in (-1, 1): + raise ValueError("direction must be -1 or 1") + if self.base_slope_pct <= 0: + raise ValueError("base_slope_pct must be positive") + + def elapsed_drift_steps(self, experiment_index: int) -> int: + """Number of drift-bearing measurements at this experiment index.""" + if experiment_index < self.onset_index: + return 0 + return experiment_index - self.onset_index + 1 + + def offset_ph_at(self, experiment_index: int) -> float: + """Physical offset for the registered offset-drift family.""" + if self.kind is not DriftKind.OFFSET: + raise NotImplementedError( + "Slope drift requires a separately frozen standardized mapping." + ) + steps = self.elapsed_drift_steps(experiment_index) + increment = self.direction * steps * self.rate_sigma_per_experiment * self.sigma0 + return self.base_offset_ph + increment + + def instrument_parameters_at(self, experiment_index: int) -> tuple[float, float]: + """Return physical offset pH and physical slope percent.""" + if self.kind is DriftKind.OFFSET: + return self.offset_ph_at(experiment_index), self.base_slope_pct + raise NotImplementedError( + "Slope drift is a preregistered secondary ablation, not part of the v0.1 primary implementation." + ) diff --git a/src/labtrust_gym/research/delayed_invalidation/instrument.py b/src/labtrust_gym/research/delayed_invalidation/instrument.py new file mode 100644 index 00000000..f7e4eadc --- /dev/null +++ b/src/labtrust_gym/research/delayed_invalidation/instrument.py @@ -0,0 +1,193 @@ +"""Deterministic, benchmark-local pH measurement semantics aligned to OPC UA LADS. + +Source model: + opcua-lads/lads-server-collection + servers/lads-ph-meter/src/ph-meter-unit-simulator.ts + +The upstream simulator computes a Nernst-based raw millivolt signal from a +simulated physical pH, temperature, slope and offset, adds small raw-signal +noise, and reconstructs reported pH using the sensor calibration values. + +This module ports that algebraic measurement boundary for reproducible research +runs. It is not a production OPC UA device implementation and it does not claim +bit-for-bit equivalence to the upstream server state machine, timers, +historization, or random-number source. +""" + +from __future__ import annotations + +from dataclasses import dataclass + +import numpy as np +from numpy.typing import NDArray + +R_GAS = 8.314 +FARADAY = 96485.0 +CELSIUS_TO_KELVIN = 273.15 + + +@dataclass(frozen=True) +class PhMeasurement: + """One benchmark pH observation with state required for adjudication.""" + + true_ph: float + temperature_c: float + raw_mv: float + reported_ph: float + physical_offset_ph: float + physical_slope_pct: float + calibration_offset_ph: float + calibration_slope_pct: float + raw_noise_mv: float + + +@dataclass(frozen=True) +class LadsAlignedPhMeter: + """Steady-state pH signal model derived from the LADS reference simulator.""" + + raw_noise_half_width_mv: float = 0.25 + + @staticmethod + def _thermal_factor(temperature_c: float) -> float: + if temperature_c <= -CELSIUS_TO_KELVIN: + raise ValueError("temperature_c must exceed absolute zero") + return np.log(10.0) * R_GAS * (temperature_c + CELSIUS_TO_KELVIN) / FARADAY + + def raw_signal_mv( + self, + true_ph: float, + *, + temperature_c: float = 25.0, + physical_offset_ph: float = 0.0, + physical_slope_pct: float = 100.0, + ) -> float: + """Return the noiseless simulated electrode signal in millivolts.""" + if physical_slope_pct <= 0: + raise ValueError("physical_slope_pct must be positive") + thermal = self._thermal_factor(temperature_c) + return ( + 1000.0 + * thermal + * (7.0 + physical_offset_ph - true_ph) + * 0.01 + * physical_slope_pct + ) + + def reconstruct_ph( + self, + raw_mv: float, + *, + temperature_c: float = 25.0, + calibration_offset_ph: float = 0.0, + calibration_slope_pct: float = 100.0, + ) -> float: + """Reconstruct pH from raw signal and calibration state.""" + if calibration_slope_pct <= 0: + raise ValueError("calibration_slope_pct must be positive") + thermal = self._thermal_factor(temperature_c) + return ( + 7.0 + + calibration_offset_ph + - 0.001 * raw_mv / (0.01 * calibration_slope_pct) / thermal + ) + + def systematic_bias_ph( + self, + true_ph: float, + *, + temperature_c: float = 25.0, + physical_offset_ph: float = 0.0, + physical_slope_pct: float = 100.0, + calibration_offset_ph: float = 0.0, + calibration_slope_pct: float = 100.0, + ) -> float: + """Expected reporting bias with raw measurement noise removed.""" + noiseless_raw = self.raw_signal_mv( + true_ph, + temperature_c=temperature_c, + physical_offset_ph=physical_offset_ph, + physical_slope_pct=physical_slope_pct, + ) + reported = self.reconstruct_ph( + noiseless_raw, + temperature_c=temperature_c, + calibration_offset_ph=calibration_offset_ph, + calibration_slope_pct=calibration_slope_pct, + ) + return float(reported - true_ph) + + def measure( + self, + true_ph: float, + *, + rng: np.random.Generator | None = None, + temperature_c: float = 25.0, + physical_offset_ph: float = 0.0, + physical_slope_pct: float = 100.0, + calibration_offset_ph: float = 0.0, + calibration_slope_pct: float = 100.0, + raw_noise_mv: float | None = None, + ) -> PhMeasurement: + """Measure one value with explicit or seeded raw-signal perturbation.""" + noiseless = self.raw_signal_mv( + true_ph, + temperature_c=temperature_c, + physical_offset_ph=physical_offset_ph, + physical_slope_pct=physical_slope_pct, + ) + if raw_noise_mv is None: + if rng is None or self.raw_noise_half_width_mv == 0: + noise = 0.0 + else: + noise = float( + rng.uniform( + -self.raw_noise_half_width_mv, + self.raw_noise_half_width_mv, + ) + ) + else: + noise = float(raw_noise_mv) + + raw_mv = noiseless + noise + reported = self.reconstruct_ph( + raw_mv, + temperature_c=temperature_c, + calibration_offset_ph=calibration_offset_ph, + calibration_slope_pct=calibration_slope_pct, + ) + return PhMeasurement( + true_ph=float(true_ph), + temperature_c=float(temperature_c), + raw_mv=float(raw_mv), + reported_ph=float(reported), + physical_offset_ph=float(physical_offset_ph), + physical_slope_pct=float(physical_slope_pct), + calibration_offset_ph=float(calibration_offset_ph), + calibration_slope_pct=float(calibration_slope_pct), + raw_noise_mv=float(noise), + ) + + def estimate_sigma0( + self, + *, + seed: int, + readings: int = 1000, + reference_ph: float = 7.0, + temperature_c: float = 25.0, + ) -> float: + """Estimate baseline reported-pH standard deviation for registered scaling.""" + if readings < 2: + raise ValueError("readings must be >= 2") + rng = np.random.default_rng(seed) + values: NDArray[np.float64] = np.asarray( + [ + self.measure( + reference_ph, + rng=rng, + temperature_c=temperature_c, + ).reported_ph + for _ in range(readings) + ], + dtype=float, + ) + return float(np.std(values, ddof=1)) diff --git a/src/labtrust_gym/research/delayed_invalidation/surface.py b/src/labtrust_gym/research/delayed_invalidation/surface.py new file mode 100644 index 00000000..8f5ae986 --- /dev/null +++ b/src/labtrust_gym/research/delayed_invalidation/surface.py @@ -0,0 +1,22 @@ +"""Frozen synthetic scientific surface for the v0.1 preregistered study.""" + +from __future__ import annotations + +import math + + +def benchmark_ph_surface(x: float) -> float: + """Return hidden true pH for x in [0, 1]. + + This is a deterministic benchmark surface. It is not presented as a + chemically realistic formulation model. + """ + if not 0.0 <= x <= 1.0: + raise ValueError("x must lie in [0, 1]") + return 7.0 + 1.20 * math.sin(2.0 * math.pi * x) + 0.35 * math.cos(6.0 * math.pi * x) + 0.25 * x + + +def benchmark_utility(x: float, *, target_ph: float = 7.40) -> float: + """Utility maximized by the adaptive policy.""" + true_ph = benchmark_ph_surface(x) + return -((true_ph - target_ph) ** 2) diff --git a/tests/research/test_delayed_invalidation_campaign.py b/tests/research/test_delayed_invalidation_campaign.py new file mode 100644 index 00000000..34635eb9 --- /dev/null +++ b/tests/research/test_delayed_invalidation_campaign.py @@ -0,0 +1,122 @@ +from __future__ import annotations + +from dataclasses import fields +from typing import Sequence + +import numpy as np + +from labtrust_gym.research.delayed_invalidation.campaign import ( + CampaignConfig, + PolicyObservation, + derive_seed_bundle, + run_campaign, + run_clean_counterfactual, +) +from labtrust_gym.research.delayed_invalidation.instrument import LadsAlignedPhMeter + + +class FixedPolicy: + def choose_x( + self, + observations: Sequence[PolicyObservation], + *, + experiment_index: int, + rng: np.random.Generator, + ) -> float: + del observations, experiment_index, rng + return 0.5 + + +def test_policy_observation_excludes_hidden_oracle_fields() -> None: + assert {field.name for field in fields(PolicyObservation)} == { + "experiment_index", + "x", + "reported_ph", + } + + +def test_rng_streams_are_deterministic_and_separate() -> None: + a = derive_seed_bundle(123) + b = derive_seed_bundle(123) + assert a == b + assert len({a.measurement_seed, a.check_seed, a.failure_seed, a.policy_seed}) == 4 + + +def test_clean_counterfactual_stays_valid_without_measurement_noise() -> None: + meter = LadsAlignedPhMeter(raw_noise_half_width_mv=0.0) + run = run_clean_counterfactual( + FixedPolicy(), + master_seed=1, + sigma0=0.01, + meter=meter, + ) + assert len(run.experiments) == 240 + assert run.physical_drift_onset is None + assert run.scientific_invalidity_onset is None + assert run.first_alarm_index is None + assert run.suspect_interval is None + assert not run.false_positive_alarm + assert not run.eligible_for_primary_recovery_estimand + assert all(record.scientifically_valid for record in run.experiments) + assert all(check.in_control for check in run.checks) + + +def test_registered_fixture_orders_drift_invalidity_and_detection() -> None: + meter = LadsAlignedPhMeter(raw_noise_half_width_mv=0.0) + config = CampaignConfig( + drift_rate_sigma_per_experiment=0.125, + application_validity_tolerance_sigma0=1.0, + ) + run = run_campaign( + FixedPolicy(), + master_seed=1, + sigma0=0.01, + config=config, + meter=meter, + drift_enabled=True, + stop_at_first_alarm=True, + ) + + assert run.physical_drift_onset == 98 + assert run.scientific_invalidity_onset == 106 + assert run.first_alarm_index == 120 + assert len(run.experiments) == 120 + assert run.eligible_for_primary_recovery_estimand + assert not run.false_positive_alarm + + assert run.suspect_interval is not None + assert run.suspect_interval.last_in_control_index == 100 + assert run.suspect_interval.detection_index == 120 + assert run.suspect_interval.contains(run.scientific_invalidity_onset) + + +def test_policy_trace_never_contains_hidden_true_ph_or_validity() -> None: + meter = LadsAlignedPhMeter(raw_noise_half_width_mv=0.0) + run = run_campaign( + FixedPolicy(), + master_seed=1, + sigma0=0.01, + meter=meter, + ) + for observation in run.policy_observations: + assert not hasattr(observation, "true_ph") + assert not hasattr(observation, "systematic_bias_ph") + assert not hasattr(observation, "scientifically_valid") + + +def test_same_seed_replays_same_hidden_failure_and_measurements() -> None: + policy = FixedPolicy() + meter = LadsAlignedPhMeter() + sigma0 = meter.estimate_sigma0(seed=20261008, readings=1000) + a = run_campaign(policy, master_seed=42, sigma0=sigma0, meter=meter) + b = run_campaign(policy, master_seed=42, sigma0=sigma0, meter=meter) + + assert a.physical_drift_onset == b.physical_drift_onset + assert a.scientific_invalidity_onset == b.scientific_invalidity_onset + assert a.first_alarm_index == b.first_alarm_index + assert [x.measurement.reported_ph for x in a.experiments] == [ + x.measurement.reported_ph for x in b.experiments + ] + assert [x.mean_reported_ph for x in a.checks] == [ + x.mean_reported_ph for x in b.checks + ] diff --git a/tests/research/test_delayed_invalidation_control.py b/tests/research/test_delayed_invalidation_control.py new file mode 100644 index 00000000..63cb6bb0 --- /dev/null +++ b/tests/research/test_delayed_invalidation_control.py @@ -0,0 +1,61 @@ +from __future__ import annotations + +import pytest + +from labtrust_gym.research.delayed_invalidation.control import ( + evaluate_check_standard, + suspect_interval_from_history, +) + + +def test_check_standard_accepts_reference_centered_readings() -> None: + result = evaluate_check_standard( + [7.001, 6.999, 7.000, 7.002, 6.998], + experiment_index=20, + sigma0=0.01, + ) + assert result.in_control + + +def test_check_standard_rejects_mean_beyond_registered_threshold() -> None: + result = evaluate_check_standard( + [7.03, 7.03, 7.03, 7.03, 7.03], + experiment_index=40, + sigma0=0.01, + ) + assert not result.in_control + assert result.threshold_ph == pytest.approx(3.0 * 0.01 / (5.0**0.5)) + + +def test_suspect_interval_uses_last_good_check_and_first_failure() -> None: + good_20 = evaluate_check_standard( + [7.0] * 5, + experiment_index=20, + sigma0=0.01, + ) + good_40 = evaluate_check_standard( + [7.0] * 5, + experiment_index=40, + sigma0=0.01, + ) + bad_60 = evaluate_check_standard( + [7.03] * 5, + experiment_index=60, + sigma0=0.01, + ) + interval = suspect_interval_from_history([bad_60, good_20, good_40]) + assert interval is not None + assert interval.last_in_control_index == 40 + assert interval.detection_index == 60 + assert not interval.contains(40) + assert interval.contains(41) + assert interval.contains(60) + + +def test_first_check_failure_does_not_invent_last_good_boundary() -> None: + bad = evaluate_check_standard( + [7.03] * 5, + experiment_index=20, + sigma0=0.01, + ) + assert suspect_interval_from_history([bad]) is None diff --git a/tests/research/test_delayed_invalidation_instrument.py b/tests/research/test_delayed_invalidation_instrument.py new file mode 100644 index 00000000..1ae5d8b7 --- /dev/null +++ b/tests/research/test_delayed_invalidation_instrument.py @@ -0,0 +1,86 @@ +from __future__ import annotations + +import math + +import numpy as np +import pytest + +from labtrust_gym.research.delayed_invalidation.failure import DriftKind, DriftSchedule +from labtrust_gym.research.delayed_invalidation.instrument import LadsAlignedPhMeter +from labtrust_gym.research.delayed_invalidation.surface import benchmark_ph_surface, benchmark_utility + + +def test_ideal_meter_reconstructs_true_ph_without_noise() -> None: + meter = LadsAlignedPhMeter() + for true_ph in (4.0, 6.5, 7.0, 9.25): + measurement = meter.measure(true_ph, raw_noise_mv=0.0) + assert measurement.reported_ph == pytest.approx(true_ph, abs=1e-12) + + +def test_physical_offset_with_stale_calibration_biases_reported_value() -> None: + meter = LadsAlignedPhMeter() + measurement = meter.measure( + 6.5, + physical_offset_ph=0.20, + calibration_offset_ph=0.0, + raw_noise_mv=0.0, + ) + assert measurement.reported_ph == pytest.approx(6.30, abs=1e-12) + + +def test_physical_slope_detuning_is_observable_with_stale_calibration() -> None: + meter = LadsAlignedPhMeter() + measurement = meter.measure( + 4.0, + physical_slope_pct=95.0, + calibration_slope_pct=100.0, + raw_noise_mv=0.0, + ) + assert measurement.reported_ph == pytest.approx(4.15, abs=1e-12) + + +def test_seeded_measurements_are_reproducible() -> None: + meter = LadsAlignedPhMeter() + rng_a = np.random.default_rng(1234) + rng_b = np.random.default_rng(1234) + seq_a = [meter.measure(7.0, rng=rng_a).reported_ph for _ in range(25)] + seq_b = [meter.measure(7.0, rng=rng_b).reported_ph for _ in range(25)] + assert seq_a == seq_b + + +def test_sigma0_is_positive_and_reproducible() -> None: + meter = LadsAlignedPhMeter() + sigma_a = meter.estimate_sigma0(seed=20261008, readings=1000) + sigma_b = meter.estimate_sigma0(seed=20261008, readings=1000) + assert sigma_a > 0.0 + assert sigma_a == pytest.approx(sigma_b, rel=0.0, abs=0.0) + + +def test_offset_drift_starts_exactly_at_registered_onset() -> None: + schedule = DriftSchedule( + onset_index=60, + rate_sigma_per_experiment=0.125, + sigma0=0.01, + kind=DriftKind.OFFSET, + ) + assert schedule.offset_ph_at(59) == pytest.approx(0.0) + assert schedule.offset_ph_at(60) == pytest.approx(0.00125) + assert schedule.offset_ph_at(61) == pytest.approx(0.00250) + + +def test_surface_contract() -> None: + assert benchmark_ph_surface(0.0) == pytest.approx(7.35) + assert math.isfinite(benchmark_ph_surface(0.5)) + assert benchmark_utility(0.25) <= 0.0 + with pytest.raises(ValueError): + benchmark_ph_surface(-0.01) + + +def test_systematic_bias_matches_stale_offset_error() -> None: + meter = LadsAlignedPhMeter() + bias = meter.systematic_bias_ph( + 6.5, + physical_offset_ph=0.20, + calibration_offset_ph=0.0, + ) + assert bias == pytest.approx(-0.20, abs=1e-12)