diff --git a/research/delayed_evidence_invalidation/AMENDMENT_0.1.1.md b/research/delayed_evidence_invalidation/AMENDMENT_0.1.1.md new file mode 100644 index 00000000..2e69b2f3 --- /dev/null +++ b/research/delayed_evidence_invalidation/AMENDMENT_0.1.1.md @@ -0,0 +1,95 @@ +# Protocol Amendment v0.1.1 — Drift onset versus scientific invalidity + +**Date:** 2026-10-08 +**Timing:** before implementation of the campaign simulator and before inspection of any primary comparative outcome +**Supersedes:** specific validity semantics in PREREGISTRATION v0.1 Sections 8, 9, 15, and 17 where “drift onset” could be read as equivalent to “scientific invalidity onset” + +## Reason for amendment + +Gradual physical drift and scientific invalidity are distinct events. + +A measurement process may begin drifting while remaining inside an application-validity tolerance and while scheduled check standards still satisfy the registered control rule. Treating every observation after the first nonzero physical drift as invalid would make the registered suspect interval logically inconsistent: an “in-control” check could occur after physical drift began. + +The correction below separates: + +1. **physical drift onset** — the first experiment whose simulated instrument parameter differs from its baseline; and +2. **scientific invalidity onset** — the first experiment whose systematic measurement bias exceeds a predeclared application-validity tolerance. + +This amendment changes no observed results and was made before the campaign implementation existed. + +## Effective validity rule + +Let the systematic pH reporting bias at experiment (t) be + +[ +b_{mathrm{sys}}(t)= +E[hat{pH}_tmid pH_t^*,M_t]-pH_t^*, +] + +where the expectation removes zero-mean measurement noise while retaining the current metrological state. + +For the primary v0.1.1 benchmark, define the application-validity tolerance + +[ +delta_{mathrm{app}} = 1.0,hatsigma_0. +] + +The hidden oracle defines the **scientific invalidity onset** + +[ +k_{mathrm{invalid}} += +min{tge k_{mathrm{drift}}: +|b_{mathrm{sys}}(t)|>delta_{mathrm{app}}}. +] + +Evidence before (k_{mathrm{invalid}}) is not labelled invalid solely because nonzero drift has begun. + +## Detection and suspect interval + +The registered check-standard rule remains unchanged: + +[ +|ar y_c-7.00|>3hatsigma_0/sqrt{5}. +] + +The operational suspect interval supplied to non-oracle recovery methods remains + +[ +(t_{mathrm{last in control}},t_{mathrm{detect}}]. +] + +It should be described as an **operational bracketing interval**, not as proof of the true physical drift onset. + +Under the primary parameters, the application-validity tolerance is lower than the check-mean alarm threshold, so delayed invalidity is possible: measurements may become scientifically invalid before the next scheduled check declares the process out of control. + +## Evaluation changes + +Primary hidden-oracle validity labels use (k_{mathrm{invalid}}), not (k_{mathrm{drift}}). + +Report both: + +- physical-drift detection latency: (t_{mathrm{detect}}-k_{mathrm{drift}}); +- scientific-invalidity detection latency: (t_{mathrm{detect}}-k_{mathrm{invalid}}). + +The primary count of “invalid observations admitted before alarm” begins at (k_{mathrm{invalid}}). + +Validity-classification accuracy is scored against invalidity labels derived from (delta_{mathrm{app}}). + +Selection-influence tracing may begin from evidence labelled invalid under this rule; physical drift alone does not initiate invalidation propagation. + +## Required sensitivity analysis + +Before broad claims, repeat the primary comparison for: + +[ +delta_{mathrm{app}}/hatsigma_0 in {0.5,1.0,2.0}. +] + +The 1.0 condition remains primary. + +If the main conclusion depends qualitatively on the 1.0 choice, the paper must narrow the claim accordingly. + +## Claim discipline + +This benchmark tolerance is a study parameter, not a universal pH-metrology criterion and not a claim about acceptable error in any production laboratory. diff --git a/research/delayed_evidence_invalidation/AMENDMENT_0.1.2.md b/research/delayed_evidence_invalidation/AMENDMENT_0.1.2.md new file mode 100644 index 00000000..10e1d470 --- /dev/null +++ b/research/delayed_evidence_invalidation/AMENDMENT_0.1.2.md @@ -0,0 +1,72 @@ +# Protocol Amendment v0.1.2 — Indexing, event ordering, drift sign, and false alarms + +**Date:** 2026-10-08 +**Timing:** before implementation of the campaign simulator and before inspection of primary comparative outcomes +**Builds on:** v0.1.1 + +This amendment freezes implementation degrees of freedom that were left implicit in v0.1/v0.1.1. + +## Experiment indexing + +Scientific experiments use **one-based** indices: + +[ +tin{1,ldots,240}. +] + +The initial design occupies experiments 1–16. Adaptive selection begins at experiment 17. + +The hidden physical-drift onset is sampled uniformly from the integer set + +[ +{60,ldots,120}. +] + +The onset experiment itself receives the first nonzero drift increment. + +## Check-standard ordering + +A scheduled check-standard event occurs immediately **after** scientific experiments + +[ +20,40,60,ldots,240. +] + +The check uses the same current instrument/metrological state as the scientific measurement just completed at that experiment index. It receives a separate measurement-noise draw. + +Thus a check at experiment 80 summarizes instrument state after the 80th scientific experiment; it does not alter the scientific-experiment count. + +## Primary drift direction + +The primary offset-drift family uses positive physical offset drift: + +[ +Delta_{mathrm{offset}}(t)>0. +] + +Under the LADS-aligned stale-calibration algebra this produces a negative systematic reported-pH bias. + +A sign-reversal ablation with otherwise matched standardized magnitude is required before making sign-invariant claims. + +## False-positive control alarms + +A check-standard alarm before the hidden scientific-invalidity onset is a **false-positive control alarm** for purposes of this benchmark. + +Such runs must: + +1. remain in the released campaign corpus; +2. be counted in the false-alarm outcome; +3. never be silently regenerated with a different seed. + +The primary delayed-invalidation recovery estimand is evaluated on campaigns in which scientific invalidity occurs before the first control alarm. This conditioning rule is fixed before results and must be reported with the unconditional alarm rate and the fraction of campaigns entering the primary recovery estimand. + +A sensitivity analysis must also report all-seed operational cost so that a method is not rewarded for expensive reactions to false alarms. + +## Additional required sensitivity analysis + +Add: + +- drift-sign reversal; +- false-positive alarm cost across all seeds. + +No production-metrology claim follows from these benchmark choices. diff --git a/research/delayed_evidence_invalidation/AMENDMENT_0.1.3.md b/research/delayed_evidence_invalidation/AMENDMENT_0.1.3.md new file mode 100644 index 00000000..66e64d3f --- /dev/null +++ b/research/delayed_evidence_invalidation/AMENDMENT_0.1.3.md @@ -0,0 +1,181 @@ +# Protocol Amendment v0.1.3 — Optimizer and non-stationarity baselines + +**Date:** 2026-10-08 +**Timing:** before implementation of any Bayesian-optimization policy and before inspection of any primary comparative recovery outcome +**Builds on:** v0.1.2 + +## Reason for amendment + +The primary failure family is gradual persistent metrological drift. RCGP-UCB is a strong robustness comparator for observation corruptions, but its published theory studies frequency-constrained corruptions that may have unbounded magnitude. Persistent drift is structurally different. Time-varying GP-bandit work instead addresses stale observations under non-stationarity using resetting, forgetting, or sliding windows. + +To avoid overstating one robustness baseline, this amendment: + +1. freezes the scalar objective supplied to all optimizers; +2. freezes a matched GP-UCB implementation profile; +3. retains an RCGP comparator using the authors' released implementation; +4. adds sliding-window GP-UCB as the primary non-stationarity comparator; and +5. separates recovery comparisons from acquisition-robustness comparisons. + +No primary campaign/recovery outcome has been inspected. + +## 1. Optimizer-visible objective + +The optimizer never receives latent true pH or latent utility. + +For policy-visible reported pH \(\widehat{pH}_t\), define the observed BO target + +\[ +y_t^{obs} += +-(\widehat{pH}_t-7.40)^2. +\] + +For hidden adjudication only, define + +\[ +y_t^{true} += +-(pH_t^*-7.40)^2. +\] + +All optimizer policies use \(y_t^{obs}\). Regret and clean-reference evaluation may use \(y_t^{true}\). + +## 2. Shared UCB schedule + +All GP-UCB-style policies use the released RCGP codebase's theory-guided exploration multiplier: + +\[ +\beta_t += +\max\left( +1.0, +1.7\sqrt{2\log(t+2)} +\right), +\] + +where \(t\) is the zero-based adaptive-BO iteration after the 16-point initial design. + +In this study, \(\beta_t\) is the literal multiplier on posterior standard deviation: + +\[ +a_t(x)=\mu_t(x)+\beta_t\sigma_t(x). +\] + +This matches the released code interface and is not presented as a re-derivation of the paper's theoretical confidence notation. + +## 3. Shared acquisition maximization + +To eliminate numerical-acquisition optimization as a confound, every primary one-dimensional GP-UCB-style policy evaluates its acquisition on the same deterministic grid: + +- domain: \([0,1]\); +- grid points: 1001, including both endpoints; +- selection: first grid index attaining the maximum acquisition value. + +The released RCGP repository uses BoTorch continuous acquisition optimization in its generic experiment runner. This study deliberately replaces only that search step with the matched deterministic grid across all compared GP policies. Accordingly, the RCGP condition is a released-RCGP-surrogate comparator with matched UCB acquisition, not a claim of bit-for-bit reproduction of every released experiment script. + +## 4. Standard GP-UCB + +Primary adaptive acquisition policy for B0–B4: + +- Gaussian-process implementation: BoTorch `SingleTaskGP`; +- outcomes standardized at each refit; +- model hyperparameters fitted by exact marginal likelihood at every adaptive iteration; +- double precision; +- CPU reference execution; +- all currently admitted policy-visible observations used; +- UCB multiplier from Section 2; +- deterministic 1001-point acquisition grid. + +Reference optimizer package versions for the frozen study environment: + +- Python 3.12; +- torch 2.8.0; +- botorch 0.15.1; +- gpytorch 1.14; +- numpy 1.26.4; +- scipy 1.12.0. + +These versions match the released RCGP repository lock where applicable. + +A study-local environment lock must record transitive dependencies before primary execution. + +## 5. RCGP comparator + +Reference implementation: + +- repository: `EZZERG/RCGP`; +- pinned commit: `68bd978dfe4de28aed02184efc64ea343cfa9afc`; +- package version metadata: 0.1.0; +- authors' released environment requires Python >=3.12,<3.13. + +The repository has no root LICENSE file at the pinned commit. The study must not vendor or redistribute its source. Integration should occur through a separately obtained local checkout/environment or a clean-room implementation based on published mathematics only after an explicit review. + +For the external released-code comparator, freeze: + +- model factory: released `create_rcgp_model`; +- standardized outcomes: true; +- P-IMQ plateau width: 1.96 on the standardized outcome scale; +- P-IMQ shape parameter \(c=1.0\); +- plateau center: fixed zero center; +- likelihood noise: fitted; +- GP mean: fitted; +- fitting objective: weighted leave-one-out cross-validation (`wloo-cv`); +- optimizer: L-BFGS using the released default path; +- UCB multiplier: Section 2; +- acquisition search: matched 1001-point grid, not the released `optimize_acqf` routine. + +The \(L=1.96,c=1\) choice is motivated by the paper's standardized-data discussion of simple manual P-IMQ settings. It is frozen before results and will not be tuned on this benchmark. + +This comparator must be described as an empirical RCGP-based robustness baseline. Its published sublinear-regret guarantees are not assumed to apply to persistent calibration drift. + +## 6. Sliding-window GP-UCB non-stationarity comparator + +Add a non-stationarity baseline family: + +- `SW-GP-UCB-20` +- `SW-GP-UCB-40` +- `SW-GP-UCB-80` + +Each uses the same Standard GP-UCB model, fitting, UCB schedule, and acquisition grid as Section 4, but the surrogate is refitted using only the most recent \(w\) scientific observations. + +All three windows are reported. No window is selected post hoc as the single representative result. + +The sliding-window family is motivated by established time-varying GP-bandit work, where resetting/sliding-window/forgetting methods address stale observations in non-stationary environments. + +## 7. Recovery versus acquisition comparisons + +The primary recovery estimand isolates recovery semantics: + +- B0, B1, B2, B3, and B4 all operate on the **same pre-alarm Standard GP-UCB campaign trace** for a given seed and drift realization. +- Their differences begin only after the registered invalidation/control event. + +Acquisition-robustness controls generate different trajectories and are analyzed separately: + +- B5: RCGP comparator + B2 recovery after alarm; +- B6-20/B6-40/B6-80: corresponding sliding-window GP-UCB policy + B2 recovery after alarm. + +B5/B6 are not used as paired evidence that B4 has better recovery semantics, because their experiment-selection trajectories differ before the alarm. + +## 8. Hyperparameter and RNG discipline + +- No optimizer hyperparameter may be selected using primary outcome performance. +- Any implementation calibration required for numerical stability must use only excluded calibration seeds. +- Torch/NumPy seeds must be deterministically derived from the registered campaign seed and adaptive iteration. +- Model-fitting failures, numerical warnings, or fallback paths must be recorded as first-class run artifacts. +- Failed optimizer fits may not be silently rerun with a new seed. + +## 9. Additional kill criterion + +The richer scientific-state recovery claim is weakened if a strong non-stationarity policy (any fixed preregistered SW-GP-UCB window) combined with B2 recovery matches B4 on post-recovery scientific-state quality and retained valid work at lower total cost. + +If that occurs, the paper should narrow the contribution to settings where delayed semantic invalidation remains consequential after non-stationarity-aware acquisition. + +## 10. Additional sensitivity analysis + +Before making optimizer-independent claims, repeat the main recovery comparison using at least one acquisition policy other than full-history GP-UCB. + +The acquisition-policy sensitivity result is secondary to the B0–B4 same-trace recovery comparison. + +## 11. Explicit exclusions + +Dynamic BO variants that add time directly as a GP input are relevant adjacent methods and should be discussed. They are not added as a primary v0.1.3 baseline because implementation and temporal-kernel choices would introduce another hyperparameter family. Sliding-window GP-UCB provides a transparent preregistered non-stationarity control. A time-input DBO replication may be added later as a versioned external-validity analysis, never retroactively as a primary baseline. diff --git a/research/delayed_evidence_invalidation/AMENDMENT_0.1.4.md b/research/delayed_evidence_invalidation/AMENDMENT_0.1.4.md new file mode 100644 index 00000000..422e30c1 --- /dev/null +++ b/research/delayed_evidence_invalidation/AMENDMENT_0.1.4.md @@ -0,0 +1,161 @@ +# Protocol Amendment v0.1.4 — Influence-graph orientation and strong recovery baselines + +**Date:** 2026-10-08 +**Timing:** before implementation of the typed influence graph or any recovery operator and before inspection of primary comparative recovery outcomes +**Builds on:** v0.1.3 + +## Reason for amendment + +Two ambiguities remained before B4 implementation: + +1. the original edge labels described semantic relations but did not freeze a single graph orientation; and +2. provenance-aware selective recomputation and active remeasurement are stronger simple baselines than flag-only provenance or random sample recovery. + +Scientific-workflow systems have long used dependency provenance for partial/smart reruns and selective recomputation. The study therefore must not attribute generic dependency-based recomputation to the proposed scientific-state semantics. + +## 1. Global graph orientation + +Every edge is stored in **influence direction**: + +[ + ext{prerequisite/source} ightarrow ext{dependent/affected object}. +] + +The frozen relation labels retain their names, but endpoint semantics are: + +- `MEASURED_UNDER`: metrological state -> measurement; +- `DERIVED_FROM`: source artifact -> derived artifact; +- `UPDATES`: evidence or prior model state -> subsequent model state; +- `SELECTED_BY`: model/policy state -> selected experiment; +- `SUPPORTS`: evidence/model -> supported claim; +- `EXECUTED_ON`: executed experiment/action -> resulting physical/sample state. + +For a measurement of a retained sample, sample -> measurement may use `DERIVED_FROM` at v0.1.4. No new edge class is added solely to improve primary results after outcome inspection. + +The label names are relation identifiers; arrow direction is always the source-to-dependent convention above even where ordinary English would phrase the inverse relation more naturally. + +## 2. Validity propagation + +Define the validity-propagating edge set + +[ +E_V= +{ + exttt{MEASURED_UNDER}, + exttt{DERIVED_FROM}, + exttt{UPDATES}, + exttt{SUPPORTS}, + exttt{EXECUTED_ON} +}. +] + +`SELECTED_BY` is intentionally excluded. + +For invalidated or quarantined roots (R), the method-visible validity closure is the reachable set using only (E_V). + +This encodes the central hypothesis that a contaminated scientific model can select a future experiment without automatically making the physical experiment or its resulting sample scientifically invalid. + +## 3. Selection-influence propagation + +Let (C_V) denote the validity closure. + +The selection-influence frontier consists of targets of `SELECTED_BY` edges whose source model/policy node lies in (C_V). + +From that frontier, selection influence propagates through all downstream influence edges, including later model updates and later selection edges. + +An object is `SELECTION_TAINTED` only if: + +1. it lies in the selection-influence closure; and +2. it is not already in the validity closure or assigned a stronger repair disposition. + +This allows a physically valid measurement to be marked as having occurred on a decision path influenced by invalid evidence without declaring the measurement itself false. + +## 4. Recovery disposition + +Primary disposition severity remains: + +[ + exttt{INVALID}> + exttt{RECOMPUTE}> + exttt{SELECTION_TAINTED}> + exttt{UNAFFECTED}. +] + +Operational meaning for the pH-drift family: + +- an original measurement under invalid metrology is `INVALID`; +- a derived observed utility or model state that can be rebuilt after removing/replacing invalid inputs is `RECOMPUTE`; +- an experiment/sample selected by a contaminated model but physically executed correctly is `SELECTION_TAINTED`; +- a remeasurement after restored metrology creates a new measurement node; it does not mutate the old invalid measurement into a valid one. + +Method-visible quarantine may be broader than hidden-oracle invalidity. Scoring compares method classifications against the hidden oracle without exposing oracle onset. + +## 5. Strengthened B2 — provenance-selective recomputation + +B2 is now explicitly: + +**QC + conservative evidence invalidation + provenance-selective computational recomputation.** + +After the alarm: + +1. quarantine/discard suspect-interval pH measurements from the scientific model; +2. identify computational/model/claim descendants using ordinary dependency provenance; +3. recompute only affected computational objects; +4. retain unaffected computational artifacts; +5. perform no physical sample remeasurement. + +This is a strong generic provenance baseline. Any B4 advantage must exceed ordinary selective recomputation. + +## 6. Strengthened active-remeasurement baseline B3U + +Add B3U for each preregistered physical remeasurement budget. + +B3U performs: + +1. the same conservative suspect-interval quarantine as B2; +2. fit the same registered GP model to remaining admitted safe measurements; +3. among retained physical samples in the suspect interval, choose remeasurement candidates sequentially by maximum posterior predictive variance at the sample's original (x); +4. after each remeasurement, refit before choosing the next candidate; +5. rebuild the model after the budget is exhausted. + +Tie break: lowest original experiment index. + +B3U uses no typed influence semantics beyond the ordinary suspect interval and retained-sample inventory. + +This baseline tests whether any B4 benefit is simply active experimental design during recovery. + +## 7. B4 budgeted remeasurement rule + +The previously provisional phrase “greedy expected model-state discrepancy reduction” is replaced by a frozen operational rule. + +B4: + +1. computes method-visible validity and selection-influence closures; +2. identifies retained physical samples whose original measurement is quarantined/invalid while the physical sample itself remains remeasurable; +3. fits the same safe-data GP as B3U; +4. sequentially chooses eligible remeasurement candidates by maximum posterior predictive variance; +5. uses the same budget and tie break as B3U; +6. creates new remeasurement/evidence nodes and recomputes only graph-affected scientific objects. + +Thus B3U and B4 share the active-selection heuristic. Their difference is typed dependency/recovery semantics, not a stronger sample-selection algorithm. + +## 8. Interpretation gate for the first failure family + +The metrological pH-drift family may legitimately yield B3U approximately equal to B4 on model recovery. Such a result would show that typed semantics add little in a single-layer retained-sample setting. + +A broad paper claim that typed scientific-state semantics improve recovery efficiency requires at least one additional preregistered failure family with a **different minimal repair action**, such as: + +- computational/analysis invalidation where raw measurement is reusable and recomputation is sufficient; or +- physical/sample invalidation where remeasurement of the same artifact is insufficient and re-execution is required. + +The first family alone may establish the problem, metrics, graph semantics, and delayed-invalidation phenomenon, but it cannot by itself justify a cross-layer recovery-superiority claim. + +## 9. Additional kill criterion + +If a generic provenance-selective recomputation baseline plus the same active remeasurement heuristic reproduces B4's classifications and recovery decisions across multiple failure families with equivalent complexity, the typed scientific-state layer has not earned a distinct technical contribution. + +## 10. Benchmark portability + +All graph/recovery code must depend on typed nodes, edges, validity states, and repair capabilities rather than pH-specific identifiers, drift parameters, or experiment indices. + +Failure-family-specific code may construct the graph and provide node capabilities; the recovery engine itself must remain domain-neutral. diff --git a/research/delayed_evidence_invalidation/PREREGISTRATION.md b/research/delayed_evidence_invalidation/PREREGISTRATION.md new file mode 100644 index 00000000..cf0c07b2 --- /dev/null +++ b/research/delayed_evidence_invalidation/PREREGISTRATION.md @@ -0,0 +1,495 @@ +# Preregistration v0.1 — Delayed Evidence Invalidation and Scientific-State Recovery + +**Registration status:** frozen before implementation of the new benchmark task and before inspection of primary comparative outcomes +**Frozen on:** 2026-10-08 +**Repository base:** `fraware/LabTrust-Gym@7e361583fb3e8ca4f482b072f4ac190e30cc69a5` +**Study branch:** `research/delayed-evidence-invalidation-v0` + +## 1. Research question + +A continuously adaptive laboratory repeatedly executes the loop + +[ +B_t ightarrow a_t ightarrow e_t ightarrow B_{t+1}, +] + +where (B_t) is scientific state, (a_t) is the next experiment, and (e_t) is newly generated evidence. + +This study asks: + +> When evidence (e_k), previously admitted and used to update scientific state, is invalidated only at a later time ( au>k), which downstream scientific objects remain valid, which require recomputation or qualification, and which recovery strategy best restores a trustworthy scientific state while preserving useful work? + +The first experimental family uses delayed discovery of calibration drift in a simulated pH measurement process. Detection of loss of measurement control is treated as an external metrology event. The contribution under test begins after that event: propagation, classification, and recovery. + +## 2. Prior-art boundary and non-novel components + +The study does **not** claim novelty for: + +- laboratory measurement assurance, check standards, control charts, or calibration-drift detection; +- scientific workflow provenance or generic dependency graphs; +- claim/evidence lineage; +- machine unlearning or the general observation that deleting stored data does not erase learned influence; +- downstream influence in sequential decision systems; +- robust Bayesian optimization under corrupted observations; +- dynamic safety cases or runtime assurance; +- laboratory device interoperability. + +Relevant adjacent work includes NIST measurement-assurance practice; ASTM D6299/E3500 statistical quality-control practice; W3C PROV and scientific-workflow provenance; robust Bayesian optimisation under corruptions; machine unlearning; offline-bandit unlearning; 2026 work on downstream “influence echoes” in self-improving agent networks; dynamic safety assurance; and OPC UA LADS. + +The prospective contribution is narrower: + +> Scientific invalidation in an adaptive physical experiment creates semantically distinct dependency classes—measurement validity, physical/sample lineage, model update, claim support, and experiment-selection dependence. Recovery should reason over those classes separately instead of treating every descendant as equally invalid. + +This statement is a hypothesis to test, not a priority claim. + +## 3. Formal state + +At experiment (t), define + +[ +S_t=(P_t,M_t,E_t,B_t,Pi_t), +] + +where: + +- (P_t): physical/sample state and retained-material inventory; +- (M_t): metrological state, including instrument identity, calibration state, and validity envelope; +- (E_t): admitted evidence and its validity status; +- (B_t): scientific model/belief state; +- (Pi_t): experiment-selection state/policy. + +A campaign also maintains a typed directed influence graph + +[ +G_t=(V,mathcal E). +] + +Nodes may represent samples, measurements, raw signals, calibration states, derived values, model versions, experiment decisions, executed experiments, and claims. + +Frozen edge types for v0.1: + +- `MEASURED_UNDER`: measurement depends on instrument/calibration state; +- `DERIVED_FROM`: derived scientific object depends computationally or analytically on another object; +- `UPDATES`: evidence contributed to a model/belief-state update; +- `SELECTED_BY`: experiment choice was produced by a model/policy state; +- `SUPPORTS`: evidence/model supports a scientific claim; +- `EXECUTED_ON`: action or measurement applies to a specific physical sample/material lineage. + +No edge type may be added to improve the primary result after primary outcomes are inspected. New types require a versioned protocol revision or secondary analysis. + +## 4. Evidence validity states + +Each evidence object carries a time-dependent state + +[ +v_i(t)in +{ + exttt{provisional}, + exttt{admitted}, + exttt{quarantined}, + exttt{invalidated}, + exttt{superseded} +}. +] + +An invalidation record is + +[ +I=(e_i, au,r,s,mathcal D), +] + +with discovery time ( au), reason (r), affected validity scope (s), and supporting invalidation evidence (mathcal D). + +The primary implementation must distinguish: + +### Validity closure + +(C_V(e)): descendants whose warrant or numerical value depends on invalid evidence through `MEASURED_UNDER`, `DERIVED_FROM`, `UPDATES`, or `SUPPORTS`. + +### Selection-influence closure + +(C_Pi(e)): experiments or artifacts whose existence/selection depends on contaminated policy/model states through `SELECTED_BY`. + +The two closures are evaluated separately. + +## 5. Descendant classification + +Every downstream object receives exactly one primary recovery disposition: + +- `INVALID`: its own scientific warrant directly depends on invalid evidence and no registered repair is available; +- `RECOMPUTE`: its source evidence remains available in a form that supports deterministic or registered recomputation; +- `SELECTION_TAINTED`: its creation/selection was influenced by invalid scientific state, while its own physical execution or measurement remains independently admissible; +- `UNAFFECTED`: no material dependency on the invalidated evidence exists. + +If more than one label appears applicable, severity order is: + +[ + exttt{INVALID} > + exttt{RECOMPUTE} > + exttt{SELECTION_TAINTED} > + exttt{UNAFFECTED}. +] + +This rule is frozen for v0.1. + +## 6. Experimental substrate + +### 6.1 Instrument layer + +Primary instrument semantics follow the open OPC UA LADS reference pH-meter simulator. The simulator exposes: + +- a Nernst-equation signal model; +- temperature compensation; +- pH slope and offset; +- raw signal; +- calibration values and calibration methods; +- simulated deliberate slope/offset detuning. + +The benchmark integration may reproduce these semantics in Python for deterministic offline runs. If the TypeScript LADS server is used directly, the study must record its commit and preserve the same benchmark-level state variables. + +### 6.2 Scientific response surface + +The first study uses a synthetic, deterministic formulation-response surface. This surface is intentionally a benchmark function, not a claim of chemical realism. + +Input: + +[ +xin[0,1]. +] + +True property: + +[ +pH^*(x)=7.0+1.20sin(2pi x)+0.35cos(6pi x)+0.25x. +] + +Scientific objective: + +[ +u(x)=-(pH^*(x)-7.40)^2. +] + +Measurement noise is introduced only through the instrument model. + +The benchmark must expose the latent (pH^*(x)) only to the hidden adjudicator, never to the optimizer or recovery policy. + +### 6.3 Adaptive policy + +Primary policy: Gaussian-process upper-confidence-bound Bayesian optimization. + +If the repository lacks an acceptable GP implementation without adding a heavy core dependency, the benchmark may place the BO implementation behind a study-only optional dependency. The exact package/version and kernel are frozen before primary campaign execution. + +A robust-BO comparator must be included. Preferred comparator: RCGP-UCB or the closest reproducible public implementation from Ezzerg, Bogunovic, and Knoblauch (ICML 2026). If exact reuse proves impractical, the deviation must be documented before results are inspected. + +### 6.4 Campaign horizon + +- total experiments: (T=240); +- initial non-adaptive design: 16 space-filling points; +- adaptive experiments: 224; +- one retained physical sample record per experiment; +- check-standard event every 20 experiment slots; +- primary reference check: pH 7.00 at fixed temperature. + +The check-standard operations are instrumentation events and do not count toward the 240 scientific experiments. + +## 7. Calibration-only pre-study + +Before primary campaign generation, run a calibration-only noise characterization at pH 7.00 with no drift. + +- readings: 1,000; +- fixed calibration-study seed: 20261008; +- result: (hatsigma_0), the baseline standard deviation of the reported pH measurement. + +No adaptive scientific campaign data are included in this estimate. + +All primary drift magnitudes are expressed relative to (hatsigma_0), so the protocol remains portable across simulator implementations. + +## 8. Drift process + +Primary failure: gradual offset drift. + +For each campaign seed: + +1. drift onset (k) is sampled uniformly from experiment indices 60 through 120 using a hidden failure RNG stream; +2. after onset, offset grows linearly; +3. drift rate is drawn from the frozen set + [ + {0.075,0.125,0.200}hatsigma_0 + ] + per scientific experiment; +4. the optimizer does not observe onset or drift parameters. + +Secondary ablation: slope drift with matched standardized effect size. + +No combined slope+offset result may replace the offset-drift primary result. + +## 9. Measurement-control rule + +At each scheduled check-standard event, obtain five repeated readings. + +Let (ar y_c) be the mean check reading. Trigger an out-of-control event if + +[ +|ar y_c-7.00| > 3hatsigma_0/sqrt{5}. +] + +The immediately preceding check event that satisfied the control rule is the last-known-in-control checkpoint. + +Primary suspect interval: + +[ +(t_{mathrm{last good}},t_{mathrm{detect}}]. +] + +This deliberately conservative interval is supplied to recovery methods that do not estimate a more precise onset. + +The hidden oracle retains the true onset for evaluation only. + +## 10. Stored-sample assumption + +Primary study assumes physical samples remain available and stable for remeasurement through the campaign horizon. + +This assumption is explicit because it creates a scientifically meaningful recovery action: remeasure a previously generated sample after restoring instrument control. + +Secondary ablation will impose sample expiry windows. No general claim about recoverability is permitted without reporting this ablation. + +## 11. Recovery conditions + +### B0 — no retroactive recovery + +The control alarm affects future operation only. Previously admitted evidence and model state remain unchanged. + +### B1 — provenance flagging + +The suspect interval is marked in the execution record. No model reconstruction occurs. + +### B2 — QC + discard + retrain + +All pH observations in the conservative suspect interval are removed. The scientific model is retrained on retained observations. No stored sample is remeasured. + +### B3 — QC + blanket remeasurement + retrain + +All retained samples from the conservative suspect interval are remeasured after instrument recovery, then the model is rebuilt. + +This is a deliberately strong recovery baseline. + +### B4 — typed dependency-aware recovery + +The influence graph classifies affected artifacts, invalidates or recomputes dependent objects, and selects stored samples for remeasurement under the same recovery budget used by the matched B3 budgeted condition. + +B4 may use graph structure and model-state information available before hidden adjudication. It may not use true drift onset, true latent pH, or hidden labels. + +### B5 — robust BO + B2 recovery + +Robust BO operates during acquisition, followed by the same conservative discard/retrain procedure as B2 after control failure is detected. + +This tests whether robustness during optimization makes the recovery layer unnecessary. + +## 12. Recovery budgets + +Two analyses are preregistered. + +### Exact/upper-bound recovery + +B3 remeasures the full suspect interval. B4 may also use the full interval. This establishes the best attainable recovery under retained-sample access. + +### Budgeted recovery + +Remeasurement budgets are: + +[ +bin{0.10,0.25,0.50} +] + +of samples in the conservative suspect interval, rounded up. + +For B3 budgeted recovery, sample selection is uniform random with a dedicated RNG stream. + +For B4, the selection rule must be frozen before primary outcomes. The intended rule is greedy expected reduction in model-state discrepancy estimated from the observed model and influence graph. If implementation requires a different criterion, it must be versioned before outcome inspection. + +## 13. Hidden clean counterfactual + +For each campaign seed, run a clean campaign with identical initial design, policy hyperparameters, non-failure RNG stream, and no drift. + +Because adaptive action sequences diverge after corruption, the clean campaign is not treated as a pointwise paired trajectory after divergence. It is used as an adjudication reference. + +The latent response surface also permits direct evaluation against physical ground truth independently of the clean policy trajectory. + +## 14. Primary outcomes + +### O1 — post-recovery scientific-model error + +On a fixed dense evaluation grid (mathcal X_{eval}), measure normalized integrated squared prediction error against (pH^*(x)). + +Primary comparison: B4 versus B2 and B3. + +### O2 — retained scientifically valid work + +Fraction of valid physical experiments / revalidated measurements retained in the recovered campaign state. + +### O3 — recovery cost + +Count of remeasurements plus normalized computational reconstruction cost. Physical remeasurement count is reported separately and receives priority over compute in interpretation. + +### O4 — validity-classification accuracy + +Against hidden oracle dependencies, macro-F1 over `INVALID`, `RECOMPUTE`, `SELECTION_TAINTED`, and `UNAFFECTED`. + +No model-performance result substitutes for failure to classify dependencies correctly. + +## 15. Secondary outcomes + +- detection latency from true drift onset to control alarm; +- cumulative optimization regret; +- number of invalid observations admitted before alarm; +- validity-closure size; +- selection-influence-closure size; +- policy-path divergence from clean reference; +- number of scientific claims requiring withdrawal/recomputation; +- wall-clock recovery time; +- sample remeasurement efficiency; +- false invalidation of unaffected artifacts. + +The following programme constructs are exploratory until validated: + +- validity contamination depth; +- selection influence depth; +- scientific-state recovery fidelity; +- scientific waste. + +They must not be presented as established community metrics. + +## 16. Primary hypotheses + +### H1 — delayed invalidation has consequences beyond a QC alarm + +B0/B1 retain materially higher post-recovery model error or unsupported downstream state than B2–B4. + +### H2 — dependency semantics preserve useful science + +At matched model-error tolerance, B4 retains more scientifically valid work or requires fewer remeasurements than full rollback / blanket recovery. + +### H3 — robust acquisition does not eliminate retroactive recovery + +B5 reduces sensitivity to corrupted observations but does not, by itself, resolve provenance/claim invalidation or recover the correct dependency state after a late control failure. + +### H4 — selection dependence differs from validity dependence + +There exist campaigns containing artifacts correctly classified as `SELECTION_TAINTED` but not `INVALID`. + +Failure to produce such cases in the primary family weakens the central semantic claim and requires either a second failure family or narrowing of the paper. + +## 17. Statistical design + +Primary Monte Carlo unit: campaign seed. + +Initial target: 200 seeds per drift-rate condition, for 600 campaigns per method before recovery-budget expansion. + +All methods operate on the same campaign seed and hidden failure realization where their acquisition semantics permit paired comparison. + +Primary uncertainty reporting: + +- bootstrap 95% confidence intervals over campaign seeds; +- paired bootstrap differences where trajectories share the same pre-divergence seed; +- effect sizes alongside p-values; +- full empirical distributions for heavy-tailed outcomes. + +No claim rests solely on statistical significance. + +A calibration run with 20 seeds per condition is permitted solely to verify code, runtime, nondegenerate drift detection, and absence of ceiling/floor effects. Those seeds are permanently excluded from primary analysis. + +Any sample-size change after the 20-seed calibration must be justified from runtime/variance estimates and frozen before primary results are inspected. + +## 18. Ablations required before broad claims + +1. slope drift instead of offset drift; +2. different check intervals: 10, 20, 40; +3. stored-sample expiry / unavailable sample recovery; +4. different BO acquisition policy; +5. different latent response surface; +6. graph-edge removal, one type at a time; +7. incorrect or incomplete dependency metadata; +8. false-positive QC alarm; +9. multiple separate invalidation events. + +A paper may report fewer ablations only if its claims are narrowed accordingly. + +## 19. Kill criteria + +The richer architecture is not supported if any of the following survives adequate testing: + +1. B2 (QC + discard + retrain) matches B4 on model error, retained valid work, and recovery cost within practically negligible differences. +2. B3 blanket recovery dominates B4 across the recovery Pareto frontier. +3. validity and selection closures collapse to the same set in realistic benchmark variants. +4. B4's advantage depends on hidden information unavailable to a deployed system. +5. the result disappears under a second latent response surface or slope-drift family. +6. results depend on benchmark-specific identifiers or hand-coded knowledge of the injected failure. +7. a simpler generic provenance traversal reproduces B4 behavior with equivalent effort and outputs. +8. the LADS-aligned benchmark semantics materially differ from the reference implementation in ways that drive the result. + +A null result must be reported as a programme-narrowing result, not repaired by weakening baselines after inspection. + +## 20. Reproducibility requirements + +Every released run must record: + +- repository commit; +- study-contract digest; +- dependency versions; +- Python version; +- seed streams; +- latent-surface version; +- instrument-model version; +- drift realization; +- check-standard observations; +- recovery method configuration; +- influence graph before and after invalidation; +- raw and derived metrics. + +Primary figures and tables must rebuild from released run artifacts. + +The existing LabTrust paper-claims regression and release-pack machinery should be reused where practical. + +## 21. External-validity ladder + +Claims are tiered. + +### Tier S0 — benchmark semantics + +The implementation behaves as registered inside the deterministic benchmark. + +### Tier S1 — standards-aligned simulation + +Results reproduce through the OPC UA LADS-compatible instrument semantics. + +### Tier S2 — second failure family / second substrate + +The recovery semantics transfer to a non-metrological invalidation mechanism or another open autonomous-lab stack. + +### Tier S3 — real laboratory shadow evaluation + +The system processes authentic run artifacts without controlling live scientific actions. + +### Tier S4 — prospective bounded pilot + +Recovery outputs influence a real laboratory workflow under an independently approved operating envelope. + +The first paper should claim no higher tier than the evidence actually achieved. + +## 22. Implementation ownership + +- **LabTrust-Gym:** campaign simulator, hidden oracle, failures, stored samples, benchmark runner, metrics. +- **EnvAssure:** metrological/environment validity semantics and validity-envelope transitions. +- **Scientific Memory:** persistent claim/evidence/model dependency records and typed influence graph. +- **AKTA:** consumes scientific-state disposition when deciding whether future action is admissible. +- **VerifierLab:** adversarial/repair campaign methodology and comparative qualification. + +No new repository is authorized by this preregistration. + +## 23. Planned first paper claim + +The aspirational claim to test is: + +> Delayed invalidation in adaptive autonomous experimentation creates scientifically distinct validity and selection dependencies. A typed, dependency-aware recovery process can restore scientific state while preserving more legitimate experimental work than coarse rollback or provenance/QC alone. + +This sentence is not a supported result until the registered comparisons succeed. diff --git a/research/delayed_evidence_invalidation/README.md b/research/delayed_evidence_invalidation/README.md new file mode 100644 index 00000000..fb66c12c --- /dev/null +++ b/research/delayed_evidence_invalidation/README.md @@ -0,0 +1,42 @@ +# Delayed Evidence Invalidation Study + +**Status:** preregistration v0.1.4 effective / no primary results inspected +**Branch:** `research/delayed-evidence-invalidation-v0` +**Base commit:** `7e361583fb3e8ca4f482b072f4ac190e30cc69a5` + +This directory freezes the first study in the long-horizon scientific-state assurance programme. + +## Research object + +The study asks what a continuously adaptive laboratory should do when evidence previously admitted into its scientific state is later found invalid. + +The central distinction is between: + +- **validity dependence**: an artifact, model, or claim depends on evidence whose own validity has failed; and +- **selection dependence**: an experiment was selected by a scientific state influenced by invalid evidence, although the experiment or later measurement may remain scientifically usable. + +The first benchmark uses delayed calibration-drift discovery in an OPC UA LADS-aligned pH-measurement loop. Metrology owns detection that the measurement process has left control. The study begins at that detection event and evaluates propagation and recovery. + +## Frozen artifacts + +- `PREREGISTRATION.md` — original v0.1 hypotheses, prior-art boundary, design, baselines, outcomes, analysis, ablations, and kill criteria. +- `AMENDMENT_0.1.1.md` — pre-outcome amendment separating physical drift onset from scientific invalidity onset. +- `AMENDMENT_0.1.2.md` — pre-outcome amendment freezing indexing, event ordering, drift sign, and false-alarm handling. +- `AMENDMENT_0.1.3.md` — pre-outcome amendment freezing the optimizer objective, GP-UCB profile, RCGP comparator, and non-stationarity controls. +- `AMENDMENT_0.1.4.md` — pre-outcome amendment freezing influence-graph orientation and strengthening provenance/active-recovery baselines. +- `study_contract.v0.1.yaml` — historical machine-readable v0.1 contract. +- `study_contract.v0.1.1.yaml` — historical amended contract. +- `study_contract.v0.1.2.yaml` — historical amended contract. +- `study_contract.v0.1.3.yaml` — historical amended contract. +- `study_contract.v0.1.4.yaml` — **effective** machine-readable contract for implementation and analysis. +- `REFERENCES.md` — literature and standards that define the prior-art boundary. + +## Claim discipline + +This study is a simulation / reference-integration study. It does not establish production laboratory safety, clinical validity, general real-world failure rates, or a universal theory of scientific recovery. Those claims require external laboratory data and independent replication. + +No implementation decision may silently weaken a frozen baseline, outcome definition, or kill criterion. Any protocol change after primary outcomes are inspected must be versioned and reported as post-registration. + +## Effective protocol + +Implementation and primary analysis must follow `study_contract.v0.1.4.yaml` together with amendments 0.1.1–0.1.4. Earlier versions remain immutable historical records. diff --git a/research/delayed_evidence_invalidation/REFERENCES.md b/research/delayed_evidence_invalidation/REFERENCES.md new file mode 100644 index 00000000..c9927eef --- /dev/null +++ b/research/delayed_evidence_invalidation/REFERENCES.md @@ -0,0 +1,118 @@ +# Prior-art and standards boundary + +This file records sources that constrain novelty and benchmark design. It is not a claim that every source directly studies delayed evidence invalidation in autonomous laboratories. + +## Autonomous laboratories and scientific-agent assurance + +1. Canty, R. B.; Abolhasani, M. **The past, present and future of self-driving laboratories.** *Nature Reviews Chemistry* 10, 523–537 (2026). DOI: 10.1038/s41570-026-00847-2. + Relevance: scalability, generalizability, and provenance-complete experimentation as interdependent SDL requirements. + +2. Chen, L.; Li, X.; Lin, Q.; et al. **Self-driving laboratories need an autonomy safety harness.** *Nature Synthesis* (2026). DOI: 10.1038/s44160-026-01120-6. + Relevance: AI intent, executable experiments, monitored actions, and trustworthy downstream evidence. + +3. Yin, X.; Du, M.; Prince, M. H.; Cherukara, M. J. **Artifact-centered Claim-aware Observability for Autonomous Scientific Agents.** arXiv:2608.18312 (2026). + Relevance: claim-aware artifact lineage; prevents this study from claiming generic claim/evidence provenance as novel. + +4. Bi, J.; Pinilla, S.; Zhu, C. **Certifying when decision-time information justifies adaptive experimentation.** arXiv:2607.27651 (2026). + Relevance: authorization is a distinct layer in adaptive science; prevents conflation of recovery with initial authorization. + +5. Alnasir, J. J. **AI-Driven Scientific Computing Workflows: A Systems Review of Orchestration, Execution, Reproducibility and Provenance.** arXiv:2609.21162 (2026). + Relevance: identifies state-aware recovery, model-mediated decision provenance, and reproducible adaptive execution as open systems problems. + +## Metrology and quality control + +6. National Institute of Standards and Technology. **Measurement Assurance Programs / Calibration Policies** and **Measurement Assurance Basics**. + Relevance: periodic check standards, uncertainty estimation, statistical control, and continuous monitoring of measurement-process quality. + +7. ASTM D6299-26. **Standard Practice for Applying Statistical Quality Assurance and Control Charting Techniques to Evaluate Analytical Measurement System Performance.** + Relevance: ongoing monitoring of precision/bias and analytical measurement system state of control. + +8. ASTM E3500-25. **Standard Practice for Quality Control of Routine Testing in a Laboratory.** + Relevance: control samples, drift detection, out-of-control action plans, and acceptance/evaluation of sample results. + +## Robust adaptive optimization and inference + +9. Ezzerg, A.; Bogunovic, I.; Knoblauch, J. **Robust Bayesian Optimisation with Unbounded Corruptions.** ICML 2026, PMLR 306:28528–28565. + Relevance: strong comparator for acquisition robustness under corrupted observations; robust acquisition does not itself specify scientific invalidation semantics. + +10. Adaptive-experiment and sequential-inference literature reviewed in: + **The Blessings and Curses of Adaptivity in Sequential Experiments** (Annual Review of Statistics and Its Application). + Relevance: adaptively collected data require explicit inferential treatment; later observations are not automatically equivalent to a fixed-design sample. + +## Unlearning and downstream influence + +11. Bourtoule, L.; et al. **Machine Unlearning.** arXiv:1912.03817. + Relevance: data influence persists in trained models after storage deletion; retraining/structured recovery baselines. + +12. Ye, Z.; Wang, R.; Wang, X.; Liu, X.; Li, S.; Hajiesmaili, M. **Unlearning Offline Stochastic Multi-Armed Bandits.** arXiv:2605.00638 (2026). + Relevance: unlearning in sequential decision-making and rollback/utility trade-offs. The present study must not claim that unlearning in sequential decision systems is new. + +13. Ding, Z.; Huang, J.; Dong, L. **When Unlearning Fails: Reliable Data Deletion under Post-Training in Agent Networks.** arXiv:2607.28829 (2026). + Relevance: especially close adjacent work. It identifies downstream “influence echoes” when target data have already shaped later retained trajectories. This prevents any priority claim for the generic idea that corrupted/deleted data influence later collected trajectories. + +14. Nguyen, T. T.; et al. **A Survey of Machine Unlearning.** *ACM Transactions on Intelligent Systems and Technology* (2025). DOI: 10.1145/3749987. + Relevance: unlearning scope, streaming removal, retained utility, and exact-versus-approximate removal. + +## Runtime/dynamic assurance + +15. **Safe autonomous systems in a changing world: Operationalising dynamic safety cases.** *Safety Science* 191, 106965 (2025). DOI: 10.1016/j.ssci.2025.106965. + Relevance: monitorability, assessability, and updateability of assurance arguments at runtime. + +16. Herd, B.; Kelly, J.; Heinemann, C.; Zacchi, J.-V. **A Subjective Logic-based method for runtime confidence updates in safety arguments.** arXiv:2605.22530 (2026). + Relevance: runtime evidence updates confidence in assurance claims; the present work must distinguish scientific validity propagation from generic assurance-case updating. + +## Laboratory interoperability substrate + +17. OPC Foundation / SPECTARIS. **OPC UA Laboratory and Analytical Device Standard (LADS), Companion Specification 30500.** + Relevance: standards-aligned laboratory device state, measurements, alarms/events, methods, and result semantics. + +18. `opcua-lads/lads-server-collection`, pH-meter reference simulator (2025–2026). + Relevance: Nernst-equation signal model, temperature compensation, slope/offset detuning, raw signal, calibration methods, AFO metadata, and ASM result payloads. The repository explicitly describes the servers as proof-of-concept simulators for demos/integration/reference use. + +## Novelty discipline + +The study should avoid priority claims until a systematic literature review and external expert review are complete. + +The most defensible provisional distinction is scientific-semantic: + +- metrological invalidity is not the same relation as computational derivation; +- claim invalidity is not the same relation as experiment-selection influence; +- an experiment selected by a contaminated model may still produce a physically valid artifact or later measurement; +- recovery should preserve these distinctions and be evaluated against strong rollback, retraining, robust-optimization, and blanket-remeasurement baselines. + +The July 2026 “influence echo” work in agent networks is the closest conceptual warning found so far. Any paper must cite and distinguish it explicitly. + + +## Non-stationary and robust Bayesian optimization + +19. Bogunovic, I.; Scarlett, J.; Cevher, V. **Time-Varying Gaussian Process Bandit Optimization.** AISTATS 2016, PMLR 51:314–323. + Relevance: establishes that classical GP-UCB can underperform when stale and fresh observations are treated equally; proposes resetting and smooth-forgetting GP-UCB variants. + +20. Deng, Y.; Zhou, X.; Kim, B.; Tewari, A.; Gupta, A.; Shroff, N. **Weighted Gaussian Process Bandits for Non-stationary Environments.** AISTATS 2022, PMLR 151:6909–6932. + Relevance: develops weighted GP regression/UCB for drifting nonlinear reward functions and provides a stronger adjacent non-stationarity baseline family. + +21. Kim, G.; Sergi, F. **Validation of Dynamic Bayesian Optimization for a Non-Stationary Human-in-The-Loop Optimization Problem.** *IEEE Robotics and Automation Letters* 11(5):5733–5740 (2026). DOI: 10.1109/LRA.2026.3665072. + Relevance: prospective physical-system validation under deliberately induced gradual non-stationarity; dynamic BO better tracked the changing input-output relationship than conventional BO. + +22. Ezzerg, A.; Bogunovic, I.; Knoblauch, J. **Robust Bayesian Optimisation with Unbounded Corruptions.** ICML 2026, PMLR 306:28528–28565. + Relevance: RCGP-UCB is a strong robustness baseline for frequency-constrained unbounded corruptions. Its assumptions do not automatically cover persistent gradual calibration drift. Official software is linked from PMLR; the released repository commit frozen for this study is `68bd978dfe4de28aed02184efc64ea343cfa9afc`. + +### Comparator discipline + +The primary study now includes both: +- a corruption-robust RCGP-based comparator; and +- sliding-window GP-UCB controls for non-stationarity. + +Weighted/time-input dynamic BO methods remain important adjacent work. Their omission from the primary v0.1.3 benchmark is an implementation-scope decision, not evidence that they are weaker or irrelevant. Broad optimizer-independent claims require later sensitivity to additional non-stationarity methods. + + +## Selective recomputation and provenance-based recovery + +23. Cała, J.; Missier, P. **Selective and Recurring Re-computation of Big Data Analytics Tasks: Insights from a Genomics Case Study.** *Big Data Research* 13:76–94 (2018). DOI: 10.1016/j.bdr.2018.06.001. + Relevance: provenance/process-history metadata can identify affected outcomes and selectively recompute only impacted workflow fragments. This is a required strong baseline, not a novel contribution of the present study. + +24. Kepler scientific-workflow provenance / Smart Rerun work. + Relevance: provenance caches and dependency analysis support efficient partial reruns by replaying unaffected intermediate products. Generic “dependency graph implies selective rerun” must not be claimed as new. + +25. Leo, S.; et al. **Recording provenance of workflow runs with RO-Crate.** *PLOS ONE* 19(9):e0309210 (2024). + Relevance: interoperable workflow-run provenance and partial-rerun support; reinforces the distinction between generic computational lineage and the proposed physical/metrological/selection semantics. diff --git a/research/delayed_evidence_invalidation/study_contract.v0.1.1.yaml b/research/delayed_evidence_invalidation/study_contract.v0.1.1.yaml new file mode 100644 index 00000000..b34d1fa9 --- /dev/null +++ b/research/delayed_evidence_invalidation/study_contract.v0.1.1.yaml @@ -0,0 +1,284 @@ +schema_version: delayed_evidence_invalidation.v0.1.1 +status: preregistered_amended +frozen_date: "2026-10-08" +repository: + name: fraware/LabTrust-Gym + base_commit: 7e361583fb3e8ca4f482b072f4ac190e30cc69a5 + branch: research/delayed-evidence-invalidation-v0 + +study: + id: dei_ph_drift_v0_1_1 + title: Delayed Evidence Invalidation and Scientific-State Recovery + mode: simulation + primary_failure_family: metrological_delayed_invalidation + no_new_repository: true + +state_model: + components: + - physical_state + - metrological_state + - evidence_state + - scientific_model_state + - selection_policy_state + influence_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SELECTED_BY + - SUPPORTS + - EXECUTED_ON + validity_states: + - provisional + - admitted + - quarantined + - invalidated + - superseded + descendant_dispositions: + severity_order: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +substrate: + instrument_semantics: opcua_lads_ph_meter_reference + reference_repository: opcua-lads/lads-server-collection + reference_component: servers/lads-ph-meter + required_features: + - nernst_signal_model + - temperature_compensation + - slope + - offset + - raw_signal + - calibration_values + - calibration_methods + note: > + Python reimplementation is permitted for deterministic offline campaigns only if + benchmark-level semantics are checked against the reference implementation. + +scientific_surface: + domain: + x_min: 0.0 + x_max: 1.0 + true_property: + expression: "7.0 + 1.20*sin(2*pi*x) + 0.35*cos(6*pi*x) + 0.25*x" + hidden_from_policy: true + target_ph: 7.40 + utility: + expression: "-(true_ph - 7.40)^2" + realism_claim: false + +campaign: + scientific_experiments: 240 + initial_design_points: 16 + adaptive_experiments: 224 + retained_sample_per_experiment: true + check_interval_experiments: 20 + check_replicates: 5 + check_reference_ph: 7.00 + +noise_calibration: + seed: 20261008 + readings: 1000 + reference_ph: 7.00 + output: sigma0 + excluded_from_primary_analysis: true + +failure: + type: gradual_offset_drift + onset: + distribution: discrete_uniform + min_experiment: 60 + max_experiment: 120 + hidden_from_methods: true + standardized_rates_per_experiment: + - 0.075 + - 0.125 + - 0.200 + rate_units: sigma0 + secondary_failure: + type: slope_drift + match_standardized_effect: true + +application_validity: + primary_tolerance_sigma0: 1.0 + systematic_bias_definition: "E[reported_ph | true_ph, metrological_state] - true_ph" + scientific_invalidity_onset: "first t >= physical_drift_onset with abs(systematic_bias(t)) > tolerance" + physical_drift_alone_implies_invalidity: false + oracle_only: true + sensitivity_tolerance_sigma0: + - 0.5 + - 1.0 + - 2.0 + production_metrology_claim: false + +qc: + rule: shewhart_mean_threshold + threshold_expression: "abs(mean_check - 7.00) > 3*sigma0/sqrt(5)" + suspect_interval: "(last_in_control_check, detection_check]" + true_physical_drift_onset_available_to_hidden_adjudicator_only: true + true_scientific_invalidity_onset_available_to_hidden_adjudicator_only: true + suspect_interval_semantics: operational_bracketing_not_true_onset + +adaptive_policy: + primary: gp_ucb + exact_implementation: freeze_before_primary_runs + robust_comparator: + preferred: rcgp_ucb + reference: "Ezzerg, Bogunovic, Knoblauch, ICML 2026" + deviation_requires_preoutcome_protocol_update: true + +recovery_methods: + B0: + name: no_retroactive_recovery + retroactive_change: false + B1: + name: provenance_flagging + rebuild_model: false + B2: + name: qc_discard_retrain + discard_suspect_ph: true + remeasure_samples: false + B3: + name: qc_blanket_remeasure_retrain + exact_mode: + remeasure_fraction: 1.0 + budgeted_selection: uniform_random + B4: + name: typed_dependency_aware_recovery + hidden_oracle_access: false + exact_mode: + max_remeasure_fraction: 1.0 + budgeted_selection: + intended_rule: greedy_expected_model_discrepancy_reduction + freeze_before_primary_runs: true + B5: + name: robust_bo_plus_B2 + acquisition_robustness: true + recovery_semantics: B2 + +recovery_budgets: + fractions: + - 0.10 + - 0.25 + - 0.50 + rounding: ceil + +primary_outcomes: + O1: + name: post_recovery_model_error + measure: normalized_integrated_squared_error + reference: latent_true_surface + O2: + name: retained_valid_work + measure: fraction_valid_physical_experiments_or_revalidated_measurements_retained + O3: + name: recovery_cost + measures: + - remeasurement_count + - normalized_compute_cost + interpretation_priority: remeasurement_count + O4: + name: validity_classification_accuracy + measure: macro_f1 + classes: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +secondary_outcomes: + - physical_drift_detection_latency + - scientific_invalidity_detection_latency + - cumulative_optimization_regret + - invalid_observations_admitted_from_scientific_invalidity_onset + - validity_closure_size + - selection_influence_closure_size + - policy_path_divergence + - claims_withdrawn_or_recomputed + - wall_clock_recovery_time + - remeasurement_efficiency + - false_invalidation_count + +hypotheses: + H1: delayed_invalidity_requires_retroactive_recovery + H2: dependency_semantics_preserve_valid_science + H3: robust_acquisition_does_not_replace_recovery + H4: selection_dependence_differs_from_validity_dependence + +statistics: + monte_carlo_unit: campaign_seed + target_seeds_per_drift_rate_per_method: 200 + drift_rate_conditions: 3 + initial_total_campaigns_per_method_before_budget_expansion: 600 + calibration_seeds_per_condition: 20 + calibration_seeds_excluded_from_primary: true + confidence_intervals: + method: bootstrap + level: 0.95 + paired_bootstrap_when_valid: true + report_effect_sizes: true + significance_alone_sufficient: false + sample_size_change: + allowed_before_primary_outcome_inspection: true + requires_variance_or_runtime_justification: true + +required_ablations: + - slope_drift + - check_interval_10_20_40 + - sample_expiry + - alternate_acquisition_policy + - alternate_latent_surface + - edge_type_removal + - incomplete_dependency_metadata + - false_positive_qc_alarm + - multiple_invalidation_events + +kill_criteria: + - "B2 matches B4 on model error, retained valid work, and recovery cost within practically negligible differences." + - "B3 dominates B4 across the recovery Pareto frontier." + - "Validity and selection closures collapse to the same set in realistic variants." + - "B4 advantage depends on hidden information unavailable to deployed systems." + - "Effect disappears under a second response surface or slope-drift family." + - "Effect depends on benchmark-specific identifiers or hand-coded injected-failure knowledge." + - "A simpler generic provenance traversal reproduces B4 with equivalent outputs and effort." + - "LADS-aligned semantics differ from the reference implementation in result-driving ways." + +claim_tiers: + S0: benchmark_semantics + S1: standards_aligned_simulation + S2: second_failure_family_or_substrate + S3: real_lab_shadow_evaluation + S4: prospective_bounded_pilot + +ownership: + LabTrust-Gym: + - campaign_simulator + - hidden_oracle + - failure_injection + - stored_samples + - benchmark_runner + - metrics + EnvAssure: + - metrological_validity_semantics + - environment_validity_transitions + Scientific-Memory: + - typed_influence_graph + - evidence_invalidation_records + - claim_model_dependencies + AKTA: + - consume_scientific_state_disposition_for_action_admissibility + VerifierLab: + - comparative_campaign_methodology + - repair_and_recovery_evaluation + +amendments: + effective: + - file: AMENDMENT_0.1.1.md + reason: distinguish physical drift onset from scientific invalidity onset + timing: before_campaign_implementation_and_before_primary_outcomes + +result_blinding: + implementation_before_primary_results: required + protocol_changes_after_primary_results: + permitted_only_as: post_registration_analysis diff --git a/research/delayed_evidence_invalidation/study_contract.v0.1.2.yaml b/research/delayed_evidence_invalidation/study_contract.v0.1.2.yaml new file mode 100644 index 00000000..3dd6dfd4 --- /dev/null +++ b/research/delayed_evidence_invalidation/study_contract.v0.1.2.yaml @@ -0,0 +1,299 @@ +schema_version: delayed_evidence_invalidation.v0.1.2 +status: preregistered_amended +frozen_date: "2026-10-08" +repository: + name: fraware/LabTrust-Gym + base_commit: 7e361583fb3e8ca4f482b072f4ac190e30cc69a5 + branch: research/delayed-evidence-invalidation-v0 + +study: + id: dei_ph_drift_v0_1_2 + title: Delayed Evidence Invalidation and Scientific-State Recovery + mode: simulation + primary_failure_family: metrological_delayed_invalidation + no_new_repository: true + +state_model: + components: + - physical_state + - metrological_state + - evidence_state + - scientific_model_state + - selection_policy_state + influence_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SELECTED_BY + - SUPPORTS + - EXECUTED_ON + validity_states: + - provisional + - admitted + - quarantined + - invalidated + - superseded + descendant_dispositions: + severity_order: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +substrate: + instrument_semantics: opcua_lads_ph_meter_reference + reference_repository: opcua-lads/lads-server-collection + reference_component: servers/lads-ph-meter + required_features: + - nernst_signal_model + - temperature_compensation + - slope + - offset + - raw_signal + - calibration_values + - calibration_methods + note: > + Python reimplementation is permitted for deterministic offline campaigns only if + benchmark-level semantics are checked against the reference implementation. + +scientific_surface: + domain: + x_min: 0.0 + x_max: 1.0 + true_property: + expression: "7.0 + 1.20*sin(2*pi*x) + 0.35*cos(6*pi*x) + 0.25*x" + hidden_from_policy: true + target_ph: 7.40 + utility: + expression: "-(true_ph - 7.40)^2" + realism_claim: false + +campaign: + experiment_indexing: one_based + scientific_experiments: 240 + initial_design_points: 16 + adaptive_experiments: 224 + retained_sample_per_experiment: true + check_interval_experiments: 20 + check_replicates: 5 + check_reference_ph: 7.00 + check_ordering: immediately_after_scientific_experiment + check_uses_same_current_metrological_state: true + +noise_calibration: + seed: 20261008 + readings: 1000 + reference_ph: 7.00 + output: sigma0 + excluded_from_primary_analysis: true + +failure: + type: gradual_offset_drift + onset: + distribution: discrete_uniform + min_experiment: 60 + max_experiment: 120 + hidden_from_methods: true + primary_direction: positive_physical_offset + primary_reported_bias_direction: negative + standardized_rates_per_experiment: + - 0.075 + - 0.125 + - 0.200 + rate_units: sigma0 + secondary_failure: + type: slope_drift + match_standardized_effect: true + +application_validity: + primary_tolerance_sigma0: 1.0 + systematic_bias_definition: "E[reported_ph | true_ph, metrological_state] - true_ph" + scientific_invalidity_onset: "first t >= physical_drift_onset with abs(systematic_bias(t)) > tolerance" + physical_drift_alone_implies_invalidity: false + oracle_only: true + sensitivity_tolerance_sigma0: + - 0.5 + - 1.0 + - 2.0 + production_metrology_claim: false + +qc: + rule: shewhart_mean_threshold + threshold_expression: "abs(mean_check - 7.00) > 3*sigma0/sqrt(5)" + suspect_interval: "(last_in_control_check, detection_check]" + true_physical_drift_onset_available_to_hidden_adjudicator_only: true + true_scientific_invalidity_onset_available_to_hidden_adjudicator_only: true + suspect_interval_semantics: operational_bracketing_not_true_onset + false_positive_alarm_definition: alarm_before_scientific_invalidity_onset + false_positive_runs_retained: true + primary_recovery_estimand_condition: scientific_invalidity_precedes_first_alarm + report_unconditional_alarm_rate: true + report_fraction_entering_primary_estimand: true + +adaptive_policy: + primary: gp_ucb + exact_implementation: freeze_before_primary_runs + robust_comparator: + preferred: rcgp_ucb + reference: "Ezzerg, Bogunovic, Knoblauch, ICML 2026" + deviation_requires_preoutcome_protocol_update: true + +recovery_methods: + B0: + name: no_retroactive_recovery + retroactive_change: false + B1: + name: provenance_flagging + rebuild_model: false + B2: + name: qc_discard_retrain + discard_suspect_ph: true + remeasure_samples: false + B3: + name: qc_blanket_remeasure_retrain + exact_mode: + remeasure_fraction: 1.0 + budgeted_selection: uniform_random + B4: + name: typed_dependency_aware_recovery + hidden_oracle_access: false + exact_mode: + max_remeasure_fraction: 1.0 + budgeted_selection: + intended_rule: greedy_expected_model_discrepancy_reduction + freeze_before_primary_runs: true + B5: + name: robust_bo_plus_B2 + acquisition_robustness: true + recovery_semantics: B2 + +recovery_budgets: + fractions: + - 0.10 + - 0.25 + - 0.50 + rounding: ceil + +primary_outcomes: + O1: + name: post_recovery_model_error + measure: normalized_integrated_squared_error + reference: latent_true_surface + O2: + name: retained_valid_work + measure: fraction_valid_physical_experiments_or_revalidated_measurements_retained + O3: + name: recovery_cost + measures: + - remeasurement_count + - normalized_compute_cost + interpretation_priority: remeasurement_count + O4: + name: validity_classification_accuracy + measure: macro_f1 + classes: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +secondary_outcomes: + - physical_drift_detection_latency + - scientific_invalidity_detection_latency + - cumulative_optimization_regret + - invalid_observations_admitted_from_scientific_invalidity_onset + - validity_closure_size + - selection_influence_closure_size + - policy_path_divergence + - claims_withdrawn_or_recomputed + - wall_clock_recovery_time + - remeasurement_efficiency + - false_invalidation_count + +hypotheses: + H1: delayed_invalidity_requires_retroactive_recovery + H2: dependency_semantics_preserve_valid_science + H3: robust_acquisition_does_not_replace_recovery + H4: selection_dependence_differs_from_validity_dependence + +statistics: + monte_carlo_unit: campaign_seed + target_seeds_per_drift_rate_per_method: 200 + drift_rate_conditions: 3 + initial_total_campaigns_per_method_before_budget_expansion: 600 + calibration_seeds_per_condition: 20 + calibration_seeds_excluded_from_primary: true + confidence_intervals: + method: bootstrap + level: 0.95 + paired_bootstrap_when_valid: true + report_effect_sizes: true + significance_alone_sufficient: false + sample_size_change: + allowed_before_primary_outcome_inspection: true + requires_variance_or_runtime_justification: true + +required_ablations: + - slope_drift + - check_interval_10_20_40 + - sample_expiry + - alternate_acquisition_policy + - alternate_latent_surface + - edge_type_removal + - incomplete_dependency_metadata + - false_positive_qc_alarm + - multiple_invalidation_events + - drift_sign_reversal + - all_seed_false_alarm_operational_cost + +kill_criteria: + - "B2 matches B4 on model error, retained valid work, and recovery cost within practically negligible differences." + - "B3 dominates B4 across the recovery Pareto frontier." + - "Validity and selection closures collapse to the same set in realistic variants." + - "B4 advantage depends on hidden information unavailable to deployed systems." + - "Effect disappears under a second response surface or slope-drift family." + - "Effect depends on benchmark-specific identifiers or hand-coded injected-failure knowledge." + - "A simpler generic provenance traversal reproduces B4 with equivalent outputs and effort." + - "LADS-aligned semantics differ from the reference implementation in result-driving ways." + +claim_tiers: + S0: benchmark_semantics + S1: standards_aligned_simulation + S2: second_failure_family_or_substrate + S3: real_lab_shadow_evaluation + S4: prospective_bounded_pilot + +ownership: + LabTrust-Gym: + - campaign_simulator + - hidden_oracle + - failure_injection + - stored_samples + - benchmark_runner + - metrics + EnvAssure: + - metrological_validity_semantics + - environment_validity_transitions + Scientific-Memory: + - typed_influence_graph + - evidence_invalidation_records + - claim_model_dependencies + AKTA: + - consume_scientific_state_disposition_for_action_admissibility + VerifierLab: + - comparative_campaign_methodology + - repair_and_recovery_evaluation + +amendments: + effective: + - file: AMENDMENT_0.1.1.md + reason: distinguish physical drift onset from scientific invalidity onset + timing: before_campaign_implementation_and_before_primary_outcomes + - file: AMENDMENT_0.1.2.md + reason: freeze one-based indexing, check ordering, primary drift sign, and false-alarm handling + timing: before_campaign_implementation_and_before_primary_outcomes + +result_blinding: + implementation_before_primary_results: required + protocol_changes_after_primary_results: + permitted_only_as: post_registration_analysis diff --git a/research/delayed_evidence_invalidation/study_contract.v0.1.3.yaml b/research/delayed_evidence_invalidation/study_contract.v0.1.3.yaml new file mode 100644 index 00000000..e8fd9722 --- /dev/null +++ b/research/delayed_evidence_invalidation/study_contract.v0.1.3.yaml @@ -0,0 +1,360 @@ +schema_version: delayed_evidence_invalidation.v0.1.3 +status: preregistered_amended +frozen_date: "2026-10-08" +repository: + name: fraware/LabTrust-Gym + base_commit: 7e361583fb3e8ca4f482b072f4ac190e30cc69a5 + branch: research/delayed-evidence-invalidation-v0 + +study: + id: dei_ph_drift_v0_1_3 + title: Delayed Evidence Invalidation and Scientific-State Recovery + mode: simulation + primary_failure_family: metrological_delayed_invalidation + no_new_repository: true + +state_model: + components: + - physical_state + - metrological_state + - evidence_state + - scientific_model_state + - selection_policy_state + influence_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SELECTED_BY + - SUPPORTS + - EXECUTED_ON + validity_states: + - provisional + - admitted + - quarantined + - invalidated + - superseded + descendant_dispositions: + severity_order: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +substrate: + instrument_semantics: opcua_lads_ph_meter_reference + reference_repository: opcua-lads/lads-server-collection + reference_component: servers/lads-ph-meter + required_features: + - nernst_signal_model + - temperature_compensation + - slope + - offset + - raw_signal + - calibration_values + - calibration_methods + note: > + Python reimplementation is permitted for deterministic offline campaigns only if + benchmark-level semantics are checked against the reference implementation. + +scientific_surface: + domain: + x_min: 0.0 + x_max: 1.0 + true_property: + expression: "7.0 + 1.20*sin(2*pi*x) + 0.35*cos(6*pi*x) + 0.25*x" + hidden_from_policy: true + target_ph: 7.40 + utility: + expression: "-(true_ph - 7.40)^2" + realism_claim: false + +campaign: + experiment_indexing: one_based + scientific_experiments: 240 + initial_design_points: 16 + adaptive_experiments: 224 + retained_sample_per_experiment: true + check_interval_experiments: 20 + check_replicates: 5 + check_reference_ph: 7.00 + check_ordering: immediately_after_scientific_experiment + check_uses_same_current_metrological_state: true + +noise_calibration: + seed: 20261008 + readings: 1000 + reference_ph: 7.00 + output: sigma0 + excluded_from_primary_analysis: true + +failure: + type: gradual_offset_drift + onset: + distribution: discrete_uniform + min_experiment: 60 + max_experiment: 120 + hidden_from_methods: true + primary_direction: positive_physical_offset + primary_reported_bias_direction: negative + standardized_rates_per_experiment: + - 0.075 + - 0.125 + - 0.200 + rate_units: sigma0 + secondary_failure: + type: slope_drift + match_standardized_effect: true + +application_validity: + primary_tolerance_sigma0: 1.0 + systematic_bias_definition: "E[reported_ph | true_ph, metrological_state] - true_ph" + scientific_invalidity_onset: "first t >= physical_drift_onset with abs(systematic_bias(t)) > tolerance" + physical_drift_alone_implies_invalidity: false + oracle_only: true + sensitivity_tolerance_sigma0: + - 0.5 + - 1.0 + - 2.0 + production_metrology_claim: false + +qc: + rule: shewhart_mean_threshold + threshold_expression: "abs(mean_check - 7.00) > 3*sigma0/sqrt(5)" + suspect_interval: "(last_in_control_check, detection_check]" + true_physical_drift_onset_available_to_hidden_adjudicator_only: true + true_scientific_invalidity_onset_available_to_hidden_adjudicator_only: true + suspect_interval_semantics: operational_bracketing_not_true_onset + false_positive_alarm_definition: alarm_before_scientific_invalidity_onset + false_positive_runs_retained: true + primary_recovery_estimand_condition: scientific_invalidity_precedes_first_alarm + report_unconditional_alarm_rate: true + report_fraction_entering_primary_estimand: true + +adaptive_policy: + visible_target: + expression: "-(reported_ph - 7.40)^2" + latent_adjudication_target: "-(true_ph - 7.40)^2" + latent_target_hidden_from_policy: true + acquisition_grid: + points: 1001 + domain: [0.0, 1.0] + includes_endpoints: true + tie_break: first_lowest_grid_index + ucb_schedule: + expression: "max(1.0, 1.7*sqrt(2*log(adaptive_iteration_zero_based + 2)))" + interpretation: literal_posterior_std_multiplier + primary: + name: gp_ucb_full_history + model: botorch_SingleTaskGP + standardize_outcomes: true + fit_hyperparameters_each_iteration: true + fit_objective: exact_marginal_likelihood + dtype: float64 + reference_device: cpu + frozen_optimizer_environment: + python: "3.12" + torch: "2.8.0" + botorch: "0.15.1" + gpytorch: "1.14" + numpy: "1.26.4" + scipy: "1.12.0" + robust_comparator: + name: released_rcgp_surrogate_matched_ucb + reference: "Ezzerg, Bogunovic, Knoblauch, ICML 2026" + repository: EZZERG/RCGP + commit: 68bd978dfe4de28aed02184efc64ea343cfa9afc + source_redistribution: prohibited_without_license_review + standardize_outcomes: true + plateau_width: 1.96 + plateau_shape_c: 1.0 + plateau_center: zero + sigma: fit + mean: fit + fitting_objective: wloo-cv + optimizer: lbfgs_released_default + acquisition: matched_grid_ucb + theory_guarantees_assumed_for_persistent_drift: false + nonstationarity_comparators: + family: sliding_window_gp_ucb + windows: [20, 40, 80] + report_all_windows: true + model_and_acquisition_otherwise_match_primary: true + rng: + derive_from_campaign_seed_and_adaptive_iteration: true + failed_fit_reseed: false + +recovery_methods: + B0: + name: no_retroactive_recovery + retroactive_change: false + B1: + name: provenance_flagging + rebuild_model: false + B2: + name: qc_discard_retrain + discard_suspect_ph: true + remeasure_samples: false + B3: + name: qc_blanket_remeasure_retrain + exact_mode: + remeasure_fraction: 1.0 + budgeted_selection: uniform_random + B4: + name: typed_dependency_aware_recovery + hidden_oracle_access: false + exact_mode: + max_remeasure_fraction: 1.0 + budgeted_selection: + intended_rule: greedy_expected_model_discrepancy_reduction + freeze_before_primary_runs: true + B5: + name: rcgp_matched_ucb_plus_B2 + acquisition_robustness: true + pre_alarm_trajectory_differs_from_primary: true + recovery_semantics: B2 + B6: + name: sliding_window_gp_ucb_plus_B2 + windows: [20, 40, 80] + pre_alarm_trajectory_differs_from_primary: true + recovery_semantics: B2 + +recovery_comparison_design: + same_trace_methods: [B0, B1, B2, B3, B4] + acquisition_robustness_controls: [B5, B6] + paired_recovery_claims_may_use_acquisition_controls: false + +recovery_budgets: + fractions: + - 0.10 + - 0.25 + - 0.50 + rounding: ceil + +primary_outcomes: + O1: + name: post_recovery_model_error + measure: normalized_integrated_squared_error + reference: latent_true_surface + O2: + name: retained_valid_work + measure: fraction_valid_physical_experiments_or_revalidated_measurements_retained + O3: + name: recovery_cost + measures: + - remeasurement_count + - normalized_compute_cost + interpretation_priority: remeasurement_count + O4: + name: validity_classification_accuracy + measure: macro_f1 + classes: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +secondary_outcomes: + - physical_drift_detection_latency + - scientific_invalidity_detection_latency + - cumulative_optimization_regret + - invalid_observations_admitted_from_scientific_invalidity_onset + - validity_closure_size + - selection_influence_closure_size + - policy_path_divergence + - claims_withdrawn_or_recomputed + - wall_clock_recovery_time + - remeasurement_efficiency + - false_invalidation_count + +hypotheses: + H1: delayed_invalidity_requires_retroactive_recovery + H2: dependency_semantics_preserve_valid_science + H3: robust_acquisition_does_not_replace_recovery + H4: selection_dependence_differs_from_validity_dependence + +statistics: + monte_carlo_unit: campaign_seed + target_seeds_per_drift_rate_per_method: 200 + drift_rate_conditions: 3 + initial_total_campaigns_per_method_before_budget_expansion: 600 + calibration_seeds_per_condition: 20 + calibration_seeds_excluded_from_primary: true + confidence_intervals: + method: bootstrap + level: 0.95 + paired_bootstrap_when_valid: true + report_effect_sizes: true + significance_alone_sufficient: false + sample_size_change: + allowed_before_primary_outcome_inspection: true + requires_variance_or_runtime_justification: true + +required_ablations: + - slope_drift + - check_interval_10_20_40 + - sample_expiry + - alternate_acquisition_policy + - alternate_latent_surface + - edge_type_removal + - incomplete_dependency_metadata + - false_positive_qc_alarm + - multiple_invalidation_events + - drift_sign_reversal + - all_seed_false_alarm_operational_cost + - acquisition_policy_sensitivity + +kill_criteria: + - "B2 matches B4 on model error, retained valid work, and recovery cost within practically negligible differences." + - "B3 dominates B4 across the recovery Pareto frontier." + - "Validity and selection closures collapse to the same set in realistic variants." + - "B4 advantage depends on hidden information unavailable to deployed systems." + - "Effect disappears under a second response surface or slope-drift family." + - "Effect depends on benchmark-specific identifiers or hand-coded injected-failure knowledge." + - "A simpler generic provenance traversal reproduces B4 with equivalent outputs and effort." + - "LADS-aligned semantics differ from the reference implementation in result-driving ways." + - "A preregistered sliding-window GP-UCB control plus B2 matches B4 on recovered scientific-state quality and retained valid work at lower total cost." + +claim_tiers: + S0: benchmark_semantics + S1: standards_aligned_simulation + S2: second_failure_family_or_substrate + S3: real_lab_shadow_evaluation + S4: prospective_bounded_pilot + +ownership: + LabTrust-Gym: + - campaign_simulator + - hidden_oracle + - failure_injection + - stored_samples + - benchmark_runner + - metrics + EnvAssure: + - metrological_validity_semantics + - environment_validity_transitions + Scientific-Memory: + - typed_influence_graph + - evidence_invalidation_records + - claim_model_dependencies + AKTA: + - consume_scientific_state_disposition_for_action_admissibility + VerifierLab: + - comparative_campaign_methodology + - repair_and_recovery_evaluation + +amendments: + effective: + - file: AMENDMENT_0.1.1.md + reason: distinguish physical drift onset from scientific invalidity onset + timing: before_campaign_implementation_and_before_primary_outcomes + - file: AMENDMENT_0.1.2.md + reason: freeze one-based indexing, check ordering, primary drift sign, and false-alarm handling + timing: before_campaign_implementation_and_before_primary_outcomes + - file: AMENDMENT_0.1.3.md + reason: freeze optimizer-visible objective, matched GP-UCB settings, RCGP comparator, and non-stationarity controls + timing: before_optimizer_implementation_and_before_primary_outcomes + +result_blinding: + implementation_before_primary_results: required + protocol_changes_after_primary_results: + permitted_only_as: post_registration_analysis diff --git a/research/delayed_evidence_invalidation/study_contract.v0.1.4.yaml b/research/delayed_evidence_invalidation/study_contract.v0.1.4.yaml new file mode 100644 index 00000000..4bcd9fd7 --- /dev/null +++ b/research/delayed_evidence_invalidation/study_contract.v0.1.4.yaml @@ -0,0 +1,403 @@ +schema_version: delayed_evidence_invalidation.v0.1.4 +status: preregistered_amended +frozen_date: "2026-10-08" +repository: + name: fraware/LabTrust-Gym + base_commit: 7e361583fb3e8ca4f482b072f4ac190e30cc69a5 + branch: research/delayed-evidence-invalidation-v0 + +study: + id: dei_ph_drift_v0_1_4 + title: Delayed Evidence Invalidation and Scientific-State Recovery + mode: simulation + primary_failure_family: metrological_delayed_invalidation + no_new_repository: true + +state_model: + components: + - physical_state + - metrological_state + - evidence_state + - scientific_model_state + - selection_policy_state + graph_orientation: prerequisite_source_to_dependent + influence_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SELECTED_BY + - SUPPORTS + - EXECUTED_ON + edge_endpoint_semantics: + MEASURED_UNDER: metrological_state_to_measurement + DERIVED_FROM: source_artifact_to_derived_artifact + UPDATES: evidence_or_prior_model_to_subsequent_model + SELECTED_BY: model_or_policy_to_selected_experiment + SUPPORTS: evidence_or_model_to_claim + EXECUTED_ON: experiment_or_action_to_resulting_physical_sample_state + validity_propagating_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SUPPORTS + - EXECUTED_ON + selection_edge: SELECTED_BY + selection_tainted_requires_not_in_validity_closure: true + validity_states: + - provisional + - admitted + - quarantined + - invalidated + - superseded + descendant_dispositions: + severity_order: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +substrate: + instrument_semantics: opcua_lads_ph_meter_reference + reference_repository: opcua-lads/lads-server-collection + reference_component: servers/lads-ph-meter + required_features: + - nernst_signal_model + - temperature_compensation + - slope + - offset + - raw_signal + - calibration_values + - calibration_methods + note: > + Python reimplementation is permitted for deterministic offline campaigns only if + benchmark-level semantics are checked against the reference implementation. + +scientific_surface: + domain: + x_min: 0.0 + x_max: 1.0 + true_property: + expression: "7.0 + 1.20*sin(2*pi*x) + 0.35*cos(6*pi*x) + 0.25*x" + hidden_from_policy: true + target_ph: 7.40 + utility: + expression: "-(true_ph - 7.40)^2" + realism_claim: false + +campaign: + experiment_indexing: one_based + scientific_experiments: 240 + initial_design_points: 16 + adaptive_experiments: 224 + retained_sample_per_experiment: true + check_interval_experiments: 20 + check_replicates: 5 + check_reference_ph: 7.00 + check_ordering: immediately_after_scientific_experiment + check_uses_same_current_metrological_state: true + +noise_calibration: + seed: 20261008 + readings: 1000 + reference_ph: 7.00 + output: sigma0 + excluded_from_primary_analysis: true + +failure: + type: gradual_offset_drift + onset: + distribution: discrete_uniform + min_experiment: 60 + max_experiment: 120 + hidden_from_methods: true + primary_direction: positive_physical_offset + primary_reported_bias_direction: negative + standardized_rates_per_experiment: + - 0.075 + - 0.125 + - 0.200 + rate_units: sigma0 + secondary_failure: + type: slope_drift + match_standardized_effect: true + +application_validity: + primary_tolerance_sigma0: 1.0 + systematic_bias_definition: "E[reported_ph | true_ph, metrological_state] - true_ph" + scientific_invalidity_onset: "first t >= physical_drift_onset with abs(systematic_bias(t)) > tolerance" + physical_drift_alone_implies_invalidity: false + oracle_only: true + sensitivity_tolerance_sigma0: + - 0.5 + - 1.0 + - 2.0 + production_metrology_claim: false + +qc: + rule: shewhart_mean_threshold + threshold_expression: "abs(mean_check - 7.00) > 3*sigma0/sqrt(5)" + suspect_interval: "(last_in_control_check, detection_check]" + true_physical_drift_onset_available_to_hidden_adjudicator_only: true + true_scientific_invalidity_onset_available_to_hidden_adjudicator_only: true + suspect_interval_semantics: operational_bracketing_not_true_onset + false_positive_alarm_definition: alarm_before_scientific_invalidity_onset + false_positive_runs_retained: true + primary_recovery_estimand_condition: scientific_invalidity_precedes_first_alarm + report_unconditional_alarm_rate: true + report_fraction_entering_primary_estimand: true + +adaptive_policy: + visible_target: + expression: "-(reported_ph - 7.40)^2" + latent_adjudication_target: "-(true_ph - 7.40)^2" + latent_target_hidden_from_policy: true + acquisition_grid: + points: 1001 + domain: [0.0, 1.0] + includes_endpoints: true + tie_break: first_lowest_grid_index + ucb_schedule: + expression: "max(1.0, 1.7*sqrt(2*log(adaptive_iteration_zero_based + 2)))" + interpretation: literal_posterior_std_multiplier + primary: + name: gp_ucb_full_history + model: botorch_SingleTaskGP + standardize_outcomes: true + fit_hyperparameters_each_iteration: true + fit_objective: exact_marginal_likelihood + dtype: float64 + reference_device: cpu + frozen_optimizer_environment: + python: "3.12" + torch: "2.8.0" + botorch: "0.15.1" + gpytorch: "1.14" + numpy: "1.26.4" + scipy: "1.12.0" + robust_comparator: + name: released_rcgp_surrogate_matched_ucb + reference: "Ezzerg, Bogunovic, Knoblauch, ICML 2026" + repository: EZZERG/RCGP + commit: 68bd978dfe4de28aed02184efc64ea343cfa9afc + source_redistribution: prohibited_without_license_review + standardize_outcomes: true + plateau_width: 1.96 + plateau_shape_c: 1.0 + plateau_center: zero + sigma: fit + mean: fit + fitting_objective: wloo-cv + optimizer: lbfgs_released_default + acquisition: matched_grid_ucb + theory_guarantees_assumed_for_persistent_drift: false + nonstationarity_comparators: + family: sliding_window_gp_ucb + windows: [20, 40, 80] + report_all_windows: true + model_and_acquisition_otherwise_match_primary: true + rng: + derive_from_campaign_seed_and_adaptive_iteration: true + failed_fit_reseed: false + +recovery_methods: + B0: + name: no_retroactive_recovery + retroactive_change: false + B1: + name: provenance_flagging + rebuild_model: false + B2: + name: qc_provenance_selective_recompute + quarantine_suspect_ph: true + selective_dependency_recompute: true + retain_unaffected_computational_artifacts: true + remeasure_samples: false + B3: + name: qc_blanket_remeasure_retrain + exact_mode: + remeasure_fraction: 1.0 + budgeted_selection: uniform_random + B3U: + name: qc_uncertainty_ranked_remeasure_retrain + typed_dependency_semantics: false + quarantine_semantics: B2 + budgeted_selection: + rule: sequential_max_safe_gp_posterior_variance + refit_after_each_remeasurement: true + tie_break: lowest_original_experiment_index + B4: + name: typed_dependency_aware_recovery + hidden_oracle_access: false + graph_orientation: prerequisite_source_to_dependent + exact_mode: + max_remeasure_fraction: 1.0 + budgeted_selection: + rule: sequential_max_safe_gp_posterior_variance_over_graph_eligible_samples + refit_after_each_remeasurement: true + tie_break: lowest_original_experiment_index + matched_heuristic_baseline: B3U + B5: + name: rcgp_matched_ucb_plus_B2 + acquisition_robustness: true + pre_alarm_trajectory_differs_from_primary: true + recovery_semantics: B2 + B6: + name: sliding_window_gp_ucb_plus_B2 + windows: [20, 40, 80] + pre_alarm_trajectory_differs_from_primary: true + recovery_semantics: B2 + +recovery_comparison_design: + same_trace_methods: [B0, B1, B2, B3, B3U, B4] + acquisition_robustness_controls: [B5, B6] + paired_recovery_claims_may_use_acquisition_controls: false + +recovery_budgets: + fractions: + - 0.10 + - 0.25 + - 0.50 + rounding: ceil + +primary_outcomes: + O1: + name: post_recovery_model_error + measure: normalized_integrated_squared_error + reference: latent_true_surface + O2: + name: retained_valid_work + measure: fraction_valid_physical_experiments_or_revalidated_measurements_retained + O3: + name: recovery_cost + measures: + - remeasurement_count + - normalized_compute_cost + interpretation_priority: remeasurement_count + O4: + name: validity_classification_accuracy + measure: macro_f1 + classes: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +secondary_outcomes: + - physical_drift_detection_latency + - scientific_invalidity_detection_latency + - cumulative_optimization_regret + - invalid_observations_admitted_from_scientific_invalidity_onset + - validity_closure_size + - selection_influence_closure_size + - policy_path_divergence + - claims_withdrawn_or_recomputed + - wall_clock_recovery_time + - remeasurement_efficiency + - false_invalidation_count + +hypotheses: + H1: delayed_invalidity_requires_retroactive_recovery + H2: dependency_semantics_preserve_valid_science + H3: robust_acquisition_does_not_replace_recovery + H4: selection_dependence_differs_from_validity_dependence + +statistics: + monte_carlo_unit: campaign_seed + target_seeds_per_drift_rate_per_method: 200 + drift_rate_conditions: 3 + initial_total_campaigns_per_method_before_budget_expansion: 600 + calibration_seeds_per_condition: 20 + calibration_seeds_excluded_from_primary: true + confidence_intervals: + method: bootstrap + level: 0.95 + paired_bootstrap_when_valid: true + report_effect_sizes: true + significance_alone_sufficient: false + sample_size_change: + allowed_before_primary_outcome_inspection: true + requires_variance_or_runtime_justification: true + +required_ablations: + - slope_drift + - check_interval_10_20_40 + - sample_expiry + - alternate_acquisition_policy + - alternate_latent_surface + - edge_type_removal + - incomplete_dependency_metadata + - false_positive_qc_alarm + - multiple_invalidation_events + - drift_sign_reversal + - all_seed_false_alarm_operational_cost + - acquisition_policy_sensitivity + +kill_criteria: + - "B2 matches B4 on model error, retained valid work, and recovery cost within practically negligible differences." + - "B3 dominates B4 across the recovery Pareto frontier." + - "Validity and selection closures collapse to the same set in realistic variants." + - "B4 advantage depends on hidden information unavailable to deployed systems." + - "Effect disappears under a second response surface or slope-drift family." + - "Effect depends on benchmark-specific identifiers or hand-coded injected-failure knowledge." + - "A simpler generic provenance traversal reproduces B4 with equivalent outputs and effort." + - "LADS-aligned semantics differ from the reference implementation in result-driving ways." + - "A preregistered sliding-window GP-UCB control plus B2 matches B4 on recovered scientific-state quality and retained valid work at lower total cost." + - "Generic provenance-selective recomputation plus the same active-remeasurement heuristic reproduces B4 classifications and recovery decisions across multiple failure families with equivalent complexity." + +paper_level_interpretation_gate: + cross_layer_recovery_superiority_requires_distinct_failure_families: 2 + first_family_alone_sufficient_for: + - delayed_invalidation_problem_demonstration + - metric_validation + - graph_semantics_validation + - benchmark_S0_S1_claims + first_family_alone_insufficient_for: + - general_cross_layer_recovery_superiority + +claim_tiers: + S0: benchmark_semantics + S1: standards_aligned_simulation + S2: second_failure_family_or_substrate + S3: real_lab_shadow_evaluation + S4: prospective_bounded_pilot + +ownership: + LabTrust-Gym: + - campaign_simulator + - hidden_oracle + - failure_injection + - stored_samples + - benchmark_runner + - metrics + EnvAssure: + - metrological_validity_semantics + - environment_validity_transitions + Scientific-Memory: + - typed_influence_graph + - evidence_invalidation_records + - claim_model_dependencies + AKTA: + - consume_scientific_state_disposition_for_action_admissibility + VerifierLab: + - comparative_campaign_methodology + - repair_and_recovery_evaluation + +amendments: + effective: + - file: AMENDMENT_0.1.1.md + reason: distinguish physical drift onset from scientific invalidity onset + timing: before_campaign_implementation_and_before_primary_outcomes + - file: AMENDMENT_0.1.2.md + reason: freeze one-based indexing, check ordering, primary drift sign, and false-alarm handling + timing: before_campaign_implementation_and_before_primary_outcomes + - file: AMENDMENT_0.1.3.md + reason: freeze optimizer-visible objective, matched GP-UCB settings, RCGP comparator, and non-stationarity controls + timing: before_optimizer_implementation_and_before_primary_outcomes + - file: AMENDMENT_0.1.4.md + reason: freeze graph orientation, strengthen provenance baseline, and match active remeasurement heuristic + timing: before_graph_recovery_implementation_and_before_primary_outcomes + +result_blinding: + implementation_before_primary_results: required + protocol_changes_after_primary_results: + permitted_only_as: post_registration_analysis diff --git a/research/delayed_evidence_invalidation/study_contract.v0.1.yaml b/research/delayed_evidence_invalidation/study_contract.v0.1.yaml new file mode 100644 index 00000000..cad87432 --- /dev/null +++ b/research/delayed_evidence_invalidation/study_contract.v0.1.yaml @@ -0,0 +1,263 @@ +schema_version: delayed_evidence_invalidation.v0.1 +status: preregistered +frozen_date: "2026-10-08" +repository: + name: fraware/LabTrust-Gym + base_commit: 7e361583fb3e8ca4f482b072f4ac190e30cc69a5 + branch: research/delayed-evidence-invalidation-v0 + +study: + id: dei_ph_drift_v0_1 + title: Delayed Evidence Invalidation and Scientific-State Recovery + mode: simulation + primary_failure_family: metrological_delayed_invalidation + no_new_repository: true + +state_model: + components: + - physical_state + - metrological_state + - evidence_state + - scientific_model_state + - selection_policy_state + influence_edges: + - MEASURED_UNDER + - DERIVED_FROM + - UPDATES + - SELECTED_BY + - SUPPORTS + - EXECUTED_ON + validity_states: + - provisional + - admitted + - quarantined + - invalidated + - superseded + descendant_dispositions: + severity_order: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +substrate: + instrument_semantics: opcua_lads_ph_meter_reference + reference_repository: opcua-lads/lads-server-collection + reference_component: servers/lads-ph-meter + required_features: + - nernst_signal_model + - temperature_compensation + - slope + - offset + - raw_signal + - calibration_values + - calibration_methods + note: > + Python reimplementation is permitted for deterministic offline campaigns only if + benchmark-level semantics are checked against the reference implementation. + +scientific_surface: + domain: + x_min: 0.0 + x_max: 1.0 + true_property: + expression: "7.0 + 1.20*sin(2*pi*x) + 0.35*cos(6*pi*x) + 0.25*x" + hidden_from_policy: true + target_ph: 7.40 + utility: + expression: "-(true_ph - 7.40)^2" + realism_claim: false + +campaign: + scientific_experiments: 240 + initial_design_points: 16 + adaptive_experiments: 224 + retained_sample_per_experiment: true + check_interval_experiments: 20 + check_replicates: 5 + check_reference_ph: 7.00 + +noise_calibration: + seed: 20261008 + readings: 1000 + reference_ph: 7.00 + output: sigma0 + excluded_from_primary_analysis: true + +failure: + type: gradual_offset_drift + onset: + distribution: discrete_uniform + min_experiment: 60 + max_experiment: 120 + hidden_from_methods: true + standardized_rates_per_experiment: + - 0.075 + - 0.125 + - 0.200 + rate_units: sigma0 + secondary_failure: + type: slope_drift + match_standardized_effect: true + +qc: + rule: shewhart_mean_threshold + threshold_expression: "abs(mean_check - 7.00) > 3*sigma0/sqrt(5)" + suspect_interval: "(last_in_control_check, detection_check]" + true_onset_available_to_hidden_adjudicator_only: true + +adaptive_policy: + primary: gp_ucb + exact_implementation: freeze_before_primary_runs + robust_comparator: + preferred: rcgp_ucb + reference: "Ezzerg, Bogunovic, Knoblauch, ICML 2026" + deviation_requires_preoutcome_protocol_update: true + +recovery_methods: + B0: + name: no_retroactive_recovery + retroactive_change: false + B1: + name: provenance_flagging + rebuild_model: false + B2: + name: qc_discard_retrain + discard_suspect_ph: true + remeasure_samples: false + B3: + name: qc_blanket_remeasure_retrain + exact_mode: + remeasure_fraction: 1.0 + budgeted_selection: uniform_random + B4: + name: typed_dependency_aware_recovery + hidden_oracle_access: false + exact_mode: + max_remeasure_fraction: 1.0 + budgeted_selection: + intended_rule: greedy_expected_model_discrepancy_reduction + freeze_before_primary_runs: true + B5: + name: robust_bo_plus_B2 + acquisition_robustness: true + recovery_semantics: B2 + +recovery_budgets: + fractions: + - 0.10 + - 0.25 + - 0.50 + rounding: ceil + +primary_outcomes: + O1: + name: post_recovery_model_error + measure: normalized_integrated_squared_error + reference: latent_true_surface + O2: + name: retained_valid_work + measure: fraction_valid_physical_experiments_or_revalidated_measurements_retained + O3: + name: recovery_cost + measures: + - remeasurement_count + - normalized_compute_cost + interpretation_priority: remeasurement_count + O4: + name: validity_classification_accuracy + measure: macro_f1 + classes: + - INVALID + - RECOMPUTE + - SELECTION_TAINTED + - UNAFFECTED + +secondary_outcomes: + - detection_latency + - cumulative_optimization_regret + - invalid_observations_admitted + - validity_closure_size + - selection_influence_closure_size + - policy_path_divergence + - claims_withdrawn_or_recomputed + - wall_clock_recovery_time + - remeasurement_efficiency + - false_invalidation_count + +hypotheses: + H1: delayed_invalidity_requires_retroactive_recovery + H2: dependency_semantics_preserve_valid_science + H3: robust_acquisition_does_not_replace_recovery + H4: selection_dependence_differs_from_validity_dependence + +statistics: + monte_carlo_unit: campaign_seed + target_seeds_per_drift_rate_per_method: 200 + drift_rate_conditions: 3 + initial_total_campaigns_per_method_before_budget_expansion: 600 + calibration_seeds_per_condition: 20 + calibration_seeds_excluded_from_primary: true + confidence_intervals: + method: bootstrap + level: 0.95 + paired_bootstrap_when_valid: true + report_effect_sizes: true + significance_alone_sufficient: false + sample_size_change: + allowed_before_primary_outcome_inspection: true + requires_variance_or_runtime_justification: true + +required_ablations: + - slope_drift + - check_interval_10_20_40 + - sample_expiry + - alternate_acquisition_policy + - alternate_latent_surface + - edge_type_removal + - incomplete_dependency_metadata + - false_positive_qc_alarm + - multiple_invalidation_events + +kill_criteria: + - "B2 matches B4 on model error, retained valid work, and recovery cost within practically negligible differences." + - "B3 dominates B4 across the recovery Pareto frontier." + - "Validity and selection closures collapse to the same set in realistic variants." + - "B4 advantage depends on hidden information unavailable to deployed systems." + - "Effect disappears under a second response surface or slope-drift family." + - "Effect depends on benchmark-specific identifiers or hand-coded injected-failure knowledge." + - "A simpler generic provenance traversal reproduces B4 with equivalent outputs and effort." + - "LADS-aligned semantics differ from the reference implementation in result-driving ways." + +claim_tiers: + S0: benchmark_semantics + S1: standards_aligned_simulation + S2: second_failure_family_or_substrate + S3: real_lab_shadow_evaluation + S4: prospective_bounded_pilot + +ownership: + LabTrust-Gym: + - campaign_simulator + - hidden_oracle + - failure_injection + - stored_samples + - benchmark_runner + - metrics + EnvAssure: + - metrological_validity_semantics + - environment_validity_transitions + Scientific-Memory: + - typed_influence_graph + - evidence_invalidation_records + - claim_model_dependencies + AKTA: + - consume_scientific_state_disposition_for_action_admissibility + VerifierLab: + - comparative_campaign_methodology + - repair_and_recovery_evaluation + +result_blinding: + implementation_before_primary_results: required + protocol_changes_after_primary_results: + permitted_only_as: post_registration_analysis