Skip to content

[research] Preregister delayed evidence invalidation study - #7

Draft
fraware wants to merge 18 commits into
mainfrom
research/delayed-evidence-invalidation-v0
Draft

fraware wants to merge 18 commits into
mainfrom
research/delayed-evidence-invalidation-v0

Conversation

@fraware

@fraware fraware commented Oct 8, 2026

Copy link
Copy Markdown
Owner

Purpose

Freeze the v0.1 research protocol for delayed evidence invalidation and scientific-state recovery before implementing the benchmark or inspecting primary comparative outcomes.

Research question

When evidence previously admitted into a continuously adaptive laboratory is invalidated only after it has changed the scientific model and influenced later experiment selection, how should downstream scientific state be classified and recovered?

What this PR contains

  • preregistration with hypotheses, formal state, edge semantics, baselines, outcomes, statistical plan, ablations, kill criteria, and claim tiers;
  • machine-readable study contract;
  • prior-art/standards boundary, including close adjacent work on downstream influence echoes in adaptive agent systems;
  • no benchmark implementation and no primary results.

Frozen novelty boundary

This work does not claim novelty for metrology/QC, provenance, claim-aware lineage, robust BO, machine unlearning, sequential-decision unlearning, downstream adaptive influence, or dynamic assurance.

The provisional scientific distinction under test is between validity dependence and experiment-selection dependence in adaptive physical science, with recovery evaluated for correctness, retained valid work, and cost.

Review focus before implementation

  1. Does the distinction between validity closure and selection-influence closure survive the strongest adjacent prior art?
  2. Are B2/B3/B5 strong enough to prevent a straw-man comparison?
  3. Is the pH-drift/LADS substrate sufficiently faithful for S1 claims?
  4. Are the primary outcomes and kill criteria capable of falsifying the richer architecture?
  5. Does the study preserve a clean path to second-family and real-lab validation?

No merge is required before implementation; the draft PR serves as the timestamped preregistration boundary.

fraware commented Oct 8, 2026

Copy link
Copy Markdown
Owner Author

Protocol amendment v0.1.1 is now effective, still before campaign implementation and before inspection of primary comparative outcomes.

The amendment separates:

  • physical drift onset; and
  • scientific invalidity onset, defined by a declared application-validity tolerance on systematic measurement bias.

This avoids treating every nonzero drift as scientific invalidity and preserves the operational meaning of the check-standard suspect interval. The primary tolerance is 1.0× the calibration-only sigma0 estimate, with preregistered sensitivity at 0.5× and 2.0×. This is a benchmark parameter, not a production pH-metrology criterion.

Effective implementation contract: research/delayed_evidence_invalidation/study_contract.v0.1.1.yaml. The original v0.1 artifacts remain unchanged as historical records.

fraware commented Oct 8, 2026

Copy link
Copy Markdown
Owner Author

Protocol amendment v0.1.2 is now effective, still before campaign implementation and before primary comparative outcomes.

It freezes the remaining campaign degrees of freedom:

  • one-based experiment indices;
  • check standards immediately after experiments 20, 40, … under the same current metrological state;
  • positive physical-offset drift as the primary sign, with sign reversal required as a sensitivity test;
  • false-positive alarms retained in the corpus, reported explicitly, and never silently reseeded.

The primary delayed-invalidation recovery estimand is conditional on scientific invalidity preceding the first control alarm; unconditional alarm rate and all-seed operational cost remain required outputs.

Effective contract: study_contract.v0.1.2.yaml.

fraware commented Oct 8, 2026

Copy link
Copy Markdown
Owner Author

Protocol amendment v0.1.3 is now effective, still before optimizer implementation and before primary comparative recovery outcomes.

The optimizer audit showed that RCGP-UCB is a strong corruption baseline but its published theory targets frequency-constrained corruptions, while the primary benchmark uses persistent gradual metrological drift. The protocol now includes both:

  • a pinned released-RCGP surrogate comparator; and
  • preregistered sliding-window GP-UCB controls (20/40/80 observations) for non-stationarity.

It also freezes the policy-visible target, matched UCB schedule, 1001-point acquisition grid, optimizer environment, and the rule that B0–B4 share the same pre-alarm GP-UCB trace while B5/B6 are analyzed separately as acquisition-robustness controls.

Effective contract: research/delayed_evidence_invalidation/study_contract.v0.1.3.yaml.

fraware commented Oct 8, 2026

Copy link
Copy Markdown
Owner Author

Protocol amendment v0.1.4 is now effective, still before influence-graph/recovery implementation and before primary comparative outcomes.

Two changes strengthen falsifiability:

  1. every influence edge now has one global source/prerequisite -> dependent orientation, with validity propagation explicitly excluding SELECTED_BY;
  2. B2 is upgraded to provenance-selective computational recomputation, and B3U adds uncertainty-ranked physical remeasurement using the same active-selection heuristic planned for B4.

The consequence is deliberate: B4 cannot win merely because it knows how to traverse a dependency graph or choose informative samples. A broad recovery-superiority claim now requires at least two failure families with distinct minimal repair actions.

Effective contract: study_contract.v0.1.4.yaml.

@fraware
fraware force-pushed the research/delayed-evidence-invalidation-v0 branch from 7482934 to 94ffc7f Compare October 12, 2026 14:11

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant