Skip to content

Fix ExperimentMemory recording and surface silent failures as once-per-run degradations - #323

Open
DermotOBrien-EC wants to merge 7 commits into
aiming-lab:mainfrom
DermotOBrien-EC:upstream-fast-fixes
Open

Fix ExperimentMemory recording and surface silent failures as once-per-run degradations#323
DermotOBrien-EC wants to merge 7 commits into
aiming-lab:mainfrom
DermotOBrien-EC:upstream-fast-fixes

Conversation

@DermotOBrien-EC

Copy link
Copy Markdown
Contributor

This is the focused resubmission invited in the #302 review: the ExperimentMemory constructor fix and the once-per-run degradation logging, fully decoupled from the result gate (which will follow separately, reworked per the review).

What this fixes

1. ExperimentMemory never recorded anything. execute_pipeline calls ExperimentMemory(store_dir=...), but the constructor is __init__(store, retriever, embed_fn=None), so every run raised a TypeError that the except Exception swallowed at debug level. The hook now builds a MemoryStore + MemoryRetriever and adds the ExperimentOutcome record type plus record_outcome() (persisted immediately).

2. Memory outcomes recorded metric 0.0 on every real run. The outcome hook extracted the metric from run_dir/results.json, a file production never writes (real per-run metrics live under stage-*/runs/*.json). It now aggregates the actual run artifacts via _collect_experiment_results and records the contract metric's mean, matching condition-prefixed keys such as ppo/primary_metric.

3. Silent failures become once-per-run degradation records. Experiment-memory init failures, outcome-recording failures, and knowledge-base export failures were swallowed (logger.debug or bare pass). Each now appends one {key, message} record to a degradations list written into pipeline_summary.json, with a single WARNING log the first time. Runs are never failed by these paths; the point is that the final summary tells the truth about what silently stopped working.

4. Degradations survive recursive PIVOT/REFINE runs. The rollback path re-enters execute_pipeline, and the outer frame's summary write would otherwise discard anything recorded inside the recursion. The accumulator is now threaded through the recursive call (a new optional kw-only parameter; all existing callers are unaffected).

5. Dead hooks removed. The EventLog and PitfallDetector hooks import researchclaw.pipeline.event_log and researchclaw.pipeline.pitfall_detector, neither of which exists in the repo, so both blocks were unreachable code inside try/except. Removed outright.

Testing

Every behavior above has a test, and each fix was verified red-first (the new test fails on the pre-fix code):

  • memory init failure recorded exactly once
  • outcome recorded with the real metric (0.42 from a stage-12/runs/run_01.json fixture matching the real artifact layout; the old phantom run_dir/results.json fixture is gone)
  • outcome-recording failure recorded exactly once
  • KB export failure recorded exactly once, pipeline completes
  • a KB failure occurring only inside a recursive REFINE frame reaches the final pipeline_summary.json

Full suite: 2956 passed, 56 skipped (the skips are the env-gated live suites).

PIVOT/REFINE rollback re-enters execute_pipeline with a fresh frame; the
outer frame later overwrites pipeline_summary.json with only its own
degradation list, so records made during the recursive run were lost.
Thread one shared accumulator through the recursive call.
The outcome hook read run_dir/results.json, which production never
writes, so every remembered outcome carried metric 0.0. Aggregate the
actual stage-*/runs artifacts via _collect_experiment_results and record
the contract metric mean, matching condition-prefixed keys.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant