Fix ExperimentMemory recording and surface silent failures as once-per-run degradations - #323
Open
DermotOBrien-EC wants to merge 7 commits into
Open
Fix ExperimentMemory recording and surface silent failures as once-per-run degradations#323DermotOBrien-EC wants to merge 7 commits into
DermotOBrien-EC wants to merge 7 commits into
Conversation
PIVOT/REFINE rollback re-enters execute_pipeline with a fresh frame; the outer frame later overwrites pipeline_summary.json with only its own degradation list, so records made during the recursive run were lost. Thread one shared accumulator through the recursive call.
The outcome hook read run_dir/results.json, which production never writes, so every remembered outcome carried metric 0.0. Aggregate the actual stage-*/runs artifacts via _collect_experiment_results and record the contract metric mean, matching condition-prefixed keys.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is the focused resubmission invited in the #302 review: the ExperimentMemory constructor fix and the once-per-run degradation logging, fully decoupled from the result gate (which will follow separately, reworked per the review).
What this fixes
1. ExperimentMemory never recorded anything.
execute_pipelinecallsExperimentMemory(store_dir=...), but the constructor is__init__(store, retriever, embed_fn=None), so every run raised aTypeErrorthat theexcept Exceptionswallowed at debug level. The hook now builds aMemoryStore+MemoryRetrieverand adds theExperimentOutcomerecord type plusrecord_outcome()(persisted immediately).2. Memory outcomes recorded metric 0.0 on every real run. The outcome hook extracted the metric from
run_dir/results.json, a file production never writes (real per-run metrics live understage-*/runs/*.json). It now aggregates the actual run artifacts via_collect_experiment_resultsand records the contract metric's mean, matching condition-prefixed keys such asppo/primary_metric.3. Silent failures become once-per-run degradation records. Experiment-memory init failures, outcome-recording failures, and knowledge-base export failures were swallowed (
logger.debugor barepass). Each now appends one{key, message}record to adegradationslist written intopipeline_summary.json, with a single WARNING log the first time. Runs are never failed by these paths; the point is that the final summary tells the truth about what silently stopped working.4. Degradations survive recursive PIVOT/REFINE runs. The rollback path re-enters
execute_pipeline, and the outer frame's summary write would otherwise discard anything recorded inside the recursion. The accumulator is now threaded through the recursive call (a new optional kw-only parameter; all existing callers are unaffected).5. Dead hooks removed. The EventLog and PitfallDetector hooks import
researchclaw.pipeline.event_logandresearchclaw.pipeline.pitfall_detector, neither of which exists in the repo, so both blocks were unreachable code inside try/except. Removed outright.Testing
Every behavior above has a test, and each fix was verified red-first (the new test fails on the pre-fix code):
stage-12/runs/run_01.jsonfixture matching the real artifact layout; the old phantomrun_dir/results.jsonfixture is gone)pipeline_summary.jsonFull suite:
2956 passed, 56 skipped(the skips are the env-gated live suites).