ExperimentSpec v1: versioned schema with result gating against real stage-14 metrics

[DermotOBrien-EC]

Resubmission of #301, with the result gate reworked to resolve the three blockers from the #301/#302 closing reviews. Stacked on #323: the first 7 commits here belong to that PR; this PR adds the last 2 (Evolve ExperimentSpec v1 schema and gate spec violations in the runner, Record unenforced cost budgets as a degradation). Kept as a draft until #323 lands, then I will rebase so the diff shows only the schema and gate work.

The three blockers, resolved

1. The result gate now validates the artifact the pipeline actually writes. The old gate read run_dir/results.json, which production never creates, so stage 14 hard-failed every real run. The reworked gate reads stage-14/experiment_summary.json (the file the just-completed stage writes; the stage-14_vN directories are pre-rollback snapshots, since _version_rollback_stages renames the old directory before a re-run) and validates the contract metrics against its metrics_summary:

• Keys are condition-prefixed, so contract metric accuracy matches ppo/accuracy as well as a bare accuracy key.
• Values are the standard {min, max, mean, count} aggregates; the mean must be finite and numeric. Bare numeric values are also accepted, mirroring the BUG-09 tolerance in _analysis.py for metrics that do not use the standard shape.
• A missing summary file, or a missing or empty metrics_summary, DEFERS: the stage result is returned unchanged and a once-per-run result_gate_deferred degradation is recorded in pipeline_summary.json (stage 15 diagnosis and the R6-4 tolerance already handle the empty-metrics situation downstream).
• Real corruption still fails the stage with a spec_violations.json artifact: an unreadable or invalid-JSON summary, a non-object metrics_summary, a contract metric with no matching key, or a non-finite value.

2. The cost budget can no longer hard-fail a run. max_budget_usd defaults to 5.0 and researchclaw/cost_tracker.py does not exist in the repo, so the old #302 change failed every CLI-agent run at the first stage. Now a missing tracker records a once-per-run cost_budget_unenforced degradation for CLI-agent providers (silent for provider llm, where the knob is inert by design), the stop-on-exceeded path is preserved unchanged when a tracker is importable, and tracker errors are recorded once instead of swallowed.

3. No importlib.util.find_spec. The missing-import bug is gone structurally: tracker availability is detected with try/except ImportError.

Also in the schema commit

• UniversalExperimentPlan gains the v1 contract fields (schema_version, mode, budget, seeds, preregistered prediction), strict from_dict parsing that rejects unknown keys, YAML round-trip (to_yaml_v1 / from_yaml_v1), and validate(strict=...).
• from_legacy_exp_plan now preserves every contract metric: the first becomes the primary, the rest become secondary_metrics (previously they were silently dropped, so a plan with metrics: [accuracy, loss] never enforced loss).
• After EXPERIMENT_DESIGN, the plan is converted and written as stage-09/experiment_spec.yaml; the gate skips with a warning when that file is absent (pre-v1 runs).

Testing

The old gate tests manually created run_dir/results.json, which is what masked blocker 1; no test does that anymore. The rewritten tests build the real artifact layout (stage-14/experiment_summary.json with condition-prefixed metrics_summary) and every behavioral change was verified red-first against the pre-fix code. Coverage includes: prefixed and exact metric matching, a missing secondary metric failing the stage, non-finite means failing, empty and absent summaries deferring with exactly one degradation record, a non-object metrics_summary failing as malformed, the gate validating the current stage-14/ rather than a stale _v2 snapshot, budget degradation recorded once for CLI-agent runs and not for llm, the preserved stop-on-exceeded path, schema round-trip and validation, and the converter keeping secondary metrics in both dict and list forms.

Full suite: 2976 passed, 56 skipped (the skips are the env-gated live suites).