diff --git a/CHANGELOG.md b/CHANGELOG.md index 3766589..be5474b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- Failure categories count examples, not scored rows. `tool_selection` stamps + `TOOL_SELECTION_DRIFT` on both its conformance and its divergence row, so one example + that picked another tool read "Different tools chosen (2 examples)" in the narrative + and 2 in the report's top regression causes. Each label is now counted once per + distinct example, as `INVARIANT_VIOLATION` already was. +- The cached run narrative no longer outlives the figures it describes. `insights.json` + was keyed on the run's `config_hash`, which is fixed when the run is created, while + `evaluate` and `analyze` re-read `evalshift.yaml` every time: loosening a budget and + re-running `analyze` + `report` rendered a PASS verdict beside a cached "FAIL under the + configured policy" paragraph. The key now also covers the narrative's facts (scores, + analysis and the persisted decision), so any change to them regenerates it on the next + `report`; a plain re-render still reuses it. Existing caches miss once and regenerate. + ## [1.4.2] - 2026-10-12 ### Fixed diff --git a/DOCS.md b/DOCS.md index d0d9f41..7035ba4 100644 --- a/DOCS.md +++ b/DOCS.md @@ -533,6 +533,8 @@ Pairwise A/B comparison per criterion: the judge sees the two outputs anonymised Regressions carry machine-readable labels that the report and hosted diff group by: `FORMAT_FAILURE`, `SEMANTIC_REGRESSION`, `TOOL_SELECTION_DRIFT`, `TOOL_GROUND_TRUTH_MISS` (both models missed the same recorded tool ground truth — a broken-harness signal, not a migration finding; see [Migration policy and CI gating](#migration-policy-and-ci-gating)), `ARGUMENT_VALUE_DRIFT`, `TOOL_TRACE_STRUCTURE_DRIFT`, `TOOL_ORDER_DRIFT`, `DANGEROUS_ACTION_DRIFT`, `MISSING_VERIFICATION_STEP`, `UNNECESSARY_TOOL_CALL`, `REFUSAL_REGRESSION`, and `INVARIANT_VIOLATION` (the target broke a hand-written [trace rule](#trace-invariants), stamped whatever the source did). That is the complete set — every label the evaluators emit is declared in `evaluators/failures.py`. The machine labels live in `scores.jsonl`, `report.json` and the bundle; every rendered surface (the HTML report, decision prose, the run narrative) shows the plain-language display name instead — `TOOL_SELECTION_DRIFT` renders as "Different tools chosen" — with the mapping declared beside the labels in `evaluators/failures.py`. +The decision's `failure_categories` (the report's **Top regression causes** panel, and the narrative's "Most common failure … (N examples)") count each label once per **distinct example**, not per scored row: `tool_selection` stamps `TOOL_SELECTION_DRIFT` on both its conformance and its divergence row, so one example that picked another tool is one, not two. `INVARIANT_VIOLATION` is further narrowed to blocking entries — see [Trace invariants](#trace-invariants). + `ARGUMENT_VALUE_DRIFT` counts **regressions**: it is stamped only when the target scored below the source. Under `against: expected` both models can miss the same recorded expectation by the same margin — a zero delta, and a fact about your ground truth rather than a migration defect, already reported as such. Policy budgets are unaffected by the label: `max_tool_argument_drift` counts calls whose *target* score fell below `tool_argument_drift_floor`. ### Cost @@ -783,14 +785,14 @@ CI wiring (on `analyze` and `compare`): **A FAIL's cause is read off the facts, not inferred.** Every gating budget, overall and slice rows alike, enters the prompt with its status beside its limit — `budget "