diff --git a/CHANGELOG.md b/CHANGELOG.md index 77cc6a2..cdf2760 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -30,6 +30,18 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 under its verdict) warns that the evaluator checked nothing because no example offered tools. Coverage is written by `evaluate`, so an older run needs `evaluate` re-run to drop the stale coverage; `analyze` alone keeps it, and the `conditional_pass` with it. +- The run-insights narrative no longer guesses why a run failed. The prompt listed each + budget's limit but not its result, so a FAIL on the latency ceiling alone was explained + as "failed to meet the budget for Trace-rule violations" — a budget that held. Every + budget, slice rows included, now enters the prompt with its status, the breached ones + are listed as `failed_budgets`, and the decision's `reason` travels with them in plain + words, without `evalshift.yaml` keys. Only a breached budget may be named as the cause, + and on a FAIL a summary or recommendation that blames a budget that held is rejected + and retried, then replaced by the templated prose. Naming a held budget to clear it + ("Cost increase stayed within its limit", "not Cost increase") is not blame, and + neither is a figure "over the source". Narratives are cached on the run's + config hash and model, not the CLI version: delete a run's `insights.json` and re-run + `report` to rewrite an existing one. ## [1.4.1] - 2026-10-11 diff --git a/DOCS.md b/DOCS.md index ba95693..07432e6 100644 --- a/DOCS.md +++ b/DOCS.md @@ -781,6 +781,10 @@ CI wiring (on `analyze` and `compare`): **A gate that measured nothing is never counted as a gate that passed.** A budget handed an empty sample — a tool-divergence ceiling on a run that scored no divergence row, a cost ceiling on a run with no priced call — is within its limit by arithmetic: `0/0` is below every ceiling. Counted naively that renders "7 of 7 budgets passed", and a narrative restates it as "all hard constraints are met" while the report body two sections above names the blocking evaluators that scored nothing. So `budgets_passed` counts only budgets that were *measured*, the blind ones are named on their own `unmeasured_budgets` / `unmeasured_evaluators` facts with a digit-free `coverage_basis` saying why, and the instruction forbids claiming a clean sweep, a met constraint or a safe migration while that basis is present. The blind-gate set is the same one `migration_decision.json`'s own `recommendations` name, computed by one shared function, so the two surfaces cannot disagree. This is distinct from the rates rule above and not covered by it: that fires when the *whole run* measured nothing, this fires when only *some* gate did. +**A FAIL's cause is read off the facts, not inferred.** Every gating budget, overall and slice rows alike, enters the prompt with its status beside its limit — `budget "