diff --git a/CHANGELOG.md b/CHANGELOG.md index 99c3609..77cc6a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- A `semantic` evaluator set to `blocking: true` no longer fails runs whose every row + stays within `min_similarity`. The source is pinned at 1.0, so any rewording is a + small negative delta, and the paired significance test ran on those raw deltas: + nine rows at 0.90–0.98 tested as a `critical` regression and failed a run whose every + budget passed. Rows within `min_similarity` now enter the test as no change — the + rule the budgets already used — and the comparison notes how many. `delta_avg_score` + and the aggregates still show the raw drift. Re-analysing an older run can lower its + semantic severities. +- A fail caused by comparison severity alone now carries a `reason` naming each + comparison, so the report no longer shows "7 of 7 budgets within policy" under FAIL. + The templated recommendation names the actual cause instead of always claiming a + budget breach — a breached budget, including a slice's own (`the 'security' slice + overall regression rate budget was breached`), or else the overall blocking + regressions — and the summary's critical count reads the blocking comparisons. +- Tool evaluators (including a top-level `trace_invariants` inherited by every suite) no + longer attempt pairs that were offered no tools. On a tool-free suite they used to + book every pair as unmeasured, demote a clean `pass` to `conditional_pass`, and blame + "tool calls and no prose". They now skip those pairs, and `evaluate` (and `compare`, + under its verdict) warns that the evaluator checked nothing because no example offered + tools. Coverage is written by `evaluate`, so an older run needs `evaluate` re-run to + drop the stale coverage; `analyze` alone keeps it, and the `conditional_pass` with it. + ## [1.4.1] - 2026-10-11 ### Fixed diff --git a/DOCS.md b/DOCS.md index 000669d..ba95693 100644 --- a/DOCS.md +++ b/DOCS.md @@ -378,7 +378,7 @@ Slices holding exactly the same examples are collapsed to one before any test ru ### Per-suite evaluators -A project's suites are rarely homogeneous — one calls tools, six answer in prose — and one top-level `evaluators:` block either leaves the tool-calling suite unmeasured or hands the tool-free ones a tool evaluator with an empty denominator, which the policy reads as an *inconclusive* gate rather than as "not applicable here". So a `suites:` entry can carry its own evaluator block, and `capture sync` generates one per suite from what that suite's rows actually contain: +A project's suites are rarely homogeneous — one calls tools, six answer in prose — and one top-level `evaluators:` block either leaves the tool-calling suite unmeasured or hands the tool-free ones tool evaluators with nothing to check (they skip every pair that was offered no tools, and `evaluate` — or `compare`, under its verdict — warns that they checked nothing). So a `suites:` entry can carry its own evaluator block, and `capture sync` generates one per suite from what that suite's rows actually contain: ```yaml # >>> evalshift suites (managed by `evalshift capture sync`) >>> @@ -511,7 +511,7 @@ When an evaluator's own measurement breaks (judge call fails, embedding call fai ### Semantic (`evaluators.semantic`, single block) -Embeds both outputs and scores the target by cosine similarity to the source (source score pinned at 1.0). Config: `embedding_model` (default `text-embedding-3-small`, an OpenAI model; the Gemini scaffold uses `gemini/gemini-embedding-001`, and the Anthropic and DeepSeek scaffolds borrow OpenAI's or Gemini's — see [Project setup](#project-setup)), `min_similarity` (default 0.9) — below it the pair is flagged `SEMANTIC_REGRESSION`. Within it, drift counts as *equivalent* for policy purposes. The yardstick is the source output, never the suite's `expected` field, so this measures drift rather than correctness: cosine distance can't tell "reworded" from "wrong", which is why the scaffold keeps it advisory (the library default is `blocking: true`; see [FAQ](docs/faq.md#why-does-a-hand-written-config-block-on-semantic-when-init-does-not)) and why correctness belongs to an `llm_judge` criterion. +Embeds both outputs and scores the target by cosine similarity to the source (source score pinned at 1.0). Config: `embedding_model` (default `text-embedding-3-small`, an OpenAI model; the Gemini scaffold uses `gemini/gemini-embedding-001`, and the Anthropic and DeepSeek scaffolds borrow OpenAI's or Gemini's — see [Project setup](#project-setup)), `min_similarity` (default 0.9) — below it the pair is flagged `SEMANTIC_REGRESSION`. Within it, drift counts as *equivalent* for policy purposes and as *no change* in the paired test that sets severity — see [methodology](docs/methodology.md#drift-inside-min_similarity-is-tested-as-no-change). The yardstick is the source output, never the suite's `expected` field, so this measures drift rather than correctness: cosine distance can't tell "reworded" from "wrong", which is why the scaffold keeps it advisory (the library default is `blocking: true`; see [FAQ](docs/faq.md#why-does-a-hand-written-config-block-on-semantic-when-init-does-not)) and why correctness belongs to an `llm_judge` criterion. Presentation is stripped before embedding, so formatting never reads as drift: a code fence wrapping the **whole** output (` ```json … ``` `) is dropped, and an output that is a JSON object or array is re-serialized canonically (sorted keys, uniform separators). A fenced, pretty-printed object and the same object returned compact score 1.0; a fence inside prose is part of the answer and kept. An empty fenced block counts as an empty output. Whether the output is *valid, unfenced* JSON is a separate question — `json_schema` and the judge answer it. @@ -701,7 +701,7 @@ Trace-aware debugging: `evalshift diff case ` (side-by-side trace Full contract in [docs/methodology.md](docs/methodology.md). Per `(prompt, evaluator, slice)` comparison over the paired per-example deltas: -1. **Sample guards.** A pair an evaluator measured nothing on (a tool-only turn, an example with no ground truth) has no row in `scores.jsonl` at all, so it is absent from `n`, from the slice aggregates and from the policy metrics by construction. The count is reconstructed from `state.json`'s `evaluator_coverage` and noted, so "8 of 10 rows not applicable" survives the rows themselves. n < 5 → skipped, severity `insufficient`. 5 ≤ n < 20 → tested, but flagged uncertain. Zero-variance deltas → skipped, severity `none`. **No applicable rows left** → severity `insufficient` with a `nothing measured:` note, never `none`; a run whose blocking evaluators all landed there cannot return `pass` — it is downgraded to `conditional_pass` (or stays `inconclusive`) with those evaluators named in `recommendations`, since an evaluator that compared nothing is unknown, not equivalent. Advisory (`blocking: false`) evaluators are exempt — their silence gates nothing by design, so they are neither named nor demote the verdict: `evaluator_coverage` carries each axis's `blocking` flag, and the synthesized comparison gets an `advisory:` note (zero rows leave the policy layer nothing else to read the flag from). The HTML report gives that row its own headline — **Nothing measured**, not the **Not enough data** the rest of `insufficient` gets. +1. **Sample guards.** A pair an evaluator measured nothing on (a tool-only turn, an example with no ground truth) has no row in `scores.jsonl` at all, so it is absent from `n`, from the slice aggregates and from the policy metrics by construction. The count is reconstructed from `state.json`'s `evaluator_coverage` and noted, so "8 of 10 rows not applicable" survives the rows themselves. n < 5 → skipped, severity `insufficient`. 5 ≤ n < 20 → tested, but flagged uncertain. Zero-variance deltas → skipped, severity `none`. A semantic row within `min_similarity` enters the test as a delta of 0 (noted `k of n rows within the evaluator's own threshold (min_similarity) — tested as no change`); `delta_avg_score` stays the raw mean. **No applicable rows left** → severity `insufficient` with a `nothing measured:` note, never `none`; a run whose blocking evaluators all landed there cannot return `pass` — it is downgraded to `conditional_pass` (or stays `inconclusive`) with those evaluators named in `recommendations`, since an evaluator that compared nothing is unknown, not equivalent. Advisory (`blocking: false`) evaluators are exempt — their silence gates nothing by design, so they are neither named nor demote the verdict: `evaluator_coverage` carries each axis's `blocking` flag, and the synthesized comparison gets an `advisory:` note (zero rows leave the policy layer nothing else to read the flag from). The HTML report gives that row its own headline — **Nothing measured**, not the **Not enough data** the rest of `insufficient` gets. A tool evaluator is never handed a pair that was offered no tools at all — like an `applies_to` miss, that pair is not attempted: no cell, no coverage entry, so it counts toward neither the "not applicable" note nor "nothing measured". 2. **Normality screen.** Shapiro–Wilk on the deltas at α = 0.05: normal → **paired t-test**, non-normal → **Wilcoxon signed-rank**. n > 5000 skips the screen (CLT, t-test). 3. **Effect size.** Paired Cohen's d = mean(Δ)/std(Δ), with a 95% CI — analytical for the t-test, percentile bootstrap (2,000 resamples, seeded, deterministic) for Wilcoxon. 4. **Multiple comparisons.** Benjamini–Hochberg FDR correction at α = 0.05 across *all* testable comparisons in the run (equivalent to `statsmodels fdr_bh`). @@ -756,10 +756,10 @@ How the verdict is computed: - **A shared ground-truth miss is not equivalence.** A `tool_selection.conformance` row grades each side *absolutely* against the ground truth the suite recorded, so both sides can miss it at the same height — `0.0 / 0.0` on an example both models answered with a tool call the recording never made. The delta is zero, which used to read as equivalence and is the whole of the `equivalent_rate: 1.0` a real run once shipped over a suite where nine pairs in ten routed differently. Those rows now leave **every** policy rate: they measure the harness (wrong toolset attached, wrong prompt, a suite promoted from a different agent), not the migration. They are still reported — as a `TOOL_GROUND_TRUTH_MISS` count in `failure_categories` and as a recommendations line naming how many were excluded. Only the *shared-height* case is dropped: a conformance row the target lost ground on (`0.8 / 0.3`) is still a regression and one it improved on (`0.2 / 0.6`) is still an improvement. A run whose every blocking row was a shared miss is `inconclusive`, and its recommendation is **Fix the eval harness before collecting more examples** — more pairs from the same setup are more excluded rows, so the denominator stays empty however many are added. The exclusion is deliberately narrower than the diagnosis, which is why `evaluate` runs its own broken-harness check over the **source side alone**: a suite the source fails and the target happens to satisfy has a *positive* delta and no `TOOL_GROUND_TRUTH_MISS` tag, so the exclusion rule cannot see it and it is exactly as misconfigured. - **A cost/latency ratio measured only from zeroes says so.** When the calls exist but every `cost_usd` (or `latency_ms`) is `0` on both models, `analyze` adds a recommendations line beside the `conclusive: false`: `The cost increase budget could not be measured: all 4 error-free calls across both models recorded a cost of 0, so its observed 0.00 is a default, not a measurement.` A run with no calls gets no line — the empty `raw.jsonl` already explains itself. Emitted once per run, since every scope reads the same calls. - **Every budget reports its own `denominator`** — the sample `observed` was computed over. Scored records for the regression rate, the equivalence rate and the critical count — counted over *measurements*, so an evaluator scoring two axes contributes two rows per example; `tool_arguments` rows for tool drift; `tool_selection.divergence` rows for tool divergence; the distinct examples with a blocking `trace_invariants` row for the trace-rule count; the error-free calls behind both averages for the cost and latency ratios. Slices report their own counts. `0` means "counted, and the sample was empty", so `observed` is a default; a *missing* `denominator` means "no sample size reported" and is **not** zero — only bundles written before the field say that, and the hosted gate falls back to `conclusive` for them. It is the same number the `1/n` granularity warning is judged on, and it is orthogonal to `conclusive`: an all-zero cost ratio counted every call it averaged and still measured nothing, so it reports a positive denominator beside `conclusive: false`. The hosted gate derives its own Wilson interval from these denominators, over the same three rate budgets and with the same confidence constant the CLI uses, so a local verdict and a hosted one now agree on whether a breach was confirmed — see [methodology](docs/methodology.md). -- Any conclusive budget failure, or any blocking critical/high-severity comparison → **`fail`**. Any lower-severity blocking regression → **`conditional_pass`**. Otherwise → **`pass`**. +- Any conclusive budget failure, or any blocking critical/high-severity comparison → **`fail`**. Any lower-severity blocking regression → **`conditional_pass`**. Otherwise → **`pass`**. When severity alone fails the run, `reason` names each blocking comparison. - **`migration_policy` is the single source of truth for these budgets.** `migration_decision.json`'s `policy` field is the resolved policy `analyze` computed the verdict under — every top-level budget with its default applied, plus `slices`. `bundle`/`push` do not read `migration_decision.json`; `bundle` re-resolves the verdict from `evalshift.yaml` at bundle time, so the bundle's `decision.policy` (see [Hosted EvalShift](#hosted-evalshift)) is the policy the config held when the bundle was built — editing `migration_policy` between `analyze` and `bundle` changes what the bundle carries. `push ` builds a bundle only when `run_bundle.json.gz` is missing — an existing bundle is uploaded as-is — so re-run `evalshift bundle ` after a config edit. Either way, that is what lets the hosted gate check a pull request against exactly the budgets the bundle's own verdict used, instead of a separate, web-edited policy. `null` when no `migration_policy` is configured, and on a `migration_decision.json` written before this field existed. `evalshift.yaml` is the source of truth for that policy, and the web app's project policy view is becoming a read-only display of the snapshot each run pushed. Because every budget is resolved, the snapshot carries `max_invariant_violations` (default `0`) even for a config with no trace rules; hosted EvalShift accepts and stores it (since 2026-10-08) but never re-evaluates it — a run that carries its own policy is answered with the CLI's decision, which did check it, and only runs pushed without a policy fall back to the server's six-budget re-check. - **A slice budget gates the run exactly like an overall one.** The budgets under `migration_policy.slices` are evaluated on the same terms as the top-level ones: a conclusively breached slice budget **fails** the run, and an unconfirmed breach makes it `inconclusive` — the same Wilson rule, counted over that slice's own denominator. Since the overall rows can all be green in a run a slice budget fails, `recommendations` names the one that blocked (`The 'security' slice breached its overall regression rate budget (the share of scored comparisons where the target did worse): 20% over n=20 vs the 0% limit.`) and the `inconclusive` `reason` scope-qualifies it the same way. Per-slice verdicts under `slices[*].verdict` are unchanged, and a slice that fails on *comparison severity* rather than a budget still only downgrades an overall `pass` to `conditional_pass`. -- Semantic drift that stays above `min_similarity` counts as equivalent, not regression. +- Semantic drift that stays above `min_similarity` counts as equivalent, not regression — in the budgets and in the paired test. CI wiring (on `analyze` and `compare`): diff --git a/docs/agents.md b/docs/agents.md index 1a2b593..ab2d6ec 100644 --- a/docs/agents.md +++ b/docs/agents.md @@ -117,10 +117,10 @@ suite's rows actually contain and writes that suite's own evaluator block — `tool_selection` when any row was offered a toolset, `tool_arguments` when any row recorded call arguments, and nothing at all for a suite whose captures never called a tool. That last case is the point: a tool evaluator pointed at a -tool-free suite scores an empty denominator, which the policy reads as an -*inconclusive* gate rather than as "not applicable here". A family a suite -declares replaces the top-level one wholesale; families it does not mention are -inherited. See +tool-free suite has nothing to check: it skips every pair that was offered no +tools — no row, no coverage entry, no effect on the verdict — and `evaluate` +(or `compare`, under its verdict) warns that it checked nothing. A family a suite declares replaces the top-level +one wholesale; families it does not mention are inherited. See [Configuration → per-suite evaluators](configuration.md#per-suite-evaluators). Hand edits inside the markers are regenerated away on the next sync. To keep diff --git a/docs/configuration.md b/docs/configuration.md index 0142a2e..a70310d 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -239,10 +239,12 @@ changes which rows the tool returns. Lower it to `0.4` if you want presence differences ignored. Semantic-evaluator drift counts toward `max_overall_regression_rate` and -`min_equivalence_rate` only when it breaches +`min_equivalence_rate` — and toward the paired test that sets each comparison's +severity — only when it breaches [`evaluators.semantic.min_similarity`](#evaluatorssemantic). A near-identical output (cosine ~0.98) that stays within `min_similarity` is treated as -*equivalent*, not a regression — so the policy gate and the report agree. +*equivalent*, not a regression, by the budgets and the test alike — so the +policy gate, the severity and the report agree. Per-slice overrides live under `migration_policy.slices` — a map of slice name (an example tag: every distinct tag is a slice) to a partial @@ -953,9 +955,9 @@ only when `suites:` is empty. A project's suites are rarely homogeneous: one calls tools, six answer in prose. One top-level `evaluators:` block either leaves the tool-calling suite unmeasured -or hands the tool-free ones a tool evaluator with an empty denominator, which -reads as an *inconclusive* gate rather than as "not applicable here". So a suite -can carry its own block: +or hands the tool-free ones tool evaluators with nothing to check — they skip +every pair that was offered no tools (no row, no coverage entry, and `evaluate` — +or `compare`, under its verdict — warns that they checked nothing). So a suite can carry its own block: ```yaml suites: @@ -1135,7 +1137,7 @@ Each suite's entry is generated from what that suite's own rows contain, tool evaluators included, so nothing has to be wired by hand: - No row was offered a toolset → no `evaluators:` block at all; the suite - inherits the top level, and no tool evaluator scores an empty denominator. + inherits the top level; a tool evaluator it inherits skips its tool-free pairs. - Any row was offered a toolset → `tool_selection: [{name: routing, conformance: expected, divergence: set}]`. - Any row recorded tool-call arguments → `tool_arguments: [{name: routing_args, diff --git a/docs/methodology.md b/docs/methodology.md index 8be806f..77cebde 100644 --- a/docs/methodology.md +++ b/docs/methodology.md @@ -112,6 +112,29 @@ After BH correction, each comparison gets a severity tag based on the | `none` | adjusted p ≥ 0.05 | | `insufficient` | n < 5 | +### Drift inside `min_similarity` is tested as no change + +`semantic` pins the source at 1.0, so every target that words anything +differently scores below it: a negative delta is the evaluator's normal output, +not evidence of a regression. Its rows therefore enter the paired test +**tolerance-adjusted** — a row within `min_similarity` (no `SEMANTIC_REGRESSION` +flag) contributes a delta of `0`, a flagged row its full drop. That is the line +the policy budgets already draw, so the two layers cannot disagree: nine rows at +0.90–0.98 once passed every budget and still failed the run as a `critical` +regression, because nine small, consistent drops have almost no variance. +`delta_avg_score` stays the raw mean — it describes the drift; severity judges +it — and the comparison carries the note `k of n rows within the evaluator's own +threshold (min_similarity) — tested as no change`. A comparison where only a few +rows are flagged may not reach significance on its own; the rate budgets +(`max_overall_regression_rate`, `min_equivalence_rate`) decide it. + +### Which severities fail the run + +A `critical` or `high` comparison from a blocking evaluator fails the run on its +own, even when every budget holds; `medium` and `low` downgrade a `pass` to +`conditional_pass`. When severity alone failed the run, the decision's `reason` +names each such comparison, so the report's verdict line explains the FAIL. + ## How policy metrics find their rows `migration_policy` budgets are computed over subsets of the scored records — @@ -152,7 +175,9 @@ a call counts as drifted only when its target score falls below the floor. The rate then measures what its name promises — the fraction of tool calls whose arguments are *materially* wrong — and the budget becomes something you can act on. The same reasoning is why semantic drift is counted against the evaluator's -own `min_similarity` rather than against `delta < 0`. +own `min_similarity` rather than against `delta < 0` — by the budgets and by the +paired test alike (see +[Drift inside `min_similarity` is tested as no change](#drift-inside-min_similarity-is-tested-as-no-change)). ### A zero delta both models earned by failing is not equivalence @@ -471,6 +496,14 @@ comparison carries an `advisory:` note — riding in `notes`, like the **Not enough data** headline the rest of `insufficient` carries — the sample was absent, not small. +A tool evaluator is never handed a pair that was offered no tools: both calls +error-free and neither carrying a tool trace is a plain text exchange with no +tool call to check. That pair gets no row and no coverage entry, so it counts +toward no note and has no effect on the verdict — and when it is every pair the +evaluator was scoped to, `evaluate` warns that it checked nothing. A pair with +an errored call keeps its error row: a failed call says nothing about whether +tools were offered. + ## What we always report Every comparison row in `analysis.json` and `report.html` carries: diff --git a/docs/superpowers/plans/2026-10-11-semantic-gate-and-tool-free-suites.md b/docs/superpowers/plans/2026-10-11-semantic-gate-and-tool-free-suites.md new file mode 100644 index 0000000..9360e05 --- /dev/null +++ b/docs/superpowers/plans/2026-10-11-semantic-gate-and-tool-free-suites.md @@ -0,0 +1,1311 @@ +# Semantic Gate + Tool-Free Suites Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A run whose semantic drift stays inside `min_similarity` and whose suite offers no tools gets the verdict its budgets earned (`pass` for `r_20261011_triage_e14cda`), and the report never claims a budget breach that did not happen. + +**Architecture:** One shared predicate (`analysis/tolerance.py::within_tolerance`) defines "drop the evaluator's own threshold forgives". The policy budgets already used that rule; the slicer now stamps it on each `SlicedScore` so the paired test runs on tolerance-adjusted deltas while every descriptive number stays raw. Separately, `evaluate` stops attempting tool evaluators on pairs that were offered no tools — the same no-cell treatment an `applies_to` miss gets — and warns once with the real cause. Three wording fixes (`reason`, fallback recommendation, critical count) ride with it. + +**Tech Stack:** Python 3.11+, numpy, scipy, pydantic v2, Typer + Rich, pytest, uv. + +**Spec:** `docs/superpowers/specs/2026-10-11-semantic-gate-and-tool-free-suites-design.md` — read it first; this plan argues from it. + +## Global Constraints + +- Python floor 3.11: no 3.12+ syntax. Every module starts with `from __future__ import annotations`; public functions get Google-style docstrings; `mypy --strict src/evalshift_cli` stays clean. +- No config, suite, `state.json`, `scores.jsonl` or bundle schema change. `ComparisonResult` gets **no new field** (the bundle's `Comparison` is `additionalProperties: false`); new information rides in `notes`. +- `delta_avg_score`, slice `aggregates`, per-example rows: **raw** deltas, unchanged. Only the test statistic, p-value, effect size, its CI, and therefore severity see tolerance-adjusted deltas. +- User-facing strings, verbatim: + - comparison note: `"{k} of {n} rows within the evaluator's own threshold (min_similarity) — tested as no change"` + - severity reason, per blocking comparison joined with `"; "`: `"{evaluator} regressed at {severity} severity on prompt {prompt} (effect size {d:+.2f}, adjusted p {p:.4f})"`, then `". A critical or high regression from a blocking evaluator fails the run on its own, whatever the budgets say."` + - tool-free warning: `"[yellow]⚠[/yellow] {kind} evaluator '{name}' checked nothing: no example in {suite 'x' | this run} offered the models any tools, so there were no tool calls to check. It did not gate this run."` + - fallback recommendation (fail): budgets → `"Do not migrate under the configured policy — the {labels} {budget was|budgets were} breached. Address the blocking evaluators, then re-run."`; severity → `"Do not migrate under the configured policy — {evaluators} regressed at blocking severity although no budget was breached. Review {its|their} regressed examples, then re-run."`; neither → `"Do not migrate under the configured policy — see the verdict's reason, address it, then re-run."` + - verdict summary: `"… with {b} blocking {regression|regressions} ({c} critical) across {n} examples."` +- Gate ("green" = all four, as CI): `uv run ruff check . && uv run ruff format --check . && uv run mypy --strict src/evalshift_cli && uv run pytest` (coverage floor 90). `make ci` is the exact CI mirror. +- Commit with `uv run --with pre-commit git commit …` (the shared hook points at a deleted venv). Never `--no-verify`. Conventional Commits, ending with `Co-Authored-By: Claude Opus 5.5 (1M context) `. `docs/superpowers/` is gitignored: add files there with `git add -f`. +- Always `git -C /home/lukas/repos/evalshift/evalshift-cli` (= `$WT`); never run git from a sibling repo's cwd. Before each commit, confirm the branch is `fix/semantic-gate-tool-free-suites`, not `main`. + +## Review Focus + +- A semantic comparison where most rows are forgiven and one is flagged (`[0]*8 + [-0.4]`) must not crash scipy or report `critical` — Task 2 `test_one_flagged_row_among_forgiven_ones_runs_and_stays_quiet`. +- Rows checkpointed before `EvalRecord.kind` existed (`kind=""`, name `semantic.cosine`) and a semantic evaluator the user renamed (`kind="semantic"`, name `meaning`) are both forgiven — Task 1 `test_legacy_rows_fall_back_to_the_name_prefix`, `test_a_renamed_semantic_evaluator_is_selected_by_kind`. +- A tool-free pair whose call **errored** keeps its error row, and a pair where only **one** side has a trace stays unmeasured — Task 5 `test_an_errored_tool_free_pair_keeps_its_error_row`, `test_a_one_sided_trace_is_still_unmeasured`. +- A run that both breaches a budget and has a critical comparison: `reason` stays `None` (the hero's budget count explains it) and the recommendation names the budget, not the severity — Task 3 `test_a_budget_breach_keeps_the_budget_line`, Task 4 `test_a_budget_fail_names_the_budget`. +- An `applies_to` that matches nothing on a tool-free run still gets the `applies_to` wording, not the tool-free one — Task 5 `test_an_applies_to_miss_keeps_its_own_wording_on_a_tool_free_run`. + +--- + +### Task 0: Branch (done 2026-10-11) + +**Files:** none. + +Work happens in the **main checkout** on branch `fix/semantic-gate-tool-free-suites`, cut from +`origin/main` at `1045843` — no worktree. The spec and this plan are committed on it. +`$WT` below always means `/home/lukas/repos/evalshift/evalshift-cli`. + +- [x] **Step 1: Create the branch from an up-to-date `main`** + +```bash +git -C /home/lukas/repos/evalshift/evalshift-cli fetch origin +git -C /home/lukas/repos/evalshift/evalshift-cli switch -c fix/semantic-gate-tool-free-suites +``` + +- [x] **Step 2: Commit the spec and plan** + +```bash +WT=/home/lukas/repos/evalshift/evalshift-cli +git -C $WT add -f docs/superpowers/specs/2026-10-11-semantic-gate-and-tool-free-suites-design.md docs/superpowers/plans/2026-10-11-semantic-gate-and-tool-free-suites.md +cd $WT && uv run --with pre-commit git commit -m "docs(plan): semantic gate and tool-free suites" +``` + +- [ ] **Step 3: Baseline the suite before Task 1** + +```bash +WT=/home/lukas/repos/evalshift/evalshift-cli +git -C $WT branch --show-current # must print fix/semantic-gate-tool-free-suites +cd $WT && uv sync --extra dev && uv run pytest -q -x -p no:cacheprovider 2>&1 | tail -3 +``` +Expected: all pass (≈3000 tests). + +--- + +### Task 1: One definition of "within tolerance" + +**Files:** +- Create: `src/evalshift_cli/analysis/tolerance.py` +- Modify: `src/evalshift_cli/analysis/policy.py:24-29` (imports), `:38`, `:59-62`, `:664-690`, `:733` +- Test: `tests/unit/test_tolerance.py` + +**Interfaces:** +- Consumes: `EvalRecord` (`evaluators/base.py`), `SEMANTIC_REGRESSION` (`evaluators/failures.py`). +- Produces: `SEMANTIC_KIND: str = "semantic"`, `is_semantic(record: EvalRecord) -> bool`, `is_semantic_regression(record: EvalRecord) -> bool`, `within_tolerance(record: EvalRecord) -> bool`. + +- [ ] **Step 1: Write the failing tests** + +Create `tests/unit/test_tolerance.py`: + +```python +"""Tests for the shared within-tolerance predicate.""" + +from __future__ import annotations + +from evalshift_cli.analysis.tolerance import ( + is_semantic, + is_semantic_regression, + within_tolerance, +) +from evalshift_cli.evaluators.base import EvalRecord +from evalshift_cli.evaluators.failures import SEMANTIC_REGRESSION + + +def _record( + *, + kind: str = "semantic", + name: str = "semantic.cosine", + target: float = 0.95, + flagged: bool = False, +) -> EvalRecord: + return EvalRecord( + run_id="r1", + prompt_id="p", + example_id="ex1", + evaluator_name=name, + kind=kind, + source_score=1.0, + target_score=target, + delta=target - 1.0, + metadata={"failure_categories": [SEMANTIC_REGRESSION]} if flagged else {}, + ) + + +def test_unflagged_semantic_drift_is_within_tolerance() -> None: + assert within_tolerance(_record(target=0.904)) + + +def test_flagged_semantic_drift_is_not() -> None: + record = _record(target=0.75, flagged=True) + assert is_semantic_regression(record) + assert not within_tolerance(record) + + +def test_other_kinds_are_never_within_tolerance() -> None: + assert not within_tolerance(_record(kind="llm_judge", name="llm_judge.equivalence")) + assert not within_tolerance(_record(kind="", name="structural.length")) + + +def test_legacy_rows_fall_back_to_the_name_prefix() -> None: + legacy = _record(kind="", name="semantic.cosine") + assert is_semantic(legacy) + assert within_tolerance(legacy) + + +def test_a_renamed_semantic_evaluator_is_selected_by_kind() -> None: + assert within_tolerance(_record(kind="semantic", name="meaning")) + + +def test_a_malformed_category_list_is_not_a_regression() -> None: + record = _record().model_copy(update={"metadata": {"failure_categories": "SEMANTIC"}}) + assert not is_semantic_regression(record) + assert within_tolerance(record) +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `uv run pytest tests/unit/test_tolerance.py -q` +Expected: FAIL — `ModuleNotFoundError: No module named 'evalshift_cli.analysis.tolerance'`. + +- [ ] **Step 3: Create the module** + +Create `src/evalshift_cli/analysis/tolerance.py`: + +```python +"""Which score drops an evaluator's own threshold already forgives. + +``semantic`` pins the source at 1.0, so every target that words anything +differently scores below it: a negative delta is the evaluator's *normal* +output, not evidence of a regression. ``min_similarity`` is where the user +drew the line, and the evaluator records which side of it a row fell on as a +``SEMANTIC_REGRESSION`` failure category at scoring time. + +The policy budgets and the paired significance test must read that one line. +When only the budgets did, nine rows at 0.90-0.98 passed every budget and still +failed the run as a "critical" regression — nine small, consistent drops have +almost no variance, so the test was certain of a difference nobody had asked +it to find. +""" + +from __future__ import annotations + +from evalshift_cli.evaluators.base import EvalRecord +from evalshift_cli.evaluators.failures import SEMANTIC_REGRESSION + +#: The ``EvalRecord.kind`` slug the semantic evaluator stamps. +SEMANTIC_KIND = "semantic" + +#: Records checkpointed before ``EvalRecord.kind`` existed carry only the name +#: prefix the evaluator was registered under. +_LEGACY_SEMANTIC_PREFIX = "semantic." + + +def is_semantic(record: EvalRecord) -> bool: + """Whether ``record`` came from a ``semantic`` evaluator. + + Selected by kind, so a user who renames the evaluator keeps the rule; + falls back to the legacy name prefix only for rows written without one. + """ + if record.kind: + return record.kind == SEMANTIC_KIND + return record.evaluator_name.startswith(_LEGACY_SEMANTIC_PREFIX) + + +def is_semantic_regression(record: EvalRecord) -> bool: + """Whether a semantic record breached ``min_similarity``. + + Reads the ``SEMANTIC_REGRESSION`` flag the evaluator wrote at scoring time + rather than re-deriving it from ``delta``: the threshold lives in config, + and the evaluator is the one place that knew it. + """ + categories = record.metadata.get("failure_categories", []) + return isinstance(categories, list) and SEMANTIC_REGRESSION in categories + + +def within_tolerance(record: EvalRecord) -> bool: + """Whether ``record``'s drop is inside its evaluator's own threshold. + + True only for a semantic row that stayed above ``min_similarity``. Such a + row counts as *equivalent* in every policy rate and as *no change* in the + paired test; every other row is taken at its delta. + """ + return is_semantic(record) and not is_semantic_regression(record) + + +__all__ = [ + "SEMANTIC_KIND", + "is_semantic", + "is_semantic_regression", + "within_tolerance", +] +``` + +- [ ] **Step 4: Run the new tests** + +Run: `uv run pytest tests/unit/test_tolerance.py -q` +Expected: 6 passed. + +- [ ] **Step 5: Make the policy read the shared predicate** + +In `src/evalshift_cli/analysis/policy.py`: + +1. Imports (lines 16-31): add `from evalshift_cli.analysis.tolerance import is_semantic, is_semantic_regression, within_tolerance` after the `statistics` import, and drop `SEMANTIC_REGRESSION,` from the `evaluators.failures` import list (nothing else in the module uses it — confirm with `grep -n SEMANTIC_REGRESSION src/evalshift_cli/analysis/policy.py`). +2. Delete line 38 `_SEMANTIC_KIND = "semantic"`. +3. In `_LEGACY_KIND_PREFIXES` (lines 59-62) delete the `_SEMANTIC_KIND: "semantic.",` entry (the legacy prefix now lives in `tolerance.py`). +4. Replace lines 664-690 (`_is_semantic_regression`, `_is_regression`, `_is_equivalent`) with: + +```python +def _is_regression(r: EvalRecord) -> bool: + """Whether a record counts as a regression for policy budgets.""" + if is_semantic(r): + return is_semantic_regression(r) + return r.delta < 0 + + +def _is_equivalent(r: EvalRecord) -> bool: + """Whether a record counts as source/target equivalent. + + Semantic drift within ``min_similarity`` is treated as equivalent (not a + regression), keeping the improved/equivalent/regressed partition intact. + The same rule decides what the paired test sees — see + :mod:`evalshift_cli.analysis.tolerance`. + """ + if is_semantic(r): + return within_tolerance(r) + return r.delta == 0 +``` + +5. In `is_shared_ground_truth_miss`'s docstring (line ~733) change ``:func:`_is_semantic_regression` `` to ``:func:`~evalshift_cli.analysis.tolerance.is_semantic_regression` ``. + +- [ ] **Step 6: Run the policy tests and the type check** + +Run: `uv run pytest tests/unit/test_policy.py tests/unit/test_tolerance.py -q && uv run mypy --strict src/evalshift_cli/analysis` +Expected: all pass (in particular `test_unflagged_semantic_drift_is_equivalent_not_regression`, `test_flagged_semantic_drift_counts_as_regression`, `test_semantic_regression_is_selected_by_kind`); mypy `Success`. + +- [ ] **Step 7: Commit** + +```bash +git -C $WT add src/evalshift_cli/analysis/tolerance.py src/evalshift_cli/analysis/policy.py tests/unit/test_tolerance.py +cd $WT && uv run --with pre-commit git commit -m "refactor(analysis): share the semantic within-tolerance rule + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 2: The paired test runs on tolerance-adjusted deltas + +**Files:** +- Modify: `src/evalshift_cli/analysis/slicing.py` (`SlicedScore`, `build_slices`) +- Modify: `src/evalshift_cli/analysis/statistics.py` (`WITHIN_TOLERANCE_REASON`, `_one_comparison`, `__all__`) +- Test: `tests/unit/test_analysis.py`, `tests/unit/test_policy.py` + +**Interfaces:** +- Consumes: `within_tolerance(record) -> bool` (Task 1). +- Produces: `SlicedScore.within_tolerance: bool = False` (last field, defaulted); `statistics.WITHIN_TOLERANCE_REASON = "tested as no change"`. + +- [ ] **Step 1: Write the failing statistics tests** + +Add to the imports of `tests/unit/test_analysis.py`: + +```python +from scipy import stats + +from evalshift_cli.analysis.statistics import WITHIN_TOLERANCE_REASON +from evalshift_cli.evaluators.failures import SEMANTIC_REGRESSION +``` + +(merge `WITHIN_TOLERANCE_REASON` into the existing `evalshift_cli.analysis.statistics` import block rather than adding a second one). Append at the end of the file: + +```python +# --------------------------------------------------------------------------- +# Semantic drift inside min_similarity — tested as no change +# --------------------------------------------------------------------------- + +#: The nine semantic.cosine target scores of run r_20261011_triage_e14cda, +#: every one above min_similarity 0.9 — the run that failed as "critical". +_TRIAGE_TARGETS = [0.9734, 0.9608, 0.9212, 0.9531, 0.9321, 0.9492, 0.9043, 0.9800, 0.9542] + + +def _semantic_record(example: str, target: float, *, flagged: bool) -> EvalRecord: + return EvalRecord( + run_id="r1", + prompt_id="replay", + example_id=example, + evaluator_name="semantic.cosine", + kind="semantic", + source_score=1.0, + target_score=target, + delta=target - 1.0, + metadata={"failure_categories": [SEMANTIC_REGRESSION]} if flagged else {}, + ) + + +def _semantic_sliced(targets: list[float], flagged: list[bool]) -> dict[str, list[SlicedScore]]: + records = [ + _semantic_record(f"ex{i}", t, flagged=f) + for i, (t, f) in enumerate(zip(targets, flagged, strict=True)) + ] + suite = Suite(examples=[suite_example(id=r.example_id) for r in records]) + return build_slices(records=records, suite=suite) + + +class TestWithinTolerance: + def test_build_slices_marks_only_unflagged_semantic_rows(self) -> None: + records = [ + _semantic_record("ex1", 0.95, flagged=False), + _semantic_record("ex2", 0.70, flagged=True), + _record(example="ex3", delta=-0.2), + ] + suite = Suite(examples=[suite_example(id=f"ex{i}") for i in (1, 2, 3)]) + sliced = build_slices(records=records, suite=suite)[ALL_SLICE] + assert [s.within_tolerance for s in sliced] == [True, False, False] + + def test_drift_inside_min_similarity_is_not_a_regression(self) -> None: + sliced = _semantic_sliced(_TRIAGE_TARGETS, [False] * 9) + (c,) = analyze(sliced_by_slice=sliced) + assert c.severity == "none" + assert c.test == "skipped" + # Descriptive mean stays the raw drift: the report still says -5%. + assert c.delta_avg_score == pytest.approx( + sum(t - 1.0 for t in _TRIAGE_TARGETS) / 9, + ) + assert ( + "9 of 9 rows within the evaluator's own threshold (min_similarity) — " + f"{WITHIN_TOLERANCE_REASON}" + ) in c.notes + + def test_drift_below_min_similarity_still_regresses(self) -> None: + targets = [0.70, 0.72, 0.75, 0.71, 0.74, 0.73, 0.76, 0.69, 0.72] + (c,) = analyze(sliced_by_slice=_semantic_sliced(targets, [True] * 9)) + assert c.severity == "critical" + assert not any(WITHIN_TOLERANCE_REASON in note for note in c.notes) + + def test_one_flagged_row_among_forgiven_ones_runs_and_stays_quiet(self) -> None: + targets = [*_TRIAGE_TARGETS[:8], 0.60] + flagged = [False] * 8 + [True] + (c,) = analyze(sliced_by_slice=_semantic_sliced(targets, flagged)) + assert c.test in ("paired_t", "wilcoxon") + assert c.severity == "none" + assert any(note.startswith("8 of 9 rows within") for note in c.notes) + + def test_non_semantic_statistics_are_unchanged(self) -> None: + pairs = _ss(40, -0.3) + (c,) = analyze(sliced_by_slice={"all": pairs}) + assert c.test == "paired_t" + expected = stats.ttest_rel( + [p.target_score for p in pairs], + [p.source_score for p in pairs], + ) + assert c.statistic == pytest.approx(float(expected.statistic)) + assert c.p_value == pytest.approx(float(expected.pvalue)) + + def test_aggregates_keep_the_raw_drift(self) -> None: + sliced = _semantic_sliced(_TRIAGE_TARGETS, [False] * 9) + agg = aggregates(sliced[ALL_SLICE], ALL_SLICE) + assert agg.delta_min_score == pytest.approx(min(_TRIAGE_TARGETS) - 1.0) +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `uv run pytest tests/unit/test_analysis.py -q -k "WithinTolerance"` +Expected: FAIL — `ImportError: cannot import name 'WITHIN_TOLERANCE_REASON'`. + +- [ ] **Step 3: Stamp the flag in the slicer** + +In `src/evalshift_cli/analysis/slicing.py`, add the import `from evalshift_cli.analysis.tolerance import within_tolerance` after the `EvalRecord` import, and add the field at the end of `SlicedScore` (after `kind`): + +```python + #: The row's drop is inside its evaluator's own threshold — a semantic row + #: above ``min_similarity`` — so the paired test reads it as no change, + #: exactly as the policy budgets count it (see + #: :mod:`evalshift_cli.analysis.tolerance`). ``delta`` stays raw for the + #: aggregates. Defaulted for the same reason as ``kind``. + within_tolerance: bool = False +``` + +In `build_slices`, add the keyword to the `SlicedScore(...)` construction: + +```python + kind=rec.kind, + within_tolerance=within_tolerance(rec), + ) +``` + +- [ ] **Step 4: Test the adjusted deltas in `_one_comparison`** + +In `src/evalshift_cli/analysis/statistics.py`: + +1. After `NOT_APPLICABLE_REASON` (line 89) add: + +```python +#: Suffix of the note counting rows whose drop sat inside the evaluator's own +#: threshold (a semantic row above ``min_similarity``). Those rows enter the +#: test as a delta of zero — see :mod:`evalshift_cli.analysis.tolerance` — +#: and the note is what tells a reader why a visible average drift tested as +#: no change. +WITHIN_TOLERANCE_REASON = "tested as no change" +``` + +and add `"WITHIN_TOLERANCE_REASON",` to `__all__` (alphabetical, after `"UNMEASURED_NOTE_PREFIX",`). + +2. In `_one_comparison`, replace the opening (lines 279-287) with: + +```python + n = len(pairs) + deltas = np.array([p.delta for p in pairs], dtype=float) + # What the test sees: a drop inside the evaluator's own threshold is no + # change, exactly as the policy budgets count it. ``deltas`` stays raw + # for ``delta_avg_score``, which describes the drift rather than judging + # it — the sign of the two means always agrees, because a forgiven row is + # a non-positive delta set to zero. + tested = np.array([0.0 if p.within_tolerance else p.delta for p in pairs], dtype=float) + forgiven = sum(1 for p in pairs if p.within_tolerance) + notes: list[str] = [] + if name_axis: + notes.append(f"{AXIS_NOTE_PREFIX} {kind}") + if n_unmeasured: + notes.append( + f"{n_unmeasured} of {n + n_unmeasured} rows not applicable — {NOT_APPLICABLE_REASON}", + ) + if forgiven: + notes.append( + f"{forgiven} of {n} rows within the evaluator's own threshold " + f"(min_similarity) — {WITHIN_TOLERANCE_REASON}", + ) +``` + +3. Zero-variance check (line 344): `if float(np.std(tested, ddof=1)) < 1e-9:` — the body keeps `delta_avg_score=float(np.mean(deltas))`. + +4. Replace the test block (lines 363-383) with: + +```python + # Pick the test based on a normality screen. + use_t = _is_normal(tested) + + if use_t: + # One-sample t on the paired deltas is ``ttest_rel(target, source)`` + # exactly; it is written this way so a forgiven row can be zero. + result = stats.ttest_1samp(tested, 0.0) + statistic = float(result.statistic) + p_value = float(result.pvalue) + d = _cohens_d_paired(tested) + ci_low, ci_high = _t_test_d_ci(d=d, n=n) + test_kind: TestKind = "paired_t" + else: + result = stats.wilcoxon(tested, zero_method="wilcox", correction=False) + statistic = float(result.statistic) + p_value = float(result.pvalue) + d = _cohens_d_paired(tested) + ci_low, ci_high = _bootstrap_d_ci(tested, rng=rng) + test_kind = "wilcoxon" + notes.append("non-normal deltas; used Wilcoxon") +``` + +The final `return ComparisonResult(...)` keeps `delta_avg_score=float(np.mean(deltas))` (raw). + +5. Update the module docstring's step 3 to `Compute Cohen's d for paired samples (mean(deltas)/std(deltas)) over the tolerance-adjusted deltas.` + +- [ ] **Step 5: Run the statistics tests** + +Run: `uv run pytest tests/unit/test_analysis.py -q` +Expected: all pass, including the six new `TestWithinTolerance` tests and the existing `test_zero_variance_skipped` (positional `SlicedScore` still works — the new field is last and defaulted). + +- [ ] **Step 6: Write the pipeline regression test for the triage run** + +In `tests/unit/test_policy.py`, add `from evalshift_cli.analysis.slicing import build_slices`, add `analyze` to the `evalshift_cli.analysis.statistics` import, add `SEMANTIC_REGRESSION` to the `evalshift_cli.evaluators.failures` import, and add `from evalshift_cli.suite.models import Suite` and `from tests.unit.suite_examples import suite_example`. Append: + +```python +class TestTriageRunRegression: + """r_20261011_triage_e14cda: every budget held and the run still failed.""" + + _TARGETS = [0.9734, 0.9608, 0.9212, 0.9531, 0.9321, 0.9492, 0.9043, 0.9800, 0.9542] + + def _records(self, *, flagged: bool) -> list[EvalRecord]: + semantic = [ + EvalRecord( + run_id="r1", + prompt_id="replay", + example_id=f"ex{i}", + evaluator_name="semantic.cosine", + kind="semantic", + source_score=1.0, + target_score=t, + delta=t - 1.0, + metadata={"failure_categories": [SEMANTIC_REGRESSION]} if flagged else {}, + ) + for i, t in enumerate(self._TARGETS) + ] + judge = [ + EvalRecord( + run_id="r1", + prompt_id="replay", + example_id=f"ex{i}", + evaluator_name="llm_judge.equivalence", + kind="llm_judge", + source_score=0.0 if i < 7 else 0.5, + target_score=1.0 if i < 7 else 0.5, + delta=1.0 if i < 7 else 0.0, + blocking=False, + ) + for i in range(9) + ] + return [*semantic, *judge] + + def _decide(self, records: list[EvalRecord]) -> MigrationDecision: + suite = Suite(examples=[suite_example(id=f"ex{i}") for i in range(9)]) + comparisons = analyze(sliced_by_slice=build_slices(records=records, suite=suite)) + return evaluate_migration_policy( + run_id="r1", + source_model="anthropic/claude-sonnet-5", + target_model="anthropic/claude-sonnet-5-5", + policy=MigrationPolicy( + max_overall_regression_rate=0.3, + max_critical_regressions=1, + min_equivalence_rate=0.75, + max_tool_argument_drift=0.2, + max_tool_divergence=0.2, + max_cost_increase=0.5, + max_latency_increase=0.3, + ), + comparisons=comparisons, + records=records, + calls=[], + ) + + def test_drift_inside_min_similarity_passes(self) -> None: + decision = self._decide(self._records(flagged=False)) + assert decision.verdict == "pass" + assert decision.blocking_regressions == [] + assert decision.recommendations[0] == "Safe to migrate under the configured policy." + + def test_the_same_drop_below_min_similarity_still_fails(self) -> None: + decision = self._decide(self._records(flagged=True)) + assert decision.verdict == "fail" +``` + +- [ ] **Step 7: Run it — it passes with Task 2's code already in** + +Run: `uv run pytest tests/unit/test_policy.py -q -k TriageRunRegression` +Expected: 2 passed. (To see it fail as a check, `git -C $WT stash` the two `src/` files, rerun → `test_drift_inside_min_similarity_passes` fails with `'fail' == 'pass'`; `git -C $WT stash pop`.) + +- [ ] **Step 8: Full gate, then commit** + +Run: `uv run ruff check . && uv run ruff format --check . && uv run mypy --strict src/evalshift_cli && uv run pytest -q` +Expected: green. If any existing test pinned a semantic comparison's `statistic`/`p_value` from raw within-tolerance rows, it was asserting the bug: update its expectation to the tolerance-adjusted value and say so in the commit body. + +```bash +git -C $WT add src/evalshift_cli/analysis/slicing.py src/evalshift_cli/analysis/statistics.py tests/unit/test_analysis.py tests/unit/test_policy.py +cd $WT && uv run --with pre-commit git commit -m "fix(analysis): test semantic drift against min_similarity, not raw deltas + +semantic pins the source at 1.0, so any rewording is a negative delta. The +paired test ran on those raw deltas and called nine consistent 0.90-0.98 +rows 'critical', failing a run whose every budget passed. Rows inside +min_similarity now enter the test as no change, matching the budgets; +delta_avg_score and the aggregates stay raw. + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 3: A severity-driven fail says why + +**Files:** +- Modify: `src/evalshift_cli/analysis/policy.py` (`evaluate_migration_policy` lines 537-541; new `_severity_reason` after `_invariant_reason`) +- Test: `tests/unit/test_policy.py` + +**Interfaces:** +- Consumes: `_BLOCKING_SEVERITIES`, `BudgetResult`, `ComparisonResult` (existing). +- Produces: `_severity_reason(comparisons: list[ComparisonResult], budgets: list[BudgetResult]) -> str | None`; `MigrationDecision.reason` populated on a severity-only fail. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/unit/test_policy.py`: + +```python +class TestSeverityDrivenFailReason: + def _clean_records(self) -> list[EvalRecord]: + return [_record(example_id=f"ex{i}", delta=0.0) for i in range(30)] + + def test_a_critical_comparison_with_every_budget_held_names_itself(self) -> None: + decision = _decide( + self._clean_records(), + comparisons=[ + _comparison( + severity="critical", + evaluator_name="semantic.cosine", + prompt_id="replay", + ), + ], + ) + assert decision.verdict == "fail" + assert decision.reason is not None + assert decision.reason.startswith( + "semantic.cosine regressed at critical severity on prompt replay " + "(effect size -0.80, adjusted p 0.0100)", + ) + assert decision.reason.endswith( + "A critical or high regression from a blocking evaluator fails the run " + "on its own, whatever the budgets say.", + ) + + def test_every_blocking_comparison_is_named(self) -> None: + decision = _decide( + self._clean_records(), + comparisons=[ + _comparison(severity="critical", evaluator_name="a.one"), + _comparison(severity="high", evaluator_name="b.two"), + _comparison(severity="medium", evaluator_name="c.three"), + ], + ) + assert decision.reason is not None + assert "a.one regressed at critical severity" in decision.reason + assert "b.two regressed at high severity" in decision.reason + assert "c.three" not in decision.reason + + def test_a_budget_breach_keeps_the_budget_line(self) -> None: + decision = _decide( + [_record(example_id=f"ex{i}", delta=-0.5) for i in range(30)], + comparisons=[_comparison(severity="critical")], + ) + assert decision.verdict == "fail" + assert decision.reason is None + + def test_a_pass_has_no_reason(self) -> None: + decision = _decide(self._clean_records(), comparisons=[_comparison(severity="none")]) + assert decision.verdict == "pass" + assert decision.reason is None +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `uv run pytest tests/unit/test_policy.py -q -k SeverityDrivenFailReason` +Expected: `test_a_critical_comparison_with_every_budget_held_names_itself` and `test_every_blocking_comparison_is_named` FAIL (`assert None is not None`); the other two pass. + +- [ ] **Step 3: Implement** + +In `evaluate_migration_policy`, replace lines 537-541: + +```python + reason = ( + _inconclusive_reason(gating_budgets, n=overall.n_records) + if verdict == "inconclusive" + else None + ) +``` + +with: + +```python + if verdict == "inconclusive": + reason = _inconclusive_reason(gating_budgets, n=overall.n_records) + elif verdict == "fail": + # A budget breach is named by the budget table and the hero's + # "N of M" line; a fail on severity alone has neither, and read + # "7 of 7 budgets within policy" under FAIL. + reason = _severity_reason(overall_comparisons, gating_budgets) + else: + reason = None +``` + +Add after `_invariant_reason`: + +```python +def _severity_reason( + comparisons: list[ComparisonResult], + budgets: list[BudgetResult], +) -> str | None: + """Why a run failed on comparison severity alone, naming every blocking row. + + ``None`` when a budget's confirmed breach is (also) why it failed — the + budget rows already say so — or when no comparison reached a blocking + severity. The invariant and dropped-params reasons are applied after this + and keep their precedence. + """ + if any(not b.passed and b.conclusive for b in budgets): + return None + blocking = [c for c in comparisons if c.severity in _BLOCKING_SEVERITIES] + if not blocking: + return None + named = "; ".join( + f"{c.evaluator_name} regressed at {c.severity} severity on prompt {c.prompt_id} " + f"(effect size {c.effect_size:+.2f}, adjusted p {c.p_value_corrected:.4f})" + for c in blocking + ) + return ( + f"{named}. A critical or high regression from a blocking evaluator fails the run " + "on its own, whatever the budgets say." + ) +``` + +- [ ] **Step 4: Run the policy tests** + +Run: `uv run pytest tests/unit/test_policy.py -q` +Expected: all pass. If an existing test asserted `decision.reason is None` on a severity-only fail, it pinned the defect: change it to assert the new sentence. + +- [ ] **Step 5: Commit** + +```bash +git -C $WT add src/evalshift_cli/analysis/policy.py tests/unit/test_policy.py +cd $WT && uv run --with pre-commit git commit -m "fix(policy): give a severity-driven fail a reason + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 4: The templated narrative names the real cause + +**Files:** +- Modify: `src/evalshift_cli/insights/facts.py` (`Facts`, `build_facts`) +- Modify: `src/evalshift_cli/insights/templates.py` (`_RECOMMENDATIONS`, `fallback_insight`, `_verdict_summary`, new `_recommendation`) +- Test: `tests/unit/test_insight_facts.py` + +**Interfaces:** +- Consumes: `MigrationDecision.blocking_regressions[*].severity`, `BudgetResult.passed/conclusive`, `BUDGET_LABELS` (already imported in `facts.py`). +- Produces: `Facts.failed_budgets: list[str]` (display labels, default `[]`); `rendered["critical_regressions"]` = count of critical blocking regressions. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/unit/test_insight_facts.py` (`decision`, `budgets` and `BlockingRegression` are already importable there — add `budgets`/`decision` to the `tests.unit.insights_factories` import if missing): + +```python +def _critical_semantic() -> BlockingRegression: + return BlockingRegression( + prompt_id="replay", + evaluator_name="semantic.cosine", + slice_name="all", + severity="critical", + delta_avg_score=-0.05, + effect_size=-2.15, + ) + + +def test_a_severity_fail_never_claims_a_budget_breach(sample_run: dict[str, Any]) -> None: + sample_run["decision"] = decision(verdict="fail", blocking=[_critical_semantic()]) + recommendation = fallback_insight(build_facts(**sample_run)).recommendation + assert recommendation == ( + "Do not migrate under the configured policy — semantic.cosine regressed at " + "blocking severity although no budget was breached. Review its regressed " + "examples, then re-run." + ) + + +def test_a_budget_fail_names_the_budget(sample_run: dict[str, Any]) -> None: + breached = BudgetResult( + name="max_overall_regression_rate", + observed=0.5, + allowed=0.03, + passed=False, + conclusive=True, + ci_low=0.3, + ci_high=0.7, + ) + sample_run["decision"] = decision( + verdict="fail", + blocking=[_critical_semantic()], + budget_results=[breached, *budgets()[1:]], + ) + facts = build_facts(**sample_run) + assert facts.failed_budgets == ["Overall regression rate"] + assert fallback_insight(facts).recommendation.startswith( + "Do not migrate under the configured policy — the overall regression rate " + "budget was breached.", + ) + + +def test_a_fail_with_no_named_cause_points_at_the_reason(sample_run: dict[str, Any]) -> None: + sample_run["decision"] = decision(verdict="fail") + assert "see the verdict's reason" in fallback_insight(build_facts(**sample_run)).recommendation + + +def test_the_critical_count_reads_the_blocking_regressions(sample_run: dict[str, Any]) -> None: + sample_run["decision"] = decision(verdict="fail", blocking=[_critical_semantic()]) + facts = build_facts(**sample_run) + assert facts.rendered["critical_regressions"] == "1" + assert "1 blocking regression (1 critical)" in fallback_insight(facts).verdict_summary +``` + +- [ ] **Step 2: Run them to see them fail** + +Run: `uv run pytest tests/unit/test_insight_facts.py -q` +Expected: the four new tests FAIL (`AttributeError: 'Facts' object has no attribute 'failed_budgets'`, wrong recommendation text, `"0" == "1"`). + +- [ ] **Step 3: Implement the facts** + +In `src/evalshift_cli/insights/facts.py`: + +1. Add to `Facts` after `coverage_basis`: + +```python + #: Display names of the budgets whose breach the sample confirmed. Empty on + #: a fail that came from comparison severity alone — which is how the + #: templated advice tells the two causes apart instead of claiming a + #: breach the budget table contradicts. + failed_budgets: list[str] = field(default_factory=list) +``` + +2. In `build_facts`, after `coverage_basis = (...)`: + +```python + failed_budgets = [ + BUDGET_LABELS.get(b.name, b.name) + for b in decision.budget_results + if not b.passed and b.conclusive + ] +``` + +3. Replace the `"critical_regressions"` entry in `rendered`: + +```python + # Counted from the blocking comparisons themselves. The policy metric + # of the same name reads a per-record severity no evaluator writes, + # so it is always 0 and once sat beside a critical blocking item. + "critical_regressions": str( + sum(1 for r in decision.blocking_regressions if r.severity == "critical"), + ), +``` + +4. Pass `failed_budgets=failed_budgets,` in the `Facts(...)` return. + +- [ ] **Step 4: Implement the template** + +In `src/evalshift_cli/insights/templates.py`: + +1. Delete the `"fail": (...)` entry from `_RECOMMENDATIONS`. +2. In `fallback_insight`, replace the `recommendation=_clamp(_RECOMMENDATIONS.get(...))` argument with `recommendation=_clamp(_recommendation(facts)),`. +3. Add after `fallback_insight`: + +```python +def _recommendation(facts: Facts) -> str: + """The verdict's advice, naming what actually failed the run. + + Keyed on the verdict alone, a ``fail`` once said "at least one regression + budget was breached" over a run whose every budget held: the fail came + from a blocking evaluator's severity. The facts carry the cause, so the + sentence names it. + """ + if facts.verdict != "fail": + return _RECOMMENDATIONS.get(facts.verdict, _RECOMMENDATIONS["inconclusive"]) + if facts.failed_budgets: + labels = ", ".join(label[:1].lower() + label[1:] for label in facts.failed_budgets) + noun = "budget was" if len(facts.failed_budgets) == 1 else "budgets were" + return ( + f"Do not migrate under the configured policy — the {labels} {noun} breached. " + "Address the blocking evaluators, then re-run." + ) + if facts.blocking_evaluators: + whose = "its" if len(facts.blocking_evaluators) == 1 else "their" + return ( + "Do not migrate under the configured policy — " + f"{', '.join(facts.blocking_evaluators)} regressed at blocking severity " + f"although no budget was breached. Review {whose} regressed examples, then re-run." + ) + return ( + "Do not migrate under the configured policy — see the verdict's reason, " + "address it, then re-run." + ) +``` + +4. In `_verdict_summary`, replace the first `parts` sentence's middle with: + +```python + blocking = rendered["blocking_regressions"] + noun = "regression" if blocking == "1" else "regressions" + parts = [ + f"{rendered['verdict']} under the configured policy. " + f"{rendered['budgets_passed']} of {rendered['budgets_total']} budgets passed, " + f"with {blocking} blocking {noun} ({rendered['critical_regressions']} critical) " + f"across {rendered['n_examples']} examples.", + ] +``` + +- [ ] **Step 5: Run the insights tests** + +Run: `uv run pytest tests/unit/test_insight_facts.py tests/unit/test_insight_generator.py tests/unit/test_insights_stage.py -q` +Expected: all pass. A test that pinned the old `"… blocking and … critical regressions"` sentence or the old fail recommendation was asserting the defect — update it to the new wording. + +- [ ] **Step 6: Commit** + +```bash +git -C $WT add src/evalshift_cli/insights/facts.py src/evalshift_cli/insights/templates.py tests/unit/test_insight_facts.py +cd $WT && uv run --with pre-commit git commit -m "fix(insights): name the real cause of a fail and count critical honestly + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 5: Tool evaluators skip pairs that offered no tools + +**Files:** +- Modify: `src/evalshift_cli/cli/commands/evaluate.py` (`run_evaluate` lines 291-293; `_score_all` line 907; new `_offers_no_tools`, `_attempts`, `_tool_free_evaluators`, `_tool_free_warning`; comments at 1262-1264 and 1321-1324) +- Test: `tests/unit/test_evaluate_command.py` + +**Interfaces:** +- Consumes: `_applies`, `_is_tool_evaluator`, `_evaluator_kind`, `_PairedCalls`, `escape` (all existing in the module). +- Produces: `_offers_no_tools(pair: _PairedCalls) -> bool`, `_attempts(evaluator: Evaluator, pair: _PairedCalls) -> bool`, `_tool_free_evaluators(evaluators: list[Evaluator], pairs: list[_PairedCalls]) -> list[Evaluator]`, `_tool_free_warning(evaluator: Evaluator, suite_name: str | None) -> str`. + +- [ ] **Step 1: Let the run scaffold carry traces** + +In `tests/unit/test_evaluate_command.py`, give `_scaffold_run` a `with_traces: bool = False` keyword (after `suite_name`) and pass it into each `Call(...)`: + +```python + text=text, + trace=( + ToolTrace(calls=[ToolCall(tool_name="lookup", sequence_index=0)]) + if with_traces + else None + ), +``` + +Then change `test_partial_scoping_stays_silent` to call `_scaffold_run(tmp_path, with_traces=True)` — it asserts two attempted pairs, which after this task requires pairs that were offered tools. + +- [ ] **Step 2: Write the failing tests** + +Add `from dataclasses import dataclass, replace` (extend the existing `dataclasses` import). Append: + +```python +# --------------------------------------------------------------------------- +# Tool evaluators on pairs that offered no tools -- not applicable, not unmeasured +# --------------------------------------------------------------------------- + + +class TestToolEvaluatorsOnToolFreePairs: + def _invariants(self) -> TraceInvariantsEvaluator: + return TraceInvariantsEvaluator(TraceInvariantsEvaluatorConfig.model_validate(_INVARIANTS)) + + async def _cells(self, evaluator: Any, pairs: list[_PairedCalls]) -> list[_ScoredCell]: + return await _score_all(Console(), [evaluator], pairs, "r1", {}, concurrency=2, quiet=True) + + async def test_a_pair_that_offered_no_tools_is_never_attempted(self) -> None: + cells = await self._cells( + self._invariants(), + [_pair_for("ex1"), _tool_pair("greet", "ex2", "lookup")], + ) + assert [(c.example_id, c.record is not None) for c in cells] == [("ex2", True)] + coverage = _coverage_for(cells) + assert (coverage[0].attempted, coverage[0].unmeasured) == (1, []) + + async def test_a_wholly_tool_free_run_books_no_coverage(self) -> None: + cells = await self._cells(self._invariants(), [_pair_for("ex1"), _pair_for("ex2")]) + assert cells == [] + assert _coverage_for(cells) == [] + + async def test_text_evaluators_still_score_tool_free_pairs(self) -> None: + cells = await self._cells(_SometimesUnmeasuredEvaluator(), [_pair_for("ex1")]) + assert [c.record is not None for c in cells] == [True] + + async def test_a_one_sided_trace_is_still_unmeasured(self) -> None: + pair = _tool_pair("greet", "ex1", "lookup") + one_sided = replace(pair, target=pair.target.model_copy(update={"trace": None})) + cells = await self._cells(self._invariants(), [one_sided]) + coverage = _coverage_for(cells) + assert [p.example_id for p in coverage[0].unmeasured] == ["ex1"] + + async def test_an_errored_tool_free_pair_keeps_its_error_row(self) -> None: + pair = _pair_for("ex1") + errored = replace(pair, source=pair.source.model_copy(update={"error": "boom"})) + cells = await self._cells(self._invariants(), [errored]) + assert len(cells) == 1 + assert cells[0].record is not None + assert (cells[0].record.error or "").startswith("upstream call failed") +``` + +And inside `class TestTraceInvariantsThatCheckedNothing`, add: + +```python + def test_a_tool_free_suite_names_the_real_cause(self, tmp_path: Path) -> None: + _write_invariants_config(tmp_path, applies_to=["*"]) + _scaffold_run(tmp_path, suite_name="triage") + + output = " ".join(_evaluate_output(tmp_path).split()) + + assert ( + "trace_invariants evaluator 'payments_contract' checked nothing: no example in " + "suite 'triage' offered the models any tools, so there were no tool calls to " + "check. It did not gate this run." + ) in output + assert "applies_to" not in output + state = read_state(tmp_path / ".evalshift" / "runs" / "r_20260601_aaaaaa") + assert state.evaluator_coverage == [] + + def test_an_applies_to_miss_keeps_its_own_wording_on_a_tool_free_run( + self, tmp_path: Path + ) -> None: + _write_invariants_config(tmp_path, applies_to=["checkout-*"]) + _scaffold_run(tmp_path) + + output = _evaluate_output(tmp_path) + + assert "['checkout-*']" in output + assert "offered the models any tools" not in output +``` + +- [ ] **Step 3: Run them to see them fail** + +Run: `uv run pytest tests/unit/test_evaluate_command.py -q -k "ToolFreePairs or ThatCheckedNothing"` +Expected: `test_a_pair_that_offered_no_tools_is_never_attempted` and `test_a_wholly_tool_free_run_books_no_coverage` FAIL (a recordless cell exists for the tool-free pair); `test_a_tool_free_suite_names_the_real_cause` FAILS (old `applies_to` wording, coverage non-empty). The rest pass already. + +- [ ] **Step 4: Implement** + +In `src/evalshift_cli/cli/commands/evaluate.py`, add after `_applies` (line 711): + +```python +def _offers_no_tools(pair: _PairedCalls) -> bool: + """Whether neither side of ``pair`` was offered a tool. + + The orchestrator dispatches with tools exactly when the example's resolved + toolset is non-empty, and that path always stores a :class:`ToolTrace` on + the call. Two error-free calls with no trace between them are therefore a + plain text exchange, and a tool evaluator has nothing it could check — the + same as an ``applies_to`` miss. An errored call proves nothing about the + toolset and keeps the error path; one trace without the other stays + unmeasured (see :func:`_score_one_tool`). + """ + return ( + pair.source.error is None + and pair.target.error is None + and pair.source.trace is None + and pair.target.trace is None + ) + + +def _attempts(evaluator: Evaluator, pair: _PairedCalls) -> bool: + """Whether ``evaluator`` is asked to score ``pair`` at all. + + ``False`` means no cell: no row, and no coverage attempt that would later + read as "unmeasured" and demote the verdict. Booking a tool-free suite as + unmeasured is what turned a clean ``pass`` into ``conditional_pass`` for a + top-level ``trace_invariants`` entry written for a different suite. + """ + if not _applies(evaluator, pair.prompt_id): + return False + return not (_is_tool_evaluator(evaluator) and _offers_no_tools(pair)) + + +def _tool_free_evaluators( + evaluators: list[Evaluator], + pairs: list[_PairedCalls], +) -> list[Evaluator]: + """Tool evaluators whose every in-scope pair offered no tools. + + Computed from the pairs rather than from coverage: an evaluator with no + coverage entry may also be an ``applies_to`` miss, which has its own + warning and its own fix. + + Args: + evaluators: Every evaluator built for the run. + pairs: Every (source, target) pair in the run. + + Returns: + The ``score_pair`` evaluators that checked nothing because no pair + they were scoped to was offered a tool, in configured order. + """ + out: list[Evaluator] = [] + for evaluator in evaluators: + if not _is_tool_evaluator(evaluator): + continue + scoped = [pair for pair in pairs if _applies(evaluator, pair.prompt_id)] + if scoped and all(_offers_no_tools(pair) for pair in scoped): + out.append(evaluator) + return out + + +def _tool_free_warning(evaluator: Evaluator, suite_name: str | None) -> str: + """The console line for a tool evaluator that a tool-free run left nothing to check.""" + where = f"suite {suite_name!r}" if suite_name else "this run" + kind = _evaluator_kind(evaluator) or "tool" + return ( + f"[yellow]⚠[/yellow] {escape(kind)} evaluator {escape(repr(evaluator.name))} " + f"checked nothing: no example in {escape(where)} offered the models any tools, " + "so there were no tool calls to check. It did not gate this run." + ) +``` + +In `_score_all`, replace the work-list filter `if _applies(evaluator, pair.prompt_id)` with `if _attempts(evaluator, pair)`. + +In `run_evaluate`, replace lines 291-293 with: + +```python + if not quiet: + tool_free = _tool_free_evaluators(evaluators, pairs) + for evaluator in tool_free: + console.print(_tool_free_warning(evaluator, state.suite_name)) + for evaluator in _unchecked_trace_invariants(evaluators, coverage): + if evaluator not in tool_free: + console.print(_unchecked_trace_invariants_warning(evaluator)) +``` + +Update the two stale comments: +- `_score_one` (lines 1262-1264): `# v0.2 — tool evaluators consume the (source, target) ToolTrace pair. A pair offered no tools never reaches here (see _attempts).` +- `_score_one_tool` (lines 1321-1324): `# Only one side carries a trace — a pair offered no tools at all never gets here (see _attempts). There is nothing to pair, so nothing to report; coverage books it unmeasured.` + +- [ ] **Step 5: Run the evaluate tests** + +Run: `uv run pytest tests/unit/test_evaluate_command.py -q` +Expected: all pass. Any other test that fed a traceless pair to a tool evaluator through `_score_all`/`run_evaluate` and asserted an unmeasured coverage entry was pinning the old behaviour: give it `with_traces=True` / `_tool_pair` if it is about something else, or update it to assert no cell if it is about tool-free pairs. + +- [ ] **Step 6: Full gate, then commit** + +Run: `uv run ruff check . && uv run ruff format --check . && uv run mypy --strict src/evalshift_cli && uv run pytest -q` +Expected: green. + +```bash +git -C $WT add src/evalshift_cli/cli/commands/evaluate.py tests/unit/test_evaluate_command.py +cd $WT && uv run --with pre-commit git commit -m "fix(evaluate): tool evaluators do not apply to pairs offered no tools + +A top-level trace_invariants entry is inherited by tool-free suites, where +every pair was booked unmeasured and the blind gate demoted a clean run to +conditional_pass with a note about 'tool calls and no prose'. Such pairs now +get no cell, like an applies_to miss, and evaluate names the real cause. + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 6: Docs and changelog + +**Files:** +- Modify: `docs/methodology.md`, `docs/configuration.md`, `docs/agents.md`, `DOCS.md`, `llms-full.txt`, `CHANGELOG.md` + +- [ ] **Step 1: `docs/methodology.md`** + +After the severity table (before `## How policy metrics find their rows`) insert: + +```markdown +### Drift inside `min_similarity` is tested as no change + +`semantic` pins the source at 1.0, so every target that words anything +differently scores below it: a negative delta is the evaluator's normal output, +not evidence of a regression. Its rows therefore enter the paired test +**tolerance-adjusted** — a row within `min_similarity` (no `SEMANTIC_REGRESSION` +flag) contributes a delta of `0`, a flagged row its full drop. That is the line +the policy budgets already draw, so the two layers cannot disagree: nine rows at +0.90–0.98 once passed every budget and still failed the run as a `critical` +regression, because nine small, consistent drops have almost no variance. +`delta_avg_score` stays the raw mean — it describes the drift; severity judges +it — and the comparison carries the note `k of n rows within the evaluator's own +threshold (min_similarity) — tested as no change`. A comparison where only a few +rows are flagged may not reach significance on its own; the rate budgets +(`max_overall_regression_rate`, `min_equivalence_rate`) decide it. + +### Which severities fail the run + +A `critical` or `high` comparison from a blocking evaluator fails the run on its +own, even when every budget holds; `medium` and `low` downgrade a `pass` to +`conditional_pass`. When severity alone failed the run, the decision's `reason` +names each such comparison, so the report's verdict line explains the FAIL. +``` + +And in the materiality paragraph (line ~153-155) replace `The same reasoning is why semantic drift is counted against the evaluator's own `min_similarity` rather than against `delta < 0`.` with `The same reasoning is why semantic drift is counted against the evaluator's own `min_similarity` rather than against `delta < 0` — by the budgets and by the paired test alike (see [Drift inside `min_similarity` is tested as no change](#drift-inside-min_similarity-is-tested-as-no-change)).` + +- [ ] **Step 2: `docs/configuration.md`** + +Replace the paragraph at lines ~241-245 (`Semantic-evaluator drift counts toward …`) with: + +```markdown +Semantic-evaluator drift counts toward `max_overall_regression_rate` and +`min_equivalence_rate` — and toward the paired test that sets each comparison's +severity — only when it breaches +[`evaluators.semantic.min_similarity`](#evaluatorssemantic). A near-identical +output (cosine ~0.98) that stays within `min_similarity` is treated as +*equivalent*, not a regression, by the budgets and the test alike — so the +policy gate, the severity and the report agree. +``` + +At lines ~955-957 replace `or hands the tool-free ones a tool evaluator with an empty denominator, which reads as an *inconclusive* gate rather than as "not applicable here". So a suite` with `or hands the tool-free ones tool evaluators with nothing to check — they skip every pair that was offered no tools (no row, no coverage entry, and `evaluate` warns that they checked nothing). So a suite`. + +At line ~1138 replace `inherits the top level, and no tool evaluator scores an empty denominator.` with `inherits the top level; a tool evaluator it inherits skips its tool-free pairs.` + +- [ ] **Step 3: `docs/agents.md`** + +Lines ~119-121: replace `a tool evaluator pointed at a tool-free suite scores an empty denominator, which the policy reads as an *inconclusive* gate rather than as "not applicable here".` with `a tool evaluator pointed at a tool-free suite has nothing to check: it skips every pair that was offered no tools — no row, no coverage entry, no effect on the verdict — and `evaluate` warns that it checked nothing.` + +- [ ] **Step 4: `DOCS.md`** + +1. Line 381: same replacement as configuration.md ~955 (`or hands the tool-free ones a tool evaluator with an empty denominator, which the policy reads as an *inconclusive* gate rather than as "not applicable here".` → `or hands the tool-free ones tool evaluators with nothing to check (they skip every pair that was offered no tools, and `evaluate` warns that they checked nothing).`). +2. Line 514 (semantic section): replace `Within it, drift counts as *equivalent* for policy purposes.` with `Within it, drift counts as *equivalent* for policy purposes and as *no change* in the paired test that sets severity — see [methodology](docs/methodology.md#drift-inside-min_similarity-is-tested-as-no-change).` +3. Line 704 (sample guards), two insertions in the same paragraph: + - after `Zero-variance deltas → skipped, severity `none`.` insert: ` A semantic row within `min_similarity` enters the test as a delta of 0 (noted `k of n rows within the evaluator's own threshold (min_similarity) — tested as no change`); `delta_avg_score` stays the raw mean.` + - at the end of the paragraph (after `…the **Not enough data** the rest of `insufficient` gets.`) append: ` A tool evaluator is never handed a pair that was offered no tools at all — that pair is "not applicable", like an `applies_to` miss, not "nothing measured".` +4. Line 759: append ` When severity alone fails the run, `reason` names each blocking comparison.` +5. Line 762: replace with `- Semantic drift that stays above `min_similarity` counts as equivalent, not regression — in the budgets and in the paired test.` + +- [ ] **Step 5: `llms-full.txt`** + +1. Lines 409-410: `(suite inherits the top level; a tool evaluator on a tool-free suite would score an empty denominator = inconclusive gate)` → `(suite inherits the top level; an inherited tool evaluator skips pairs offered no tools: no cell, no coverage, evaluate warns "checked nothing")`. +2. Line 618: `min_similarity: float = 0.9 # below -> SEMANTIC_REGRESSION flag` → `min_similarity: float = 0.9 # below -> SEMANTIC_REGRESSION flag; above -> delta tested as 0`. +3. Line 749: `Replayed pairs with no tool trace get no row.` → `Replayed pairs with no tool trace get no row; pairs offered no tools at all get no cell (not attempted).` +4. After line 905 (`… Never "none" - unmeasured is not equivalent.`) add: ` Pairs offered no tools are not handed to tool evaluators at all (no cell, no coverage).` and a new item: `0b. Tolerance: a semantic row without SEMANTIC_REGRESSION (within min_similarity) enters the test as delta 0 (SlicedScore.within_tolerance); note "k of n rows within the evaluator's own threshold (min_similarity) - tested as no change". delta_avg_score and aggregates stay raw.` +5. Line 1126: `- Semantic drift above min_similarity counts as equivalent, not regression.` → `- Semantic drift above min_similarity counts as equivalent, not regression -- in budgets and in the paired test (severity).`; and line 1108, `- fail: any conclusive budget failure OR any blocking critical/high comparison.` → `- fail: any conclusive budget failure OR any blocking critical/high comparison (when severity alone fails the run, reason names each such comparison).` +6. Lines 1478-1479: `a tool-free suite that # inherits a tool evaluator scores an empty denominator = an inconclusive gate.` → `a tool-free suite that # inherits a tool evaluator gives it nothing to check (it skips those pairs and evaluate warns).` — keep the comment-line breaks. + +- [ ] **Step 6: `CHANGELOG.md`** + +Under `## [Unreleased]` add: + +```markdown +### Fixed + +- A `semantic` evaluator set to `blocking: true` no longer fails runs whose every row + stays within `min_similarity`. The source is pinned at 1.0, so any rewording is a + small negative delta, and the paired significance test ran on those raw deltas: + nine rows at 0.90–0.98 tested as a `critical` regression and failed a run whose every + budget passed. Rows within `min_similarity` now enter the test as no change — the + rule the budgets already used — and the comparison notes how many. `delta_avg_score` + and the aggregates still show the raw drift. Re-analysing an older run can lower its + semantic severities. +- A fail caused by comparison severity alone now carries a `reason` naming the + comparison, so the report no longer shows "7 of 7 budgets within policy" under FAIL. + The templated recommendation names the actual cause instead of always claiming a + budget breach, and the summary's critical count reads the blocking comparisons. +- Tool evaluators (including a top-level `trace_invariants` inherited by every suite) no + longer attempt pairs that were offered no tools. On a tool-free suite they used to + book every pair as unmeasured, demote a clean `pass` to `conditional_pass`, and blame + "tool calls and no prose". They now skip those pairs, and `evaluate` warns that the + evaluator checked nothing because no example offered tools. +``` + +- [ ] **Step 7: Check the doc tripwires, then commit** + +Run: `uv run pytest -q -k "docs or llms or currency" && uv run pre-commit run --all-files` +Expected: green (the repo's docs-currency tests — added 2026-09-21 — pass). + +```bash +git -C $WT add docs/methodology.md docs/configuration.md docs/agents.md DOCS.md llms-full.txt CHANGELOG.md +cd $WT && uv run --with pre-commit git commit -m "docs: semantic tolerance in the paired test; tool-free suites skip tool evaluators + +Co-Authored-By: Claude Opus 5.5 (1M context) " +``` + +--- + +### Task 7: Verify on the real run, then PR + +**Files:** none changed (a backup in the scratchpad). + +- [ ] **Step 1: Back up the run, then re-score it with the branch build** + +```bash +cp -r /home/lukas/repos/testApp/.evalshift/runs/r_20261011_triage_e14cda "$SCRATCH/r_20261011_triage_e14cda.bak" +cd /home/lukas/repos/testApp +uv run --project $WT evalshift evaluate r_20261011_triage_e14cda +uv run --project $WT evalshift analyze r_20261011_triage_e14cda +uv run --project $WT evalshift report r_20261011_triage_e14cda +``` +`$SCRATCH` is the session scratchpad. The global `evalshift` on PATH is the PyPI 1.4.1 wheel, so it **must** be `uv run --project $WT`. `evaluate` re-embeds through the cache; if it refuses for a missing judge/embedder key, ask the user to run the three commands with `!` in their shell. + +Expected: +- `evaluate` prints `⚠ trace_invariants evaluator 'agent_contract' checked nothing: no example in suite 'triage' offered the models any tools, …`. +- `analysis.json`: the `semantic.cosine` comparison has `severity: "none"`, `delta_avg_score ≈ -0.0524`, a note `9 of 9 rows within the evaluator's own threshold (min_similarity) — tested as no change`; no `agent_contract` comparison. +- `migration_decision.json`: `verdict: "pass"`, `blocking_regressions: []`, `recommendations[0] == "Safe to migrate under the configured policy."`. +- `report.html` hero: **PASS**, "7 of 7 budgets within policy". + +- [ ] **Step 2: Full CI mirror** + +Run: `cd $WT && make ci` +Expected: ruff, format, mypy, pytest all green; coverage ≥ 90. + +- [ ] **Step 3: Code review** + +Use superpowers:requesting-code-review on the branch diff `origin/main...fix/semantic-gate-tool-free-suites`; apply findings with superpowers:receiving-code-review. + +- [ ] **Step 4: Push and open the PR** (only after the user approves pushing) + +```bash +cd $WT && uv run --with pre-commit git push -u origin fix/semantic-gate-tool-free-suites +gh -R evalshift/evalshift-cli pr create --base main --head fix/semantic-gate-tool-free-suites \ + --title "fix: semantic drift within min_similarity no longer fails runs; tool evaluators skip tool-free suites" \ + --body-file "$SCRATCH/pr-body.md" +``` +The PR body names the triage run, the before/after verdict (FAIL → PASS), the accepted consequence (a few flagged rows among many forgiven ones leave the decision to the rate budgets), the follow-ups from the spec, and ends with `🤖 Generated with [Claude Code](https://claude.com/claude-code)`. Releasing it as 1.4.2 is a separate `chore(release): 1.4.2` PR after merge, per `CONTRIBUTING.md`. diff --git a/docs/superpowers/specs/2026-10-11-semantic-gate-and-tool-free-suites-design.md b/docs/superpowers/specs/2026-10-11-semantic-gate-and-tool-free-suites-design.md new file mode 100644 index 0000000..3326202 --- /dev/null +++ b/docs/superpowers/specs/2026-10-11-semantic-gate-and-tool-free-suites-design.md @@ -0,0 +1,140 @@ +# Semantic drift inside `min_similarity` must not fail the run; tool evaluators skip tool-free suites + +**Date:** 2026-10-11 · **Status:** approved approaches A + D, for maintainer review · **Repos:** evalshift-cli (behaviour + docs); evalshift-server, evalshift-client, evalshift-action, evalshift-sdk (nothing) + +## Problem + +Run `r_20261011_triage_e14cda` (testApp, suite `triage`, `anthropic/claude-sonnet-5` → +`anthropic/claude-sonnet-5-5`, CLI 1.4.1) is **FAIL** although nothing in it regressed: + +- every budget passes — 100% equivalent, 0% regressed, 7 of 7 budgets within policy; +- every `semantic.cosine` row is above `min_similarity: 0.9` (lowest 0.904), so none carries + `SEMANTIC_REGRESSION`; +- yet `blocking_regressions` lists `semantic.cosine` at severity `critical` (paired t, adjusted + p 0.0004, d −2.15), and `_verdict_for` (`analysis/policy.py:1292`) fails the run on any + `critical`/`high` gating comparison. + +**Root cause.** `semantic` pins `source_score = 1.0` (`evaluators/semantic.py:118,156`), so every +target that words anything differently has a negative delta. The paired test +(`analysis/statistics.py::_one_comparison`) runs on those raw deltas; nine small, consistent +drops (−0.02…−0.10) have tiny variance, so it calls them `critical` for *any* two models whose +text differs. `min_similarity` is applied only by the policy's per-row predicates +(`_is_regression` / `_is_equivalent`, `policy.py:675–690`) — the statistics layer never sees it. +`docs/methodology.md:155` already claims semantic drift "is counted against the evaluator's own +`min_similarity` rather than against `delta < 0`"; only half the pipeline does that. Not a +regression from a recent commit: before 1.4.1's format normalization every row sat at ~0.75, +so the rate budgets failed the run first and hid it. + +**The report then contradicts itself:** + +1. The hero prints "7 of 7 budgets within policy" under **FAIL** — a severity-driven fail leaves + `decision.reason = None` (`policy.py:537–541`). +2. The templated recommendation says "at least one regression budget was breached" + (`insights/templates.py:31–34`) — it is keyed on the verdict alone. +3. The summary says "1 blocking and 0 critical regressions" while the one blocking item *is* + critical — `critical_regressions` counts a `metadata["severity"]` key no evaluator writes + (`policy.py:819`). + +**Second, independent defect.** The top-level `trace_invariants` entry `agent_contract` +(written for the planner suite's tools) is inherited by every suite without its own family +(`config/models.py:869–905`; the docs recommend top level, `DOCS.md:298`). On `triage` no example +offers tools, so no `Call.trace` exists (`runner/orchestrator.py:1219`), `_score_one_tool` +returns `[]` (`cli/commands/evaluate.py:1321`), and every pair is booked *unmeasured*. A blocking +evaluator that measured nothing demotes `pass` → `conditional_pass` (`policy.py:528–532`) and +emits a note written for semantic's both-empty case ("both models answered with tool calls and no +prose") — false here. So fixing the semantic gate alone still leaves this run at +`conditional_pass`. `DOCS.md:381` documents the current behaviour as "an empty denominator, which +the policy reads as an *inconclusive* gate rather than as 'not applicable here'". + +## Decision + +### A. The paired test sees the same "regression" the budgets do + +- One shared definition, in a new `analysis/tolerance.py`: `within_tolerance(record)` is true for + a `semantic` record that does **not** carry `SEMANTIC_REGRESSION`. The policy's + `_is_regression` / `_is_equivalent` and the slicer both read it. +- `SlicedScore` gains `within_tolerance: bool = False`, set by `build_slices`. +- `_one_comparison` tests the **tolerance-adjusted** deltas — `0.0` for a within-tolerance row, + the raw delta otherwise — for the zero-variance check, the normality screen, the test, Cohen's + d and its CI. The paired t-test becomes `stats.ttest_1samp(tested, 0.0)`, which equals + `ttest_rel(target, source)` exactly on raw deltas (verified: identical statistic and p), so no + other evaluator's numbers move. +- `delta_avg_score` stays the **raw** mean delta: it is descriptive, the report headlines it, and + "drifted −5% on average, within tolerance" is the honest reading. Its sign always matches the + tested mean for semantic (both ≤ 0), so severity classification is unaffected. +- A comparison where any row was forgiven gets a note: + `" of rows within the evaluator's own threshold (min_similarity) — tested as no change"`. +- Slice aggregates (`aggregates()`, `analysis.json.aggregates`) stay raw. Per-example rows, + `scores.jsonl` and the bundle are unchanged. + +Effect: this run's nine rows test as nine zeros → zero variance → severity `none` → no blocking +regression. A run whose every row sits below `min_similarity` (the pre-1.4.1 run) tests exactly +as today → `critical`. **Accepted consequence:** a mix where only a few rows are flagged (3 of 9 +at 0.75) no longer reaches significance on its own; the rate budgets decide it (here 33% > 30% → +breach, `inconclusive` at n=9). That is the policy's own small-sample rule, and it is the +behaviour `methodology.md` already describes. + +### Report honesty (rides with A) + +- **`reason` on a severity-driven fail.** When the overall verdict is `fail`, no budget was + conclusively breached and at least one overall gating comparison is `critical`/`high`, set + `decision.reason` to: + `" regressed at severity on prompt (effect size , adjusted p )[; …]. A critical or high regression from a blocking evaluator fails the run on its own, whatever the budgets say."` + The hero renders `reason` already (`report.html.j2:142`). Invariant and dropped-params reasons + keep their current precedence (they run after and override/append as today). +- **Cause-keyed fallback recommendation.** `Facts` gains `failed_budgets` (display labels of + conclusively breached overall budgets). For `fail`: budgets breached → name them; else blocking + evaluators → name them as regressed at blocking severity; else a generic line pointing at the + verdict's reason. Never claim a budget breach that did not happen. +- **Honest critical count in the narrative.** `rendered["critical_regressions"]` counts + `decision.blocking_regressions` with severity `critical`; the template sentence becomes + `"with blocking regressions ( critical) across examples"`. + +### D. Tool evaluators do not apply to a pair that offered no tools + +- A pair "offered no tools" when neither call errored and neither carries a `trace`. The + orchestrator dispatches with tools iff the example's resolved toolset is non-empty and always + stores a trace on that path, so for error-free calls this is exact. Errored pairs keep today's + path (error rows). +- `_score_all` builds no work item for a `score_pair` evaluator on such a pair — the same + treatment `_applies` already gives an `applies_to` miss: no cell, no row, no coverage entry, no + `n=0` comparison, no demotion, no misleading note. One-sided `trace is None` stays unmeasured + exactly as today. +- `evaluate` warns once per tool evaluator whose every in-scope pair offered no tools: + `"⚠ evaluator '' checked nothing: no example in suite '' offered the models any tools, so there were no tool calls to check. It did not gate this run."` + The existing `applies_to`-matched-nothing warning keeps its wording and only fires when the + cause really is `applies_to`. + +With A + D this run is **pass**: all budgets pass, semantic `none`, judge advisory, +`agent_contract` not attempted. + +## Not in scope (recorded follow-ups) + +- `max_critical_regressions` is inert: `_metrics` counts `metadata["severity"]`, which nothing + writes. Fixing it changes what the hosted gate reads (`evalshift-server + app/policy/service.py:366`), so it gets its own cross-repo change. +- `severity_floor` metadata written by `tool_selection` / `tool_trace_structure` is read by + nothing. +- `tool_arguments` against `source` has the same pinned-source shape (`tool_argument_drift_floor` + applies only to the rate). `within_tolerance` is the hook to extend later. +- With `samples_per_example > 1`, a reduced semantic row keeps the *lead* sample's + `failure_categories` (`evaluate.py::_reduce_one_group`) while its scores are means, so its + `SEMANTIC_REGRESSION` flag can disagree with its mean score. Pre-existing, and it already + affects the budgets the same way; A makes the test inherit it rather than fix it. +- The "both models answered with tool calls and no prose" wording in + `_recommendations.nothing_measured` is still a guess for other unmeasured causes; after D it no + longer fires for tool-free suites. + +## Docs that change + +`docs/methodology.md` (severity section: tolerance-adjusted test; which severities fail the run), +`docs/configuration.md` (semantic/policy paragraph ~l.241; per-suite evaluators ~l.955), +`docs/agents.md` (~l.118), `DOCS.md` (l.381, 514, 704, 759–762), `llms-full.txt` (l.409, 618, +749, 903–905, 1116–1127, 1479), `CHANGELOG.md` `## [Unreleased]`. + +## Compatibility + +No config, schema, bundle or `state.json` change. `analysis.json` severities for semantic +comparisons drop where rows sit within tolerance; old runs re-analysed with the new CLI get the +new answer (intended). A tool-free suite's `state.json` no longer lists tool evaluators in +`evaluator_coverage`. The `--gate critical,high` CLI gate follows the new severities. diff --git a/llms-full.txt b/llms-full.txt index b5c87a9..d8515ab 100644 --- a/llms-full.txt +++ b/llms-full.txt @@ -405,8 +405,9 @@ evalshift capture sync [--suite S] [--input-var NAME=input] [--tag T]... [--stri Advisory only: never edits the workflow, exit code unchanged. `init` (without --ci) warns the same way next to an existing workflow; `validate` prints it after its success line. Each suite's entry is DERIVED from its own rows, evaluators included: - no row offered a toolset -> no evaluators: block (suite inherits the top level; a tool - evaluator on a tool-free suite would score an empty denominator = inconclusive gate) + no row offered a toolset -> no evaluators: block (suite inherits the top level; an + inherited tool evaluator skips pairs offered no tools: no cell, no coverage, evaluate + and compare warn "checked nothing") any row offered a toolset -> tool_selection: [{name: routing, conformance: expected, divergence: set}] any row recorded call arguments -> tool_arguments: [{name: routing_args, against: expected}] @@ -615,7 +616,7 @@ evaluators: # every evaluator config also takes blocking applies_to: ["*"] semantic: # single block, not a list embedding_model: str = text-embedding-3-small # bare text-embedding-* resolves to openai - min_similarity: float = 0.9 # below -> SEMANTIC_REGRESSION flag + min_similarity: float = 0.9 # below -> SEMANTIC_REGRESSION flag; above -> delta tested as 0 # init writes it ACTIVE for every provider (anthropic/deepseek borrow an OpenAI/Gemini # embedder -- see init). No key for embedding_model -> skipped or refused before any # call; see "Missing judge/embedding keys" under Behavior rules. @@ -746,7 +747,8 @@ evaluators: # every evaluator config also takes blocking # required / min_calls are NOT vacuous (fail every in-scope trace lacking the call); on a # single-shot replay they see only the first response -> scope with applies_to, or use # --rounds all / imported traces. order/arguments/max-only call_count are vacuous when - # the tool is never called. Replayed pairs with no tool trace get no row. + # the tool is never called. Replayed pairs with no tool trace get no row; pairs offered + # no tools at all get no cell (not attempted). # Score per side = (rules - rules broken) / rules; kind slug trace_invariants; metadata # rules_checked, source_violations, target_violations (each [{rule_id, rule_type, tool, # round_index, detail}], with sample ordinal when samples_per_example > 1), owner; @@ -909,6 +911,11 @@ Per (prompt_id, evaluator_name, slice_name) over paired per-example deltas (targ that note is the only surviving carrier of the config flag for the policy layer. report.html renders such a row as "Nothing measured" (not the "Not enough data" headline the rest of `insufficient` gets) - the sample was absent, not small. + Pairs offered no tools are not handed to tool evaluators at all (no cell, no coverage). +0b. Tolerance: a semantic row without SEMANTIC_REGRESSION (within min_similarity) enters the + test as delta 0 (SlicedScore.within_tolerance); note + "k of n rows within the evaluator's own threshold (min_similarity) - tested as no change". + delta_avg_score and aggregates stay raw. 1. n<5 -> skipped, severity "insufficient". 5<=n<20 -> tested, flagged uncertain. Zero variance (std < 1e-9) -> skipped, severity "none". 2. Shapiro-Wilk on deltas at alpha=.05: normal -> paired t-test; else Wilcoxon signed-rank. @@ -1105,7 +1112,8 @@ Verdicts: pass | conditional_pass | fail | inconclusive. Written to migration_de nothing -> positive denominator, conclusive: false. The hosted gate computes its OWN Wilson interval from these denominators, over the same three proportion budgets and with the same z, so a local and a hosted verdict agree on whether a breach was confirmed. -- fail: any conclusive budget failure OR any blocking critical/high comparison. Slice budgets +- fail: any conclusive budget failure OR any blocking critical/high comparison (when severity + alone fails the run, reason names each such comparison). Slice budgets (migration_policy.slices) count here on the same terms as the top-level ones: conclusive breach -> fail, unconfirmed breach -> inconclusive, Wilson-judged over that slice's own denominator. recommendations names the blocking row ("The 'security' slice breached its overall @@ -1123,8 +1131,9 @@ Verdicts: pass | conditional_pass | fail | inconclusive. Written to migration_de inconclusive: all comparisons insufficient, or unconfirmed rate-budget breach (never when max_invariant_violations is breached -- that is fail). pass: otherwise. -- Semantic drift above min_similarity counts as equivalent, not regression. min_equivalence_rate - floors the NON-REGRESSION rate: equivalent and improved both count toward it. +- Semantic drift above min_similarity counts as equivalent, not regression -- in budgets and + in the paired test (severity). min_equivalence_rate floors the NON-REGRESSION rate: + equivalent and improved both count toward it. - cost/latency increase = max(0, (target_avg - source_avg)/source_avg) over non-errored calls. CI gates: analyze/compare --gate critical,high (exit 1 on matching severities; allowed: critical,high,medium,low); --policy-gate (exit 1 on fail OR conditional_pass, or when no @@ -1476,8 +1485,9 @@ suites: {} # init writes the managed region LAST, after # evalshift.yaml — agent prompt (tool-calling additions). Tool evaluators live in the # PER-SUITE block `capture sync` generates, not at the top level: a tool-free suite that -# inherits a tool evaluator scores an empty denominator = an inconclusive gate. Hand-written -# trace_invariants are the exception (top level is fine; sync keeps them on managed entries). +# inherits a tool evaluator gives it nothing to check (it skips those pairs and evaluate +# warns). Hand-written trace_invariants are the exception (top level is fine; sync keeps +# them on managed entries). prompts: - id: customer_routing detection: python_string diff --git a/src/evalshift_cli/analysis/policy.py b/src/evalshift_cli/analysis/policy.py index b67e9e0..7370967 100644 --- a/src/evalshift_cli/analysis/policy.py +++ b/src/evalshift_cli/analysis/policy.py @@ -19,12 +19,12 @@ UNMEASURED_NOTE_PREFIX, ComparisonResult, ) +from evalshift_cli.analysis.tolerance import is_semantic, is_semantic_regression, within_tolerance from evalshift_cli.config.models import MigrationPolicy, SliceMigrationPolicy from evalshift_cli.evaluators.base import EvalRecord from evalshift_cli.evaluators.failures import ( BROKEN_HARNESS_CAUSES, INVARIANT_VIOLATION, - SEMANTIC_REGRESSION, TOOL_GROUND_TRUTH_MISS, ) from evalshift_cli.evaluators.keys import LLM_JUDGE_KIND, skip_recommendation @@ -35,7 +35,6 @@ _REGRESSION_SEVERITIES = frozenset({"critical", "high", "medium", "low"}) _BLOCKING_SEVERITIES = frozenset({"critical", "high"}) _TOOL_ARGUMENT_KIND = "tool_arguments" -_SEMANTIC_KIND = "semantic" # The two axes ``tool_selection`` scores. They are separate measurements # against separate baselines — conformance grades each side absolutely # against the recorded ground truth, divergence grades the target against a @@ -58,7 +57,6 @@ # under a name prefix, so a row that predates them cannot belong to either. _LEGACY_KIND_PREFIXES = { _TOOL_ARGUMENT_KIND: "tool_arguments.", - _SEMANTIC_KIND: "semantic.", } # Rate *ceilings*, and the noun naming the rows each one is counted over. A @@ -534,11 +532,15 @@ def evaluate_migration_policy( s.verdict in {"fail", "conditional_pass"} for s in slice_decisions.values() ): verdict = "conditional_pass" - reason = ( - _inconclusive_reason(gating_budgets, n=overall.n_records) - if verdict == "inconclusive" - else None - ) + if verdict == "inconclusive": + reason = _inconclusive_reason(gating_budgets, n=overall.n_records) + elif verdict == "fail": + # A budget breach is named by the budget table and the hero's + # "N of M" line; a fail on severity alone has neither, and read + # "7 of 7 budgets within policy" under FAIL. + reason = _severity_reason(overall_comparisons, gating_budgets) + else: + reason = None invariant_breaches = _invariant_breaches(gating_budgets) if invariant_breaches: @@ -661,21 +663,10 @@ def _is_kind(r: EvalRecord, kind: str) -> bool: return prefix is not None and r.evaluator_name.startswith(prefix) -def _is_semantic_regression(r: EvalRecord) -> bool: - """Whether a semantic-evaluator record breached ``min_similarity``. - - Reuses the ``SEMANTIC_REGRESSION`` flag the evaluator already wrote at - scoring time, so the policy gate honours the same ``min_similarity`` - threshold as the report instead of re-deriving it from ``delta``. - """ - cats = r.metadata.get("failure_categories", []) - return isinstance(cats, list) and SEMANTIC_REGRESSION in cats - - def _is_regression(r: EvalRecord) -> bool: """Whether a record counts as a regression for policy budgets.""" - if _is_kind(r, _SEMANTIC_KIND): - return _is_semantic_regression(r) + if is_semantic(r): + return is_semantic_regression(r) return r.delta < 0 @@ -684,9 +675,11 @@ def _is_equivalent(r: EvalRecord) -> bool: Semantic drift within ``min_similarity`` is treated as equivalent (not a regression), keeping the improved/equivalent/regressed partition intact. + The same rule decides what the paired test sees — see + :mod:`evalshift_cli.analysis.tolerance`. """ - if _is_kind(r, _SEMANTIC_KIND): - return not _is_semantic_regression(r) + if is_semantic(r): + return within_tolerance(r) return r.delta == 0 @@ -730,8 +723,9 @@ def is_shared_ground_truth_miss(r: EvalRecord) -> bool: count in ``failure_categories``). Reuses the flag the evaluator already wrote, on the same rule as - :func:`_is_semantic_regression`: one definition of "both sides missed", - in the evaluator that knows the expectation. + :func:`~evalshift_cli.analysis.tolerance.is_semantic_regression`: one + definition of "both sides missed", in the evaluator that knows the + expectation. The ``delta == 0`` guard is what keeps the exclusion from eating real findings. A conformance row where both sides missed *and* the target @@ -1323,6 +1317,21 @@ def _distinct_slice_budgets( return kept +def gating_budget_results(decision: MigrationDecision) -> list[BudgetResult]: + """Every budget that gated ``decision``'s verdict, each breach listed once. + + The overall rows, then each slice row that measures something an overall + one does not — the same list :func:`evaluate_migration_policy` decides + on, rebuilt from the persisted decision. ``decision.budget_results`` alone + is only the overall scope, and a run a slice budget failed has every one + of those green. + """ + return [ + *decision.budget_results, + *_distinct_slice_budgets(decision.budget_results, decision.slices), + ] + + def _budget_identity(budget: BudgetResult) -> tuple[str, float, float, bool, bool, int | None]: """Everything about a budget except which scope reported it.""" return ( @@ -1370,6 +1379,33 @@ def _invariant_reason(breaches: list[BudgetResult]) -> str: ) +def _severity_reason( + comparisons: list[ComparisonResult], + budgets: list[BudgetResult], +) -> str | None: + """Why a run failed on comparison severity alone, naming every blocking row. + + ``None`` when a budget's confirmed breach is (also) why it failed — the + budget rows already say so — or when no comparison reached a blocking + severity. The invariant and dropped-params reasons are applied after this + and keep their precedence. + """ + if any(not b.passed and b.conclusive for b in budgets): + return None + blocking = [c for c in comparisons if c.severity in _BLOCKING_SEVERITIES] + if not blocking: + return None + named = "; ".join( + f"{c.evaluator_name} regressed at {c.severity} severity on prompt {c.prompt_id} " + f"(effect size {c.effect_size:+.2f}, adjusted p {c.p_value_corrected:.4f})" + for c in blocking + ) + return ( + f"{named}. A critical or high regression from a blocking evaluator fails the run " + "on its own, whatever the budgets say." + ) + + def _violated_rule_ids(raw: Any) -> set[str]: """The ``rule_id`` of every violation in one side's metadata list.""" if not isinstance(raw, list): @@ -1825,6 +1861,7 @@ def _relative_increase(calls: list[Call], *, field: str) -> float: "PolicyMetricSummary", "SliceDecision", "evaluate_migration_policy", + "gating_budget_results", "inconclusive_decision", "is_shared_ground_truth_miss", "unmeasured_gating_evaluators", diff --git a/src/evalshift_cli/analysis/slicing.py b/src/evalshift_cli/analysis/slicing.py index d3362ce..c8a5f7c 100644 --- a/src/evalshift_cli/analysis/slicing.py +++ b/src/evalshift_cli/analysis/slicing.py @@ -30,6 +30,7 @@ from collections.abc import Sequence from dataclasses import dataclass +from evalshift_cli.analysis.tolerance import within_tolerance from evalshift_cli.evaluators.base import EvalRecord from evalshift_cli.runner.models import EvaluatorCoverage from evalshift_cli.suite.models import Suite, SuiteExample @@ -63,6 +64,15 @@ class SlicedScore: #: before slugs existed carries none, and because it is only ever a #: grouping key — an empty one groups with its own kind, as it should. kind: str = "" + #: The row's drop is inside its evaluator's own threshold — a semantic row + #: above ``min_similarity`` — so the paired test reads it as no change, + #: exactly as the policy budgets count it (see + #: :mod:`evalshift_cli.analysis.tolerance`). ``delta`` stays raw for the + #: aggregates. Defaulted so every construction that predates the field — + #: positional or keyword, in code and tests — keeps working (it also has + #: to be, following the defaulted ``kind``); :func:`build_slices` is what + #: sets it from the record. + within_tolerance: bool = False @dataclass(frozen=True, slots=True) @@ -117,6 +127,7 @@ def build_slices( target_score=rec.target_score, delta=rec.delta, kind=rec.kind, + within_tolerance=within_tolerance(rec), ) for slice_name in _slices_of(rec.example_id, by_id): out[slice_name].append(sliced) diff --git a/src/evalshift_cli/analysis/statistics.py b/src/evalshift_cli/analysis/statistics.py index 8af137e..af109df 100644 --- a/src/evalshift_cli/analysis/statistics.py +++ b/src/evalshift_cli/analysis/statistics.py @@ -5,7 +5,8 @@ 1. Pair scores per slice; require ``n>=5`` to test, flag ``n<20``. 2. Test normality via Shapiro-Wilk on the deltas; route to a paired t-test (normal) or Wilcoxon signed-rank (non-normal). -3. Compute Cohen's d for paired samples (``mean(deltas)/std(deltas)``). +3. Compute Cohen's d for paired samples (``mean(deltas)/std(deltas)``) + over the tolerance-adjusted deltas. 4. Compute a 95% CI on the effect size — analytical for the t-test, bootstrap for Wilcoxon. 5. After every comparison in the run is collected, apply the @@ -88,6 +89,13 @@ #: :func:`evalshift_cli.analysis.slicing.build_unmeasured`. NOT_APPLICABLE_REASON = "the evaluator measured nothing on them" +#: Suffix of the note counting rows whose drop sat inside the evaluator's own +#: threshold (a semantic row above ``min_similarity``). Those rows enter the +#: test as a delta of zero — see :mod:`evalshift_cli.analysis.tolerance` — +#: and the note is what tells a reader why a visible average drift tested as +#: no change. +WITHIN_TOLERANCE_REASON = "tested as no change" + @dataclass(frozen=True, slots=True) class ComparisonResult: @@ -278,6 +286,15 @@ def _one_comparison( """ n = len(pairs) deltas = np.array([p.delta for p in pairs], dtype=float) + # What the test sees: a drop inside the evaluator's own threshold is no + # change, exactly as the policy budgets count it. ``deltas`` stays raw + # for ``delta_avg_score``, which describes the drift rather than judging + # it. Only a semantic row is ever forgiven, and a semantic delta is never + # positive (the source is pinned at 1.0), so zeroing forgiven rows moves + # the tested mean from the raw one toward zero — never past it, so the + # two means never disagree in sign. + tested = np.array([0.0 if p.within_tolerance else p.delta for p in pairs], dtype=float) + forgiven = sum(1 for p in pairs if p.within_tolerance) notes: list[str] = [] if name_axis: notes.append(f"{AXIS_NOTE_PREFIX} {kind}") @@ -285,6 +302,11 @@ def _one_comparison( notes.append( f"{n_unmeasured} of {n + n_unmeasured} rows not applicable — {NOT_APPLICABLE_REASON}", ) + if forgiven: + notes.append( + f"{forgiven} of {n} rows within the evaluator's own threshold " + f"(min_similarity) — {WITHIN_TOLERANCE_REASON}", + ) if n == 0: # Every pair was a non-measurement. This is emphatically not @@ -341,7 +363,7 @@ def _one_comparison( # Variance ≈ 0 (every delta identical, modulo float noise) → no # inference possible. Use a small tolerance to dodge catastrophic # cancellation that scipy already warns about. - if float(np.std(deltas, ddof=1)) < 1e-9: + if float(np.std(tested, ddof=1)) < 1e-9: return ComparisonResult( prompt_id=prompt_id, evaluator_name=evaluator_name, @@ -361,24 +383,23 @@ def _one_comparison( ) # Pick the test based on a normality screen. - use_t = _is_normal(deltas) + use_t = _is_normal(tested) if use_t: - result = stats.ttest_rel( - [p.target_score for p in pairs], - [p.source_score for p in pairs], - ) + # One-sample t on the paired deltas is ``ttest_rel(target, source)`` + # exactly; it is written this way so a forgiven row can be zero. + result = stats.ttest_1samp(tested, 0.0) statistic = float(result.statistic) p_value = float(result.pvalue) - d = _cohens_d_paired(deltas) + d = _cohens_d_paired(tested) ci_low, ci_high = _t_test_d_ci(d=d, n=n) test_kind: TestKind = "paired_t" else: - result = stats.wilcoxon(deltas, zero_method="wilcox", correction=False) + result = stats.wilcoxon(tested, zero_method="wilcox", correction=False) statistic = float(result.statistic) p_value = float(result.pvalue) - d = _cohens_d_paired(deltas) - ci_low, ci_high = _bootstrap_d_ci(deltas, rng=rng) + d = _cohens_d_paired(tested) + ci_low, ci_high = _bootstrap_d_ci(tested, rng=rng) test_kind = "wilcoxon" notes.append("non-normal deltas; used Wilcoxon") @@ -528,6 +549,7 @@ def _severity_sort_key(c: ComparisonResult) -> tuple[int, float, str, str, str]: "NORMALITY_ALPHA", "NOT_APPLICABLE_REASON", "UNMEASURED_NOTE_PREFIX", + "WITHIN_TOLERANCE_REASON", "ComparisonResult", "Severity", "TestKind", diff --git a/src/evalshift_cli/analysis/tolerance.py b/src/evalshift_cli/analysis/tolerance.py new file mode 100644 index 0000000..cba2dc3 --- /dev/null +++ b/src/evalshift_cli/analysis/tolerance.py @@ -0,0 +1,66 @@ +"""Which score drops an evaluator's own threshold already forgives. + +``semantic`` pins the source at 1.0, so every target that words anything +differently scores below it: a negative delta is the evaluator's *normal* +output, not evidence of a regression. ``min_similarity`` is where the user +drew the line, and the evaluator records which side of it a row fell on as a +``SEMANTIC_REGRESSION`` failure category at scoring time. + +The policy budgets and the paired significance test must read that one line. +When only the budgets did, nine rows at 0.90-0.98 passed every budget and still +failed the run as a "critical" regression — nine small, consistent drops have +almost no variance, so the test was certain of a difference nobody had asked +it to find. +""" + +from __future__ import annotations + +from evalshift_cli.evaluators.base import EvalRecord +from evalshift_cli.evaluators.failures import SEMANTIC_REGRESSION + +#: The ``EvalRecord.kind`` slug the semantic evaluator stamps. +SEMANTIC_KIND = "semantic" + +#: Records checkpointed before ``EvalRecord.kind`` existed carry only the name +#: prefix the evaluator was registered under. +_LEGACY_SEMANTIC_PREFIX = "semantic." + + +def is_semantic(record: EvalRecord) -> bool: + """Whether ``record`` came from a ``semantic`` evaluator. + + Selected by kind, so a user who renames the evaluator keeps the rule; + falls back to the legacy name prefix only for rows written without one. + """ + if record.kind: + return record.kind == SEMANTIC_KIND + return record.evaluator_name.startswith(_LEGACY_SEMANTIC_PREFIX) + + +def is_semantic_regression(record: EvalRecord) -> bool: + """Whether a semantic record breached ``min_similarity``. + + Reads the ``SEMANTIC_REGRESSION`` flag the evaluator wrote at scoring time + rather than re-deriving it from ``delta``: the threshold lives in config, + and the evaluator is the one place that knew it. + """ + categories = record.metadata.get("failure_categories", []) + return isinstance(categories, list) and SEMANTIC_REGRESSION in categories + + +def within_tolerance(record: EvalRecord) -> bool: + """Whether ``record``'s drop is inside its evaluator's own threshold. + + True only for a semantic row that stayed above ``min_similarity``. Such a + row counts as *equivalent* in every policy rate and as *no change* in the + paired test; every other row is taken at its delta. + """ + return is_semantic(record) and not is_semantic_regression(record) + + +__all__ = [ + "SEMANTIC_KIND", + "is_semantic", + "is_semantic_regression", + "within_tolerance", +] diff --git a/src/evalshift_cli/cli/commands/compare.py b/src/evalshift_cli/cli/commands/compare.py index 60548e0..8080fc4 100644 --- a/src/evalshift_cli/cli/commands/compare.py +++ b/src/evalshift_cli/cli/commands/compare.py @@ -710,6 +710,12 @@ def _on_progress(ev: ProgressEvent) -> None: if evaluate_result.harness_check is not None: render_results([evaluate_result.harness_check], console) console.print() + # Same reason: a tool evaluator that checked nothing is a gate the reader + # configured and the verdict below did not use. + if evaluate_result.tool_free_warnings: + for line in evaluate_result.tool_free_warnings: + console.print(line) + console.print() collapsed_note = format_collapsed_slices(analyze_result.collapsed_slices) if collapsed_note is not None: console.print(collapsed_note, style="dim") diff --git a/src/evalshift_cli/cli/commands/evaluate.py b/src/evalshift_cli/cli/commands/evaluate.py index a21a0d0..2ad203e 100644 --- a/src/evalshift_cli/cli/commands/evaluate.py +++ b/src/evalshift_cli/cli/commands/evaluate.py @@ -183,6 +183,11 @@ class EvaluateResult: #: Advisory evaluators dropped because their model had no API key; the #: same list ``state.json`` carries as ``skipped_evaluators``. skipped: tuple[SkippedEvaluator, ...] = () + #: One rendered console line per tool evaluator a tool-free run left + #: nothing to check. Carried for the same reason as ``harness_check``: + #: ``evalshift compare`` scores quietly and prints these itself. In memory + #: only — nothing persisted changes. + tool_free_warnings: tuple[str, ...] = () class NoEvaluatorsError(ValueError): @@ -288,9 +293,14 @@ def run_evaluate( ), ) + tool_free = _tool_free_evaluators(evaluators, pairs) + tool_free_warnings = tuple(_tool_free_warning(e, state.suite_name) for e in tool_free) if not quiet: + for line in tool_free_warnings: + console.print(line) for evaluator in _unchecked_trace_invariants(evaluators, coverage): - console.print(_unchecked_trace_invariants_warning(evaluator)) + if evaluator not in tool_free: + console.print(_unchecked_trace_invariants_warning(evaluator)) output_path = run_dir / SCORES_FILENAME with output_path.open("w", encoding="utf-8") as fh: @@ -344,6 +354,7 @@ def run_evaluate( coverage=tuple(coverage), harness_check=harness_check, skipped=skipped, + tool_free_warnings=tool_free_warnings, ) @@ -711,6 +722,77 @@ def _applies(evaluator: Evaluator, prompt_id: str) -> bool: return True if applies is None else bool(applies(prompt_id)) +def _offers_no_tools(pair: _PairedCalls) -> bool: + """Whether neither side of ``pair`` was offered a tool. + + The orchestrator dispatches with tools exactly when the example's resolved + toolset is non-empty, and that path always stores a :class:`ToolTrace` on + the call. Two error-free calls with no trace between them are therefore a + plain text exchange, and a tool evaluator has nothing it could check — the + same as an ``applies_to`` miss. An errored call proves nothing about the + toolset and keeps the error path; one trace without the other stays + unmeasured (see :func:`_score_one_tool`). + """ + return ( + pair.source.error is None + and pair.target.error is None + and pair.source.trace is None + and pair.target.trace is None + ) + + +def _attempts(evaluator: Evaluator, pair: _PairedCalls) -> bool: + """Whether ``evaluator`` is asked to score ``pair`` at all. + + ``False`` means no cell: no row, and no coverage attempt that would later + read as "unmeasured" and demote the verdict. Booking a tool-free suite as + unmeasured is what turned a clean ``pass`` into ``conditional_pass`` for a + top-level ``trace_invariants`` entry written for a different suite. + """ + if not _applies(evaluator, pair.prompt_id): + return False + return not (_is_tool_evaluator(evaluator) and _offers_no_tools(pair)) + + +def _tool_free_evaluators( + evaluators: list[Evaluator], + pairs: list[_PairedCalls], +) -> list[Evaluator]: + """Tool evaluators whose every in-scope pair offered no tools. + + Computed from the pairs rather than from coverage: an evaluator with no + coverage entry may also be an ``applies_to`` miss, which has its own + warning and its own fix. + + Args: + evaluators: Every evaluator built for the run. + pairs: Every (source, target) pair in the run. + + Returns: + The ``score_pair`` evaluators that checked nothing because no pair + they were scoped to was offered a tool, in configured order. + """ + out: list[Evaluator] = [] + for evaluator in evaluators: + if not _is_tool_evaluator(evaluator): + continue + scoped = [pair for pair in pairs if _applies(evaluator, pair.prompt_id)] + if scoped and all(_offers_no_tools(pair) for pair in scoped): + out.append(evaluator) + return out + + +def _tool_free_warning(evaluator: Evaluator, suite_name: str | None) -> str: + """The console line for a tool evaluator that a tool-free run left nothing to check.""" + where = f"suite {suite_name!r}" if suite_name else "this run" + kind = _evaluator_kind(evaluator) or "tool" + return ( + f"[yellow]⚠[/yellow] {escape(kind)} evaluator {escape(repr(evaluator.name))} " + f"checked nothing: no example in {escape(where)} offered the models any tools, " + "so there were no tool calls to check. It did not gate this run." + ) + + def _unchecked_trace_invariants( evaluators: list[Evaluator], coverage: list[EvaluatorCoverage], @@ -904,7 +986,7 @@ async def _score_all( (pair, evaluator) for pair in pairs for evaluator in evaluators - if _applies(evaluator, pair.prompt_id) + if _attempts(evaluator, pair) ] sem = asyncio.Semaphore(concurrency) progress: Progress | None = None @@ -1260,8 +1342,7 @@ async def _score_one( return _error_records(evaluator, pair, run_id, "output truncated (token cap)") # v0.2 — tool evaluators consume the (source, target) ToolTrace pair. - # Plain prompts have ``call.trace is None`` and we skip cleanly with - # a neutral record so the analysis layer doesn't see noise. + # A pair offered no tools never reaches here (see _attempts). if _is_tool_evaluator(evaluator): return await _score_one_tool(evaluator, pair, run_id, examples_by_id) @@ -1313,15 +1394,20 @@ async def _score_one_tool( run_id: str, examples_by_id: dict[str, SuiteExample], ) -> list[EvalRecord]: - """Dispatch a v0.2 tool evaluator. Skips plain (text) calls cleanly. + """Dispatch a v0.2 tool evaluator over an error-free, untruncated pair. + + A pair offered no tools never reaches here — :func:`_attempts` drops it + before scoring, so it has no row and no coverage. The only pair that + returns ``[]`` here is a one-sided one (one call carries a trace, the + other does not), which coverage then books as unmeasured. ``score_pair`` returns one record, ``None``, or — for the two-axis ``tool_selection`` — a list of them; all three normalise to a list here. """ if pair.source.trace is None or pair.target.trace is None: - # No tool trace on either side → nothing for this evaluator to - # measure, so nothing to report. The 1.0/1.0 this used to write - # claimed a perfect tool match on a plain text prompt. + # Only one side carries a trace — a pair offered no tools at all + # never gets here (see _attempts). There is nothing to pair, so + # nothing to report; coverage books it unmeasured. return [] # tools=[]: this placeholder stands in for an example the loaded suite has # no record of at all, so there is no real toolset to carry -- not an diff --git a/src/evalshift_cli/insights/facts.py b/src/evalshift_cli/insights/facts.py index 2d8ea83..e68fec1 100644 --- a/src/evalshift_cli/insights/facts.py +++ b/src/evalshift_cli/insights/facts.py @@ -17,9 +17,12 @@ from evalshift_cli.analysis.policy import ( BUDGET_LABELS, + BudgetResult, MigrationDecision, + gating_budget_results, unmeasured_gating_evaluators, ) +from evalshift_cli.analysis.slicing import ALL_SLICE from evalshift_cli.analysis.statistics import ComparisonResult from evalshift_cli.evaluators.base import EvalRecord from evalshift_cli.insights.models import ( @@ -142,6 +145,19 @@ class Facts: #: as :attr:`rates_basis`: a name without a reason is a fact the model will #: explain for itself. coverage_basis: str = "" + #: Display names of the budgets whose breach the sample confirmed — the + #: overall rows and the slice rows alike, since a slice budget gates the + #: run too; a slice row is named for its scope ("'security' slice overall + #: regression rate"). Empty on a fail that came from comparison severity + #: alone — which is how the templated advice tells the two causes apart + #: instead of claiming a breach the budget table contradicts. + failed_budgets: list[str] = field(default_factory=list) + #: Blocking evaluators with a critical/high regression on the overall + #: slice — the only ones whose severity can fail the run. A slice-only + #: blocking row demotes the verdict to ``conditional_pass`` at most, so + #: naming it as why a run failed would be false. ``blocking_evaluators`` + #: keeps every scope, for the prose that lists them rather than blames them. + overall_blocking_evaluators: list[str] = field(default_factory=list) def build_facts( @@ -201,6 +217,11 @@ def build_facts( coverage_basis = ( _COVERAGE_BASIS_UNMEASURED if (unmeasured_budgets or unmeasured_evaluators) else "" ) + failed_budgets = _unique( + _failed_budget_label(b) + for b in gating_budget_results(decision) + if not b.passed and b.conclusive + ) rendered: dict[str, str] = { "verdict": decision.verdict.replace("_", " ").upper(), @@ -212,7 +233,12 @@ def build_facts( "budgets_total": str(len(decision.budget_results)), "budgets_unmeasured": str(len(unmeasured_budgets)), "blocking_regressions": str(len(decision.blocking_regressions)), - "critical_regressions": str(decision.overall.critical_regressions), + # Counted from the blocking comparisons themselves. The policy metric + # of the same name reads a per-record severity no evaluator writes, + # so it is always 0 and once sat beside a critical blocking item. + "critical_regressions": str( + sum(1 for r in decision.blocking_regressions if r.severity == "critical"), + ), "n_examples": str(len(examples)), "n_calls": str(source.calls + target.calls), "cost_source_usd": _usd(source.total_cost_usd), @@ -278,6 +304,10 @@ def build_facts( unmeasured_budgets=unmeasured_budgets, unmeasured_evaluators=unmeasured_evaluators, coverage_basis=coverage_basis, + failed_budgets=failed_budgets, + overall_blocking_evaluators=_unique( + r.evaluator_name for r in decision.blocking_regressions if r.slice_name == ALL_SLICE + ), ) @@ -327,6 +357,20 @@ def _p_value(value: float) -> str: return f"< {P_VALUE_FLOOR}" if value < P_VALUE_FLOOR else f"{value:.4f}" +def _failed_budget_label(budget: BudgetResult) -> str: + """A breached budget's display name, scoped when it is a slice's own row. + + A bare "overall regression rate" for a slice breach would send the reader + to the overall row — green in exactly the run this names. Worded so the + templated advice reads "the 'security' slice overall regression rate + budget was breached". + """ + label = BUDGET_LABELS.get(budget.name, budget.name) + if budget.scope == "overall": + return label + return f"'{budget.scope}' slice {label[:1].lower()}{label[1:]}" + + def _budget_limit(name: str, allowed: float) -> str: if name in _COUNT_BUDGETS: return f"{round(allowed):g}" diff --git a/src/evalshift_cli/insights/templates.py b/src/evalshift_cli/insights/templates.py index 25eac04..83bda1d 100644 --- a/src/evalshift_cli/insights/templates.py +++ b/src/evalshift_cli/insights/templates.py @@ -28,10 +28,6 @@ "Regressions were detected but no budget was breached. Review the sampled " "regressions and the blocking evaluators before promoting." ), - "fail": ( - "Do not migrate under the configured policy — at least one regression budget " - "was breached. Address the blocking evaluators, then re-run." - ), "inconclusive": ( "The evidence is not conclusive at this sample size. Capture more examples " "and re-run before deciding." @@ -48,20 +44,54 @@ def fallback_insight(facts: Facts) -> Insight: verdict_summary=_clamp(_verdict_summary(facts)), advisory_summary=_clamp(_advisory_summary(facts)), economics_summary=_clamp(_economics_summary(rendered)), - recommendation=_clamp( - _RECOMMENDATIONS.get(facts.verdict, _RECOMMENDATIONS["inconclusive"]), - ), + recommendation=_clamp(_recommendation(facts)), findings=[], ) +def _recommendation(facts: Facts) -> str: + """The verdict's advice, naming what actually failed the run. + + Keyed on the verdict alone, a ``fail`` once said "at least one regression + budget was breached" over a run whose every budget held: the fail came + from a blocking evaluator's severity. The facts carry the cause, so the + sentence names it: a confirmed budget breach first (overall or slice), + then an overall-slice blocking regression. + """ + if facts.verdict != "fail": + return _RECOMMENDATIONS.get(facts.verdict, _RECOMMENDATIONS["inconclusive"]) + if facts.failed_budgets: + labels = ", ".join(label[:1].lower() + label[1:] for label in facts.failed_budgets) + noun = "budget was" if len(facts.failed_budgets) == 1 else "budgets were" + return ( + f"Do not migrate under the configured policy — the {labels} {noun} breached. " + "Address the blocking evaluators, then re-run." + ) + if facts.overall_blocking_evaluators: + names = facts.overall_blocking_evaluators + whose = "its" if len(names) == 1 else "their" + return ( + "Do not migrate under the configured policy — " + f"{', '.join(names)} regressed at blocking severity " + f"although no budget was breached. Review {whose} regressed examples, then re-run." + ) + # Not "see the verdict's reason": a fail can carry none (a slice-driven + # demotion overridden by dropped params, say), and advice pointing at an + # empty field is a dead end. + return ( + "Do not migrate under the configured policy — see the verdict's recommendations " + "and the budget table, address the cause, then re-run." + ) + + def _verdict_summary(facts: Facts) -> str: rendered = facts.rendered + blocking = rendered["blocking_regressions"] + noun = "regression" if blocking == "1" else "regressions" parts = [ f"{rendered['verdict']} under the configured policy. " f"{rendered['budgets_passed']} of {rendered['budgets_total']} budgets passed, " - f"with {rendered['blocking_regressions']} blocking and " - f"{rendered['critical_regressions']} critical regressions " + f"with {blocking} blocking {noun} ({rendered['critical_regressions']} critical) " f"across {rendered['n_examples']} examples.", ] # `budgets_passed` already excludes the gates that measured nothing, so diff --git a/tests/unit/test_analysis.py b/tests/unit/test_analysis.py index 23a9f58..1e324df 100644 --- a/tests/unit/test_analysis.py +++ b/tests/unit/test_analysis.py @@ -6,6 +6,7 @@ import numpy as np import pytest +from scipy import stats from evalshift_cli.analysis.slicing import ( ALL_SLICE, @@ -20,6 +21,7 @@ ADVISORY_NOTE_PREFIX, AXIS_NOTE_PREFIX, UNMEASURED_NOTE_PREFIX, + WITHIN_TOLERANCE_REASON, ComparisonResult, _benjamini_hochberg, _classify_severity, @@ -27,6 +29,7 @@ analyze, ) from evalshift_cli.evaluators.base import EvalRecord +from evalshift_cli.evaluators.failures import SEMANTIC_REGRESSION from evalshift_cli.runner.models import EvaluatorCoverage, UnmeasuredPair from evalshift_cli.suite.models import Suite from tests.unit.suite_examples import suite_example @@ -717,3 +720,117 @@ def test_unmeasured_note_survives_the_bundle_comparison_schema(self) -> None: "notes", } assert set(asdict(result)) == allowed + + +# --------------------------------------------------------------------------- +# Semantic drift inside min_similarity — tested as no change +# --------------------------------------------------------------------------- + +#: The nine semantic.cosine target scores of run r_20261011_triage_e14cda, +#: every one above min_similarity 0.9 — the run that failed as "critical". +_TRIAGE_TARGETS = [0.9734, 0.9608, 0.9212, 0.9531, 0.9321, 0.9492, 0.9043, 0.9800, 0.9542] + + +def _semantic_record(example: str, target: float, *, flagged: bool) -> EvalRecord: + return EvalRecord( + run_id="r1", + prompt_id="replay", + example_id=example, + evaluator_name="semantic.cosine", + kind="semantic", + source_score=1.0, + target_score=target, + delta=target - 1.0, + metadata={"failure_categories": [SEMANTIC_REGRESSION]} if flagged else {}, + ) + + +def _semantic_sliced(targets: list[float], flagged: list[bool]) -> dict[str, list[SlicedScore]]: + records = [ + _semantic_record(f"ex{i}", t, flagged=f) + for i, (t, f) in enumerate(zip(targets, flagged, strict=True)) + ] + suite = Suite(examples=[suite_example(id=r.example_id) for r in records]) + return build_slices(records=records, suite=suite) + + +class TestWithinTolerance: + def test_build_slices_marks_only_unflagged_semantic_rows(self) -> None: + records = [ + _semantic_record("ex1", 0.95, flagged=False), + _semantic_record("ex2", 0.70, flagged=True), + _record(example="ex3", delta=-0.2), + ] + suite = Suite(examples=[suite_example(id=f"ex{i}") for i in (1, 2, 3)]) + sliced = build_slices(records=records, suite=suite)[ALL_SLICE] + assert [s.within_tolerance for s in sliced] == [True, False, False] + + def test_drift_inside_min_similarity_is_not_a_regression(self) -> None: + sliced = _semantic_sliced(_TRIAGE_TARGETS, [False] * 9) + (c,) = analyze(sliced_by_slice=sliced) + assert c.severity == "none" + assert c.test == "skipped" + # Descriptive mean stays the raw drift: the report still says -5%. + assert c.delta_avg_score == pytest.approx( + sum(t - 1.0 for t in _TRIAGE_TARGETS) / 9, + ) + assert ( + "9 of 9 rows within the evaluator's own threshold (min_similarity) — " + f"{WITHIN_TOLERANCE_REASON}" + ) in c.notes + + def test_drift_below_min_similarity_still_regresses(self) -> None: + targets = [0.70, 0.72, 0.75, 0.71, 0.74, 0.73, 0.76, 0.69, 0.72] + (c,) = analyze(sliced_by_slice=_semantic_sliced(targets, [True] * 9)) + assert c.severity == "critical" + assert not any(WITHIN_TOLERANCE_REASON in note for note in c.notes) + + def test_one_flagged_row_among_forgiven_ones_runs_and_stays_quiet(self) -> None: + targets = [*_TRIAGE_TARGETS[:8], 0.60] + flagged = [False] * 8 + [True] + (c,) = analyze(sliced_by_slice=_semantic_sliced(targets, flagged)) + assert c.test == "wilcoxon" + assert c.severity == "none" + assert any(note.startswith("8 of 9 rows within") for note in c.notes) + # The effect size reads the tolerance-adjusted deltas, not the raw + # drift: on the raw ones d is about -0.78, which reads as a regression. + tested = np.array([0.0] * 8 + [0.60 - 1.0]) + assert c.effect_size == pytest.approx( + float(np.mean(tested) / np.std(tested, ddof=1)), + ) + + def test_the_t_path_sizes_its_effect_and_ci_on_adjusted_deltas(self) -> None: + """Forgiven rows reach the paired t too, and its CI is built from that d.""" + flagged_deltas = [ + -0.30, -0.26, -0.24, -0.22, -0.21, -0.20, -0.19, -0.18, + -0.17, -0.16, -0.15, -0.14, -0.12, -0.10, -0.08, -0.06, + ] # fmt: skip + forgiven_targets = [0.96, 0.95] + targets = [1.0 + d for d in flagged_deltas] + forgiven_targets + flagged = [True] * len(flagged_deltas) + [False] * len(forgiven_targets) + (c,) = analyze(sliced_by_slice=_semantic_sliced(targets, flagged)) + assert c.test == "paired_t" + tested = np.array([*flagged_deltas, 0.0, 0.0]) + d = float(np.mean(tested) / np.std(tested, ddof=1)) + assert c.effect_size == pytest.approx(d) + se = float(np.sqrt(1.0 / tested.size + d * d / (2.0 * tested.size))) + assert c.effect_size_ci_low == pytest.approx(d - 1.96 * se) + assert c.effect_size_ci_high == pytest.approx(d + 1.96 * se) + # The descriptive mean is still the raw drift. + assert c.delta_avg_score == pytest.approx(float(np.mean([t - 1.0 for t in targets]))) + + def test_non_semantic_statistics_are_unchanged(self) -> None: + pairs = _ss(40, -0.3) + (c,) = analyze(sliced_by_slice={"all": pairs}) + assert c.test == "paired_t" + expected = stats.ttest_rel( + [p.target_score for p in pairs], + [p.source_score for p in pairs], + ) + assert c.statistic == pytest.approx(float(expected.statistic)) + assert c.p_value == pytest.approx(float(expected.pvalue)) + + def test_aggregates_keep_the_raw_drift(self) -> None: + sliced = _semantic_sliced(_TRIAGE_TARGETS, [False] * 9) + agg = aggregates(sliced[ALL_SLICE], ALL_SLICE) + assert agg.delta_min_score == pytest.approx(min(_TRIAGE_TARGETS) - 1.0) diff --git a/tests/unit/test_compare_command.py b/tests/unit/test_compare_command.py index d1e7655..17f6744 100644 --- a/tests/unit/test_compare_command.py +++ b/tests/unit/test_compare_command.py @@ -365,6 +365,43 @@ def test_all_advisory_inconclusive_prints_reason_and_fix( assert "Set blocking: true" in flat assert "Collect more examples" not in flat + def test_all_prints_the_tool_free_warning_it_scored_quietly( + self, + monkeypatch: pytest.MonkeyPatch, + tmp_path: Path, + ) -> None: + """Stage 4 scores quietly inside the Live grid; the warning is not lost.""" + _scaffold(tmp_path) + config_path = tmp_path / "evalshift.yaml" + config_path.write_text( + config_path.read_text(encoding="utf-8").replace( + " max_chars: 200", + " max_chars: 200\n" + " trace_invariants:\n" + " - name: payments_contract\n" + " rules:\n" + " - id: no-v1\n" + " type: forbidden\n" + " tools: [refund_v1]", + ), + encoding="utf-8", + ) + _patch_client(monkeypatch) + monkeypatch.chdir(tmp_path) + + result = runner.invoke(app, ["all", "--yes"]) + + assert result.exit_code == 0, result.output + flat = " ".join(result.output.split()) + assert ( + "trace_invariants evaluator 'payments_contract' checked nothing: no example in " + ) in flat + assert ( + "offered the models any tools, so there were no tool calls to check. " + "It did not gate this run." + ) in flat + assert flat.count("checked nothing") == 1 + def test_all_pushes_before_gate_exit( self, monkeypatch: pytest.MonkeyPatch, diff --git a/tests/unit/test_evaluate_command.py b/tests/unit/test_evaluate_command.py index 8684d95..c6914e5 100644 --- a/tests/unit/test_evaluate_command.py +++ b/tests/unit/test_evaluate_command.py @@ -4,7 +4,7 @@ import asyncio import json -from dataclasses import dataclass +from dataclasses import dataclass, replace from datetime import UTC, datetime from pathlib import Path from types import SimpleNamespace @@ -146,6 +146,7 @@ def _scaffold_run( tmp_path: Path, completed: bool = True, suite_name: str | None = None, + with_traces: bool = False, ) -> str: """Create a minimal completed run dir and return its run_id.""" runs_base = tmp_path / ".evalshift" / "runs" @@ -181,6 +182,11 @@ def _scaffold_run( model_id=model_id, role=role, # type: ignore[arg-type] text=text, + trace=( + ToolTrace(calls=[ToolCall(tool_name="lookup", sequence_index=0)]) + if with_traces + else None + ), ), ) return run_id @@ -1801,7 +1807,7 @@ def test_an_imported_evaluator_matching_no_trace_pair_warns(self, tmp_path: Path def test_partial_scoping_stays_silent(self, tmp_path: Path) -> None: _write_invariants_config(tmp_path, applies_to=["gre*", "checkout-*"]) - _scaffold_run(tmp_path) + _scaffold_run(tmp_path, with_traces=True) assert "checked nothing" not in _evaluate_output(tmp_path) state = read_state(tmp_path / ".evalshift" / "runs" / "r_20260601_aaaaaa") @@ -1836,6 +1842,55 @@ def test_the_cli_shows_the_warning( assert result.exit_code == 0 assert "checked nothing" in " ".join(result.stdout.split()) + def test_a_tool_free_suite_names_the_real_cause(self, tmp_path: Path) -> None: + _write_invariants_config(tmp_path, applies_to=["*"]) + _scaffold_run(tmp_path, suite_name="triage") + + output = " ".join(_evaluate_output(tmp_path).split()) + + assert ( + "trace_invariants evaluator 'payments_contract' checked nothing: no example in " + "suite 'triage' offered the models any tools, so there were no tool calls to " + "check. It did not gate this run." + ) in output + assert "applies_to" not in output + state = read_state(tmp_path / ".evalshift" / "runs" / "r_20260601_aaaaaa") + assert state.evaluator_coverage == [] + + def test_a_quiet_run_carries_the_tool_free_warning_on_its_result(self, tmp_path: Path) -> None: + """``evalshift compare`` scores quietly and renders the warning itself.""" + from evalshift_cli.cli.commands.evaluate import run_evaluate + + _write_invariants_config(tmp_path, applies_to=["*"]) + _scaffold_run(tmp_path, suite_name="triage") + console = Console(record=True, width=400) + + result = run_evaluate( + run_id="r_20260601_aaaaaa", + config_path=tmp_path / "evalshift.yaml", + runs_base=tmp_path / ".evalshift" / "runs", + console=console, + quiet=True, + ) + + assert "checked nothing" not in console.export_text() + assert result.tool_free_warnings == ( + "[yellow]⚠[/yellow] trace_invariants evaluator 'payments_contract' checked " + "nothing: no example in suite 'triage' offered the models any tools, so there " + "were no tool calls to check. It did not gate this run.", + ) + + def test_an_applies_to_miss_keeps_its_own_wording_on_a_tool_free_run( + self, tmp_path: Path + ) -> None: + _write_invariants_config(tmp_path, applies_to=["checkout-*"]) + _scaffold_run(tmp_path) + + output = _evaluate_output(tmp_path) + + assert "['checkout-*']" in output + assert "offered the models any tools" not in output + class TestMissingImportedTracesNamesTheFamily: def test_imported_trace_invariants_alone_is_named( @@ -2088,3 +2143,49 @@ def test_state_without_skipped_evaluators_still_loads(self, tmp_path: Path) -> N raw.pop("skipped_evaluators", None) state_path.write_text(json.dumps(raw), encoding="utf-8") assert read_state(state_path.parent).skipped_evaluators == [] + + +# --------------------------------------------------------------------------- +# Tool evaluators on pairs that offered no tools -- not applicable, not unmeasured +# --------------------------------------------------------------------------- + + +class TestToolEvaluatorsOnToolFreePairs: + def _invariants(self) -> TraceInvariantsEvaluator: + return TraceInvariantsEvaluator(TraceInvariantsEvaluatorConfig.model_validate(_INVARIANTS)) + + async def _cells(self, evaluator: Any, pairs: list[_PairedCalls]) -> list[_ScoredCell]: + return await _score_all(Console(), [evaluator], pairs, "r1", {}, concurrency=2, quiet=True) + + async def test_a_pair_that_offered_no_tools_is_never_attempted(self) -> None: + cells = await self._cells( + self._invariants(), + [_pair_for("ex1"), _tool_pair("greet", "ex2", "lookup")], + ) + assert [(c.example_id, c.record is not None) for c in cells] == [("ex2", True)] + coverage = _coverage_for(cells) + assert (coverage[0].attempted, coverage[0].unmeasured) == (1, []) + + async def test_a_wholly_tool_free_run_books_no_coverage(self) -> None: + cells = await self._cells(self._invariants(), [_pair_for("ex1"), _pair_for("ex2")]) + assert cells == [] + assert _coverage_for(cells) == [] + + async def test_text_evaluators_still_score_tool_free_pairs(self) -> None: + cells = await self._cells(_SometimesUnmeasuredEvaluator(), [_pair_for("ex1")]) + assert [c.record is not None for c in cells] == [True] + + async def test_a_one_sided_trace_is_still_unmeasured(self) -> None: + pair = _tool_pair("greet", "ex1", "lookup") + one_sided = replace(pair, target=pair.target.model_copy(update={"trace": None})) + cells = await self._cells(self._invariants(), [one_sided]) + coverage = _coverage_for(cells) + assert [p.example_id for p in coverage[0].unmeasured] == ["ex1"] + + async def test_an_errored_tool_free_pair_keeps_its_error_row(self) -> None: + pair = _pair_for("ex1") + errored = replace(pair, source=pair.source.model_copy(update={"error": "boom"})) + cells = await self._cells(self._invariants(), [errored]) + assert len(cells) == 1 + assert cells[0].record is not None + assert (cells[0].record.error or "").startswith("upstream call failed") diff --git a/tests/unit/test_insight_facts.py b/tests/unit/test_insight_facts.py index db4afe6..e7db20a 100644 --- a/tests/unit/test_insight_facts.py +++ b/tests/unit/test_insight_facts.py @@ -8,10 +8,19 @@ from typing import Any -from evalshift_cli.analysis.policy import BlockingRegression, BudgetResult, FailureCategoryCount +from evalshift_cli.analysis.policy import ( + BlockingRegression, + BudgetResult, + FailureCategoryCount, + evaluate_migration_policy, +) +from evalshift_cli.analysis.statistics import ComparisonResult +from evalshift_cli.config.models import MigrationPolicy, SliceMigrationPolicy +from evalshift_cli.evaluators.base import EvalRecord from evalshift_cli.insights.facts import build_facts from evalshift_cli.insights.templates import fallback_insight from evalshift_cli.reports.economics import PromptEconomics +from evalshift_cli.runner.models import Call from tests.unit.insights_factories import ( COST_TARGET, budgets, @@ -428,3 +437,243 @@ def test_the_fallback_is_unchanged_when_every_gate_measured( insight = fallback_insight(build_facts(**sample_run)) assert "6 of 6 budgets passed" in insight.verdict_summary assert "measured nothing" not in insight.verdict_summary + + +def _critical_semantic() -> BlockingRegression: + return BlockingRegression( + prompt_id="replay", + evaluator_name="semantic.cosine", + slice_name="all", + severity="critical", + delta_avg_score=-0.05, + effect_size=-2.15, + ) + + +def test_a_severity_fail_never_claims_a_budget_breach(sample_run: dict[str, Any]) -> None: + sample_run["decision"] = decision(verdict="fail", blocking=[_critical_semantic()]) + recommendation = fallback_insight(build_facts(**sample_run)).recommendation + assert recommendation == ( + "Do not migrate under the configured policy — semantic.cosine regressed at " + "blocking severity although no budget was breached. Review its regressed " + "examples, then re-run." + ) + + +def test_a_budget_fail_names_the_budget(sample_run: dict[str, Any]) -> None: + breached = BudgetResult( + name="max_overall_regression_rate", + observed=0.5, + allowed=0.03, + passed=False, + conclusive=True, + ci_low=0.3, + ci_high=0.7, + ) + sample_run["decision"] = decision( + verdict="fail", + blocking=[_critical_semantic()], + budget_results=[breached, *budgets()[1:]], + ) + facts = build_facts(**sample_run) + assert facts.failed_budgets == ["Overall regression rate"] + assert fallback_insight(facts).recommendation.startswith( + "Do not migrate under the configured policy — the overall regression rate " + "budget was breached.", + ) + + +def test_a_fail_with_no_named_cause_points_at_what_the_reader_can_see( + sample_run: dict[str, Any], +) -> None: + """``decision.reason`` can be ``None`` on a fail, so the advice never points there.""" + sample_run["decision"] = decision(verdict="fail") + assert fallback_insight(build_facts(**sample_run)).recommendation == ( + "Do not migrate under the configured policy — see the verdict's recommendations " + "and the budget table, address the cause, then re-run." + ) + + +# --------------------------------------------------------------------------- +# Slice budgets gate the run too +# --------------------------------------------------------------------------- + + +def _slice_comparison( + *, + evaluator_name: str, + slice_name: str, + severity: str = "none", +) -> ComparisonResult: + blocking = severity in {"critical", "high"} + return ComparisonResult( + prompt_id="replay", + evaluator_name=evaluator_name, + slice_name=slice_name, + n=20, + test="paired_t", + statistic=1.0, + p_value=0.01 if blocking else 0.6, + p_value_corrected=0.01 if blocking else 0.6, + effect_size=-0.9 if blocking else 0.0, + effect_size_ci_low=-1.2 if blocking else -0.3, + effect_size_ci_high=-0.6 if blocking else 0.3, + delta_avg_score=-0.2 if blocking else 0.0, + severity=severity, # type: ignore[arg-type] + notes=[], + ) + + +def _scored(evaluator_name: str, *, n: int, delta: float) -> list[EvalRecord]: + return [ + EvalRecord( + run_id="r_20260803_abcdef", + prompt_id="replay", + example_id=f"{evaluator_name}_{index}", + evaluator_name=evaluator_name, + source_score=1.0, + target_score=1.0 + delta, + delta=delta, + ) + for index in range(n) + ] + + +def _slice_budget_run( + sample_run: dict[str, Any], + *, + extra: list[ComparisonResult], + policy: MigrationPolicy | None = None, + calls: list[Call] | None = None, +) -> dict[str, Any]: + """A run whose overall budgets hold and whose 'security' slice budget breaks. + + Decided by the real policy rather than a hand-built decision, so the + facts are read off exactly the shape ``evaluate_migration_policy`` + produces: the slice row lives under ``decision.slices``, never in the + overall ``budget_results``. + """ + comparisons = [ + _slice_comparison(evaluator_name="structural.length", slice_name="all"), + _slice_comparison(evaluator_name="security.judge", slice_name="security"), + *extra, + ] + records = [ + *_scored("structural.length", n=20, delta=0.0), + *_scored("security.judge", n=20, delta=-0.5), + ] + sample_run["decision"] = evaluate_migration_policy( + run_id="r_20260803_abcdef", + source_model="src", + target_model="tgt", + policy=policy + or MigrationPolicy( + max_overall_regression_rate=0.5, + min_equivalence_rate=0.0, + slices={"security": SliceMigrationPolicy(max_overall_regression_rate=0.0)}, + ), + comparisons=comparisons, + records=records, + calls=calls or [], + ) + sample_run["comparisons"] = comparisons + sample_run["records"] = records + return sample_run + + +def test_a_slice_budget_fail_names_the_slice_budget(sample_run: dict[str, Any]) -> None: + run = _slice_budget_run(sample_run, extra=[]) + assert run["decision"].verdict == "fail" + assert run["decision"].reason is None + + facts = build_facts(**run) + + assert facts.failed_budgets == ["'security' slice overall regression rate"] + assert fallback_insight(facts).recommendation == ( + "Do not migrate under the configured policy — the 'security' slice overall " + "regression rate budget was breached. Address the blocking evaluators, then re-run." + ) + + +def test_a_slice_budget_fail_with_a_slice_only_blocking_row_blames_the_budget( + sample_run: dict[str, Any], +) -> None: + """Slice severity only demotes to conditional_pass; the budget is what failed.""" + run = _slice_budget_run( + sample_run, + extra=[ + _slice_comparison( + evaluator_name="structural.length", + slice_name="security", + severity="high", + ), + ], + ) + assert run["decision"].verdict == "fail" + + recommendation = fallback_insight(build_facts(**run)).recommendation + + assert "'security' slice overall regression rate budget was breached" in recommendation + assert "no budget was breached" not in recommendation + + +def test_a_slice_row_that_restates_an_overall_breach_is_named_once( + sample_run: dict[str, Any], +) -> None: + """An inheriting slice copies the run-level cost row; one breach, one name.""" + run = _slice_budget_run( + sample_run, + extra=[], + policy=MigrationPolicy( + max_overall_regression_rate=1.0, + min_equivalence_rate=0.0, + max_cost_increase=0.0, + slices={"security": SliceMigrationPolicy()}, + ), + calls=[ + Call( + run_id="r_20260803_abcdef", + prompt_id="replay", + example_id="e1", + model_id=model_role, + role=model_role, # type: ignore[arg-type] + text="ok", + cost_usd=cost, + latency_ms=100, + ) + for model_role, cost in (("source", 1.0), ("target", 2.0)) + ], + ) + assert run["decision"].verdict == "fail" + + assert build_facts(**run).failed_budgets == ["Cost increase"] + + +def test_a_severity_fail_names_only_the_overall_blocking_evaluators( + sample_run: dict[str, Any], +) -> None: + """A slice-only blocking row did not fail the run, so it is not named as why.""" + slice_only = BlockingRegression( + prompt_id="replay", + evaluator_name="structural.length", + slice_name="security", + severity="high", + delta_avg_score=-0.2, + effect_size=-0.9, + ) + sample_run["decision"] = decision(verdict="fail", blocking=[_critical_semantic(), slice_only]) + + recommendation = fallback_insight(build_facts(**sample_run)).recommendation + + assert recommendation == ( + "Do not migrate under the configured policy — semantic.cosine regressed at " + "blocking severity although no budget was breached. Review its regressed " + "examples, then re-run." + ) + + +def test_the_critical_count_reads_the_blocking_regressions(sample_run: dict[str, Any]) -> None: + sample_run["decision"] = decision(verdict="fail", blocking=[_critical_semantic()]) + facts = build_facts(**sample_run) + assert facts.rendered["critical_regressions"] == "1" + assert "1 blocking regression (1 critical)" in fallback_insight(facts).verdict_summary diff --git a/tests/unit/test_policy.py b/tests/unit/test_policy.py index 65c2e85..6bea8d6 100644 --- a/tests/unit/test_policy.py +++ b/tests/unit/test_policy.py @@ -15,21 +15,26 @@ inconclusive_decision, unmeasured_gating_evaluators, ) +from evalshift_cli.analysis.slicing import build_slices from evalshift_cli.analysis.statistics import ( ADVISORY_NOTE_PREFIX, UNMEASURED_NOTE_PREFIX, ComparisonResult, + analyze, ) from evalshift_cli.config.models import MigrationPolicy, SliceMigrationPolicy from evalshift_cli.evaluators.base import EvalRecord from evalshift_cli.evaluators.failures import ( INVARIANT_VIOLATION, + SEMANTIC_REGRESSION, TOOL_GROUND_TRUTH_MISS, TOOL_SELECTION_DRIFT, ) from evalshift_cli.evaluators.tool_arguments import KIND as KIND_ARGUMENTS from evalshift_cli.evaluators.tool_selection import KIND_CONFORMANCE, KIND_DIVERGENCE from evalshift_cli.runner.models import Call, SkippedEvaluator +from evalshift_cli.suite.models import Suite +from tests.unit.suite_examples import suite_example def _comparison( @@ -3164,3 +3169,125 @@ def test_no_policy_decision_carries_them(self) -> None: skipped_evaluators=(_SKIPPED_SEMANTIC,), ) assert _SKIPPED_LINE in decision.recommendations + + +class TestTriageRunRegression: + """r_20261011_triage_e14cda: every budget held and the run still failed.""" + + _TARGETS = (0.9734, 0.9608, 0.9212, 0.9531, 0.9321, 0.9492, 0.9043, 0.9800, 0.9542) + + def _records(self, *, flagged: bool) -> list[EvalRecord]: + semantic = [ + EvalRecord( + run_id="r1", + prompt_id="replay", + example_id=f"ex{i}", + evaluator_name="semantic.cosine", + kind="semantic", + source_score=1.0, + target_score=t, + delta=t - 1.0, + metadata={"failure_categories": [SEMANTIC_REGRESSION]} if flagged else {}, + ) + for i, t in enumerate(self._TARGETS) + ] + judge = [ + EvalRecord( + run_id="r1", + prompt_id="replay", + example_id=f"ex{i}", + evaluator_name="llm_judge.equivalence", + kind="llm_judge", + source_score=0.0 if i < 7 else 0.5, + target_score=1.0 if i < 7 else 0.5, + delta=1.0 if i < 7 else 0.0, + blocking=False, + ) + for i in range(9) + ] + return [*semantic, *judge] + + def _decide(self, records: list[EvalRecord]) -> MigrationDecision: + suite = Suite(examples=[suite_example(id=f"ex{i}") for i in range(9)]) + comparisons = analyze(sliced_by_slice=build_slices(records=records, suite=suite)) + return evaluate_migration_policy( + run_id="r1", + source_model="anthropic/claude-sonnet-5", + target_model="anthropic/claude-sonnet-5-5", + policy=MigrationPolicy( + max_overall_regression_rate=0.3, + max_critical_regressions=1, + min_equivalence_rate=0.75, + max_tool_argument_drift=0.2, + max_tool_divergence=0.2, + max_cost_increase=0.5, + max_latency_increase=0.3, + ), + comparisons=comparisons, + records=records, + calls=[], + ) + + def test_drift_inside_min_similarity_passes(self) -> None: + decision = self._decide(self._records(flagged=False)) + assert decision.verdict == "pass" + assert decision.blocking_regressions == [] + assert decision.recommendations[0] == "Safe to migrate under the configured policy." + + def test_the_same_drop_below_min_similarity_still_fails(self) -> None: + decision = self._decide(self._records(flagged=True)) + assert decision.verdict == "fail" + + +class TestSeverityDrivenFailReason: + def _clean_records(self) -> list[EvalRecord]: + return [_record(example_id=f"ex{i}", delta=0.0) for i in range(30)] + + def test_a_critical_comparison_with_every_budget_held_names_itself(self) -> None: + decision = _decide( + self._clean_records(), + comparisons=[ + _comparison( + severity="critical", + evaluator_name="semantic.cosine", + prompt_id="replay", + ), + ], + ) + assert decision.verdict == "fail" + assert decision.reason is not None + assert decision.reason.startswith( + "semantic.cosine regressed at critical severity on prompt replay " + "(effect size -0.80, adjusted p 0.0100)", + ) + assert decision.reason.endswith( + "A critical or high regression from a blocking evaluator fails the run " + "on its own, whatever the budgets say.", + ) + + def test_every_blocking_comparison_is_named(self) -> None: + decision = _decide( + self._clean_records(), + comparisons=[ + _comparison(severity="critical", evaluator_name="a.one"), + _comparison(severity="high", evaluator_name="b.two"), + _comparison(severity="medium", evaluator_name="c.three"), + ], + ) + assert decision.reason is not None + assert "a.one regressed at critical severity" in decision.reason + assert "b.two regressed at high severity" in decision.reason + assert "c.three" not in decision.reason + + def test_a_budget_breach_keeps_the_budget_line(self) -> None: + decision = _decide( + [_record(example_id=f"ex{i}", delta=-0.5) for i in range(30)], + comparisons=[_comparison(severity="critical")], + ) + assert decision.verdict == "fail" + assert decision.reason is None + + def test_a_pass_has_no_reason(self) -> None: + decision = _decide(self._clean_records(), comparisons=[_comparison(severity="none")]) + assert decision.verdict == "pass" + assert decision.reason is None diff --git a/tests/unit/test_tolerance.py b/tests/unit/test_tolerance.py new file mode 100644 index 0000000..68e4bbd --- /dev/null +++ b/tests/unit/test_tolerance.py @@ -0,0 +1,62 @@ +"""Tests for the shared within-tolerance predicate.""" + +from __future__ import annotations + +from evalshift_cli.analysis.tolerance import ( + is_semantic, + is_semantic_regression, + within_tolerance, +) +from evalshift_cli.evaluators.base import EvalRecord +from evalshift_cli.evaluators.failures import SEMANTIC_REGRESSION + + +def _record( + *, + kind: str = "semantic", + name: str = "semantic.cosine", + target: float = 0.95, + flagged: bool = False, +) -> EvalRecord: + return EvalRecord( + run_id="r1", + prompt_id="p", + example_id="ex1", + evaluator_name=name, + kind=kind, + source_score=1.0, + target_score=target, + delta=target - 1.0, + metadata={"failure_categories": [SEMANTIC_REGRESSION]} if flagged else {}, + ) + + +def test_unflagged_semantic_drift_is_within_tolerance() -> None: + assert within_tolerance(_record(target=0.904)) + + +def test_flagged_semantic_drift_is_not() -> None: + record = _record(target=0.75, flagged=True) + assert is_semantic_regression(record) + assert not within_tolerance(record) + + +def test_other_kinds_are_never_within_tolerance() -> None: + assert not within_tolerance(_record(kind="llm_judge", name="llm_judge.equivalence")) + assert not within_tolerance(_record(kind="", name="structural.length")) + + +def test_legacy_rows_fall_back_to_the_name_prefix() -> None: + legacy = _record(kind="", name="semantic.cosine") + assert is_semantic(legacy) + assert within_tolerance(legacy) + + +def test_a_renamed_semantic_evaluator_is_selected_by_kind() -> None: + assert within_tolerance(_record(kind="semantic", name="meaning")) + + +def test_a_malformed_category_list_is_not_a_regression() -> None: + record = _record().model_copy(update={"metadata": {"failure_categories": "SEMANTIC"}}) + assert not is_semantic_regression(record) + assert within_tolerance(record)