fix: semantic drift within min_similarity no longer fails runs; tool evaluators skip tool-free suites

CHANGELOG.md

@@ -7,6 +7,30 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- A `semantic` evaluator set to `blocking: true` no longer fails runs whose every row
+  stays within `min_similarity`. The source is pinned at 1.0, so any rewording is a
+  small negative delta, and the paired significance test ran on those raw deltas:
+  nine rows at 0.90–0.98 tested as a `critical` regression and failed a run whose every
+  budget passed. Rows within `min_similarity` now enter the test as no change — the
+  rule the budgets already used — and the comparison notes how many. `delta_avg_score`
+  and the aggregates still show the raw drift. Re-analysing an older run can lower its
+  semantic severities.
+- A fail caused by comparison severity alone now carries a `reason` naming each
+  comparison, so the report no longer shows "7 of 7 budgets within policy" under FAIL.
+  The templated recommendation names the actual cause instead of always claiming a
+  budget breach — a breached budget, including a slice's own (`the 'security' slice
+  overall regression rate budget was breached`), or else the overall blocking
+  regressions — and the summary's critical count reads the blocking comparisons.
+- Tool evaluators (including a top-level `trace_invariants` inherited by every suite) no
+  longer attempt pairs that were offered no tools. On a tool-free suite they used to
+  book every pair as unmeasured, demote a clean `pass` to `conditional_pass`, and blame
+  "tool calls and no prose". They now skip those pairs, and `evaluate` (and `compare`,
+  under its verdict) warns that the evaluator checked nothing because no example offered
+  tools. Coverage is written by `evaluate`, so an older run needs `evaluate` re-run to
+  drop the stale coverage; `analyze` alone keeps it, and the `conditional_pass` with it.
+
 ## [1.4.1] - 2026-10-11
 
 ### Fixed

DOCS.md

@@ -378,7 +378,7 @@ Slices holding exactly the same examples are collapsed to one before any test ru
 
 ### Per-suite evaluators
 
-A project's suites are rarely homogeneous — one calls tools, six answer in prose — and one top-level `evaluators:` block either leaves the tool-calling suite unmeasured or hands the tool-free ones a tool evaluator with an empty denominator, which the policy reads as an *inconclusive* gate rather than as "not applicable here". So a `suites:` entry can carry its own evaluator block, and `capture sync` generates one per suite from what that suite's rows actually contain:
+A project's suites are rarely homogeneous — one calls tools, six answer in prose — and one top-level `evaluators:` block either leaves the tool-calling suite unmeasured or hands the tool-free ones tool evaluators with nothing to check (they skip every pair that was offered no tools, and `evaluate` — or `compare`, under its verdict — warns that they checked nothing). So a `suites:` entry can carry its own evaluator block, and `capture sync` generates one per suite from what that suite's rows actually contain:
 
 ```yaml
 # >>> evalshift suites (managed by `evalshift capture sync`) >>>
@@ -511,7 +511,7 @@ When an evaluator's own measurement breaks (judge call fails, embedding call fai
 
 ### Semantic (`evaluators.semantic`, single block)
 
-Embeds both outputs and scores the target by cosine similarity to the source (source score pinned at 1.0). Config: `embedding_model` (default `text-embedding-3-small`, an OpenAI model; the Gemini scaffold uses `gemini/gemini-embedding-001`, and the Anthropic and DeepSeek scaffolds borrow OpenAI's or Gemini's — see [Project setup](#project-setup)), `min_similarity` (default 0.9) — below it the pair is flagged `SEMANTIC_REGRESSION`. Within it, drift counts as *equivalent* for policy purposes. The yardstick is the source output, never the suite's `expected` field, so this measures drift rather than correctness: cosine distance can't tell "reworded" from "wrong", which is why the scaffold keeps it advisory (the library default is `blocking: true`; see [FAQ](docs/faq.md#why-does-a-hand-written-config-block-on-semantic-when-init-does-not)) and why correctness belongs to an `llm_judge` criterion.
+Embeds both outputs and scores the target by cosine similarity to the source (source score pinned at 1.0). Config: `embedding_model` (default `text-embedding-3-small`, an OpenAI model; the Gemini scaffold uses `gemini/gemini-embedding-001`, and the Anthropic and DeepSeek scaffolds borrow OpenAI's or Gemini's — see [Project setup](#project-setup)), `min_similarity` (default 0.9) — below it the pair is flagged `SEMANTIC_REGRESSION`. Within it, drift counts as *equivalent* for policy purposes and as *no change* in the paired test that sets severity — see [methodology](docs/methodology.md#drift-inside-min_similarity-is-tested-as-no-change). The yardstick is the source output, never the suite's `expected` field, so this measures drift rather than correctness: cosine distance can't tell "reworded" from "wrong", which is why the scaffold keeps it advisory (the library default is `blocking: true`; see [FAQ](docs/faq.md#why-does-a-hand-written-config-block-on-semantic-when-init-does-not)) and why correctness belongs to an `llm_judge` criterion.
 
 Presentation is stripped before embedding, so formatting never reads as drift: a code fence wrapping the **whole** output (` ```json … ``` `) is dropped, and an output that is a JSON object or array is re-serialized canonically (sorted keys, uniform separators). A fenced, pretty-printed object and the same object returned compact score 1.0; a fence inside prose is part of the answer and kept. An empty fenced block counts as an empty output. Whether the output is *valid, unfenced* JSON is a separate question — `json_schema` and the judge answer it.
 
@@ -701,7 +701,7 @@ Trace-aware debugging: `evalshift diff case <run> <example>` (side-by-side trace
 
 Full contract in [docs/methodology.md](docs/methodology.md). Per `(prompt, evaluator, slice)` comparison over the paired per-example deltas:
 
-1. **Sample guards.** A pair an evaluator measured nothing on (a tool-only turn, an example with no ground truth) has no row in `scores.jsonl` at all, so it is absent from `n`, from the slice aggregates and from the policy metrics by construction. The count is reconstructed from `state.json`'s `evaluator_coverage` and noted, so "8 of 10 rows not applicable" survives the rows themselves. n < 5 → skipped, severity `insufficient`. 5 ≤ n < 20 → tested, but flagged uncertain. Zero-variance deltas → skipped, severity `none`. **No applicable rows left** → severity `insufficient` with a `nothing measured:` note, never `none`; a run whose blocking evaluators all landed there cannot return `pass` — it is downgraded to `conditional_pass` (or stays `inconclusive`) with those evaluators named in `recommendations`, since an evaluator that compared nothing is unknown, not equivalent. Advisory (`blocking: false`) evaluators are exempt — their silence gates nothing by design, so they are neither named nor demote the verdict: `evaluator_coverage` carries each axis's `blocking` flag, and the synthesized comparison gets an `advisory:` note (zero rows leave the policy layer nothing else to read the flag from). The HTML report gives that row its own headline — **Nothing measured**, not the **Not enough data** the rest of `insufficient` gets.
+1. **Sample guards.** A pair an evaluator measured nothing on (a tool-only turn, an example with no ground truth) has no row in `scores.jsonl` at all, so it is absent from `n`, from the slice aggregates and from the policy metrics by construction. The count is reconstructed from `state.json`'s `evaluator_coverage` and noted, so "8 of 10 rows not applicable" survives the rows themselves. n < 5 → skipped, severity `insufficient`. 5 ≤ n < 20 → tested, but flagged uncertain. Zero-variance deltas → skipped, severity `none`. A semantic row within `min_similarity` enters the test as a delta of 0 (noted `k of n rows within the evaluator's own threshold (min_similarity) — tested as no change`); `delta_avg_score` stays the raw mean. **No applicable rows left** → severity `insufficient` with a `nothing measured:` note, never `none`; a run whose blocking evaluators all landed there cannot return `pass` — it is downgraded to `conditional_pass` (or stays `inconclusive`) with those evaluators named in `recommendations`, since an evaluator that compared nothing is unknown, not equivalent. Advisory (`blocking: false`) evaluators are exempt — their silence gates nothing by design, so they are neither named nor demote the verdict: `evaluator_coverage` carries each axis's `blocking` flag, and the synthesized comparison gets an `advisory:` note (zero rows leave the policy layer nothing else to read the flag from). The HTML report gives that row its own headline — **Nothing measured**, not the **Not enough data** the rest of `insufficient` gets. A tool evaluator is never handed a pair that was offered no tools at all — like an `applies_to` miss, that pair is not attempted: no cell, no coverage entry, so it counts toward neither the "not applicable" note nor "nothing measured".
 2. **Normality screen.** Shapiro–Wilk on the deltas at α = 0.05: normal → **paired t-test**, non-normal → **Wilcoxon signed-rank**. n > 5000 skips the screen (CLT, t-test).
 3. **Effect size.** Paired Cohen's d = mean(Δ)/std(Δ), with a 95% CI — analytical for the t-test, percentile bootstrap (2,000 resamples, seeded, deterministic) for Wilcoxon.
 4. **Multiple comparisons.** Benjamini–Hochberg FDR correction at α = 0.05 across *all* testable comparisons in the run (equivalent to `statsmodels fdr_bh`).
@@ -756,10 +756,10 @@ How the verdict is computed:
 - **A shared ground-truth miss is not equivalence.** A `tool_selection.conformance` row grades each side *absolutely* against the ground truth the suite recorded, so both sides can miss it at the same height — `0.0 / 0.0` on an example both models answered with a tool call the recording never made. The delta is zero, which used to read as equivalence and is the whole of the `equivalent_rate: 1.0` a real run once shipped over a suite where nine pairs in ten routed differently. Those rows now leave **every** policy rate: they measure the harness (wrong toolset attached, wrong prompt, a suite promoted from a different agent), not the migration. They are still reported — as a `TOOL_GROUND_TRUTH_MISS` count in `failure_categories` and as a recommendations line naming how many were excluded. Only the *shared-height* case is dropped: a conformance row the target lost ground on (`0.8 / 0.3`) is still a regression and one it improved on (`0.2 / 0.6`) is still an improvement. A run whose every blocking row was a shared miss is `inconclusive`, and its recommendation is **Fix the eval harness before collecting more examples** — more pairs from the same setup are more excluded rows, so the denominator stays empty however many are added. The exclusion is deliberately narrower than the diagnosis, which is why `evaluate` runs its own broken-harness check over the **source side alone**: a suite the source fails and the target happens to satisfy has a *positive* delta and no `TOOL_GROUND_TRUTH_MISS` tag, so the exclusion rule cannot see it and it is exactly as misconfigured.
 - **A cost/latency ratio measured only from zeroes says so.** When the calls exist but every `cost_usd` (or `latency_ms`) is `0` on both models, `analyze` adds a recommendations line beside the `conclusive: false`: `The cost increase budget could not be measured: all 4 error-free calls across both models recorded a cost of 0, so its observed 0.00 is a default, not a measurement.` A run with no calls gets no line — the empty `raw.jsonl` already explains itself. Emitted once per run, since every scope reads the same calls.
 - **Every budget reports its own `denominator`** — the sample `observed` was computed over. Scored records for the regression rate, the equivalence rate and the critical count — counted over *measurements*, so an evaluator scoring two axes contributes two rows per example; `tool_arguments` rows for tool drift; `tool_selection.divergence` rows for tool divergence; the distinct examples with a blocking `trace_invariants` row for the trace-rule count; the error-free calls behind both averages for the cost and latency ratios. Slices report their own counts. `0` means "counted, and the sample was empty", so `observed` is a default; a *missing* `denominator` means "no sample size reported" and is **not** zero — only bundles written before the field say that, and the hosted gate falls back to `conclusive` for them. It is the same number the `1/n` granularity warning is judged on, and it is orthogonal to `conclusive`: an all-zero cost ratio counted every call it averaged and still measured nothing, so it reports a positive denominator beside `conclusive: false`. The hosted gate derives its own Wilson interval from these denominators, over the same three rate budgets and with the same confidence constant the CLI uses, so a local verdict and a hosted one now agree on whether a breach was confirmed — see [methodology](docs/methodology.md).
-- Any conclusive budget failure, or any blocking critical/high-severity comparison → **`fail`**. Any lower-severity blocking regression → **`conditional_pass`**. Otherwise → **`pass`**.
+- Any conclusive budget failure, or any blocking critical/high-severity comparison → **`fail`**. Any lower-severity blocking regression → **`conditional_pass`**. Otherwise → **`pass`**. When severity alone fails the run, `reason` names each blocking comparison.
 - **`migration_policy` is the single source of truth for these budgets.** `migration_decision.json`'s `policy` field is the resolved policy `analyze` computed the verdict under — every top-level budget with its default applied, plus `slices`. `bundle`/`push` do not read `migration_decision.json`; `bundle` re-resolves the verdict from `evalshift.yaml` at bundle time, so the bundle's `decision.policy` (see [Hosted EvalShift](#hosted-evalshift)) is the policy the config held when the bundle was built — editing `migration_policy` between `analyze` and `bundle` changes what the bundle carries. `push <run-id>` builds a bundle only when `run_bundle.json.gz` is missing — an existing bundle is uploaded as-is — so re-run `evalshift bundle <run-id>` after a config edit. Either way, that is what lets the hosted gate check a pull request against exactly the budgets the bundle's own verdict used, instead of a separate, web-edited policy. `null` when no `migration_policy` is configured, and on a `migration_decision.json` written before this field existed. `evalshift.yaml` is the source of truth for that policy, and the web app's project policy view is becoming a read-only display of the snapshot each run pushed. Because every budget is resolved, the snapshot carries `max_invariant_violations` (default `0`) even for a config with no trace rules; hosted EvalShift accepts and stores it (since 2026-10-08) but never re-evaluates it — a run that carries its own policy is answered with the CLI's decision, which did check it, and only runs pushed without a policy fall back to the server's six-budget re-check.
 - **A slice budget gates the run exactly like an overall one.** The budgets under `migration_policy.slices` are evaluated on the same terms as the top-level ones: a conclusively breached slice budget **fails** the run, and an unconfirmed breach makes it `inconclusive` — the same Wilson rule, counted over that slice's own denominator. Since the overall rows can all be green in a run a slice budget fails, `recommendations` names the one that blocked (`The 'security' slice breached its overall regression rate budget (the share of scored comparisons where the target did worse): 20% over n=20 vs the 0% limit.`) and the `inconclusive` `reason` scope-qualifies it the same way. Per-slice verdicts under `slices[*].verdict` are unchanged, and a slice that fails on *comparison severity* rather than a budget still only downgrades an overall `pass` to `conditional_pass`.
-- Semantic drift that stays above `min_similarity` counts as equivalent, not regression.
+- Semantic drift that stays above `min_similarity` counts as equivalent, not regression — in the budgets and in the paired test.
 
 CI wiring (on `analyze` and `compare`):
 

docs/agents.md

@@ -117,10 +117,10 @@ suite's rows actually contain and writes that suite's own evaluator block —
 `tool_selection` when any row was offered a toolset, `tool_arguments` when any
 row recorded call arguments, and nothing at all for a suite whose captures never
 called a tool. That last case is the point: a tool evaluator pointed at a
-tool-free suite scores an empty denominator, which the policy reads as an
-*inconclusive* gate rather than as "not applicable here". A family a suite
-declares replaces the top-level one wholesale; families it does not mention are
-inherited. See
+tool-free suite has nothing to check: it skips every pair that was offered no
+tools — no row, no coverage entry, no effect on the verdict — and `evaluate`
+(or `compare`, under its verdict) warns that it checked nothing. A family a suite declares replaces the top-level
+one wholesale; families it does not mention are inherited. See
 [Configuration → per-suite evaluators](configuration.md#per-suite-evaluators).
 
 Hand edits inside the markers are regenerated away on the next sync. To keep

docs/configuration.md

@@ -239,10 +239,12 @@ changes which rows the tool returns. Lower it to `0.4` if you want presence
 differences ignored.
 
 Semantic-evaluator drift counts toward `max_overall_regression_rate` and
-`min_equivalence_rate` only when it breaches
+`min_equivalence_rate` — and toward the paired test that sets each comparison's
+severity — only when it breaches
 [`evaluators.semantic.min_similarity`](#evaluatorssemantic). A near-identical
 output (cosine ~0.98) that stays within `min_similarity` is treated as
-*equivalent*, not a regression — so the policy gate and the report agree.
+*equivalent*, not a regression, by the budgets and the test alike — so the
+policy gate, the severity and the report agree.
 
 Per-slice overrides live under `migration_policy.slices` — a map of
 slice name (an example tag: every distinct tag is a slice) to a partial
@@ -953,9 +955,9 @@ only when `suites:` is empty.
 
 A project's suites are rarely homogeneous: one calls tools, six answer in prose.
 One top-level `evaluators:` block either leaves the tool-calling suite unmeasured
-or hands the tool-free ones a tool evaluator with an empty denominator, which
-reads as an *inconclusive* gate rather than as "not applicable here". So a suite
-can carry its own block:
+or hands the tool-free ones tool evaluators with nothing to check — they skip
+every pair that was offered no tools (no row, no coverage entry, and `evaluate` —
+or `compare`, under its verdict — warns that they checked nothing). So a suite can carry its own block:
 
 ```yaml
 suites:
@@ -1135,7 +1137,7 @@ Each suite's entry is generated from what that suite's own rows contain, tool
 evaluators included, so nothing has to be wired by hand:
 
 - No row was offered a toolset → no `evaluators:` block at all; the suite
-  inherits the top level, and no tool evaluator scores an empty denominator.
+  inherits the top level; a tool evaluator it inherits skips its tool-free pairs.
 - Any row was offered a toolset → `tool_selection: [{name: routing, conformance:
   expected, divergence: set}]`.
 - Any row recorded tool-call arguments → `tool_arguments: [{name: routing_args,