Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

## [1.4.2] - 2026-10-12

### Fixed

- A `semantic` evaluator set to `blocking: true` no longer fails runs whose every row
Expand Down
2 changes: 1 addition & 1 deletion DOCS.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ hosted (opt-in) run history, diffs, PR gates

The suite is the crux, so the capture SDK is the recommended way to build one: it records real production runs to disk and `evalshift capture sync` promotes them into golden suites. Hand-written suites are fully supported — see [The golden suite](#the-golden-suite).

- **Package name:** `evalshift` · **CLI entry point:** `evalshift` · **version:** 1.4.1
- **Package name:** `evalshift` · **CLI entry point:** `evalshift` · **version:** 1.4.2
- **Python:** >= 3.11 · **License:** Apache-2.0 · **Status:** stable
- **Local-first.** Runs, scores, stats, and reports all happen on your machine under `.evalshift/`. The only network calls are the model API calls you asked for — and, if you opt in, pushes to the hosted service and reads from a capture bucket you own (`captures.store`).
- **Four pieces:** CLI (this doc), SDK, GitHub Action, hosted server — each with its own machine-readable reference for AI tools. See [Ecosystem and AI-tool references](#ecosystem-and-ai-tool-references).
Expand Down
15 changes: 14 additions & 1 deletion docs/evaluators.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,16 @@ part of the answer and kept; an empty fenced block counts as an empty
output. Whether an output is valid, unfenced JSON is a job for
`structural.json_schema` or an `llm_judge` criterion.

**Drift inside `min_similarity` is no change.** Because the source is
pinned at 1.0, any rewording is a small negative delta. A row that stays
within `min_similarity` therefore counts as *equivalent* in the policy
budgets and enters the paired significance test as a delta of 0 — nine
rows at 0.90–0.98 used to test as a `critical` regression and fail a run
whose every budget passed. Only flagged rows (`SEMANTIC_REGRESSION`) carry
their full drop into severity. `delta_avg_score` still shows the raw
drift, and the comparison notes how many rows were within the threshold.
See [methodology](methodology.md#drift-inside-min_similarity-is-tested-as-no-change).

Use when:
* You don't have a clean structural check.
* You want to detect "wandered off" outputs that still look fine
Expand Down Expand Up @@ -205,7 +215,10 @@ refusal changes scored separately.
You rarely wire the first two by hand: `evalshift capture sync` writes
`tool_selection` and `tool_arguments` into each suite's own `suites:` entry
based on what that suite's captures contain, so a tool-free suite gets no
tool evaluators at all. See
tool evaluators at all. A tool evaluator it inherits from the top level —
`trace_invariants` included — skips every pair that was offered no tools:
no row, no coverage entry, no effect on the verdict, and `evaluate` (or
`compare`, under its verdict) warns that it checked nothing. See
[Agent migrations](agents.md#configuration) and
[Configuration → per-suite evaluators](configuration.md#per-suite-evaluators).

Expand Down
2 changes: 1 addition & 1 deletion llms-full.txt
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# evalshift (CLI) — complete reference for AI tools

Canonical hosted copy: https://www.evalshift.dev/cli-llms-full.txt
Package: evalshift (PyPI) | CLI entry point: evalshift | version: 1.4.1
Package: evalshift (PyPI) | CLI entry point: evalshift | version: 1.4.2
Python: >=3.11 | license: Apache-2.0 | status: stable
Install: pip install evalshift (or: uv pip install evalshift)

Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ build-backend = "hatchling.build"

[project]
name = "evalshift"
version = "1.4.1"
version = "1.4.2"
description = "Run your prompts on two LLMs and find out, with statistical confidence, what regressed."
readme = "README.md"
license = "Apache-2.0"
Expand Down
Loading