fix(semantic): strip code fences and canonicalize JSON before embedding

CHANGELOG.md

@@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- `semantic` no longer reads formatting as drift. Before embedding, both
+  outputs drop a code fence wrapping the whole output, and JSON objects and
+  arrays are re-serialized canonically (sorted keys). A source that fences
+  pretty-printed JSON against a target returning the same JSON compact used
+  to score ~0.73 and get flagged; it now scores 1.0. Semantic scores for
+  structured outputs change on upgrade — expect higher similarity where the
+  models only differed in layout.
+
 ## [1.4.0] - 2026-10-11
 
 ### Added

DOCS.md

@@ -513,6 +513,8 @@ When an evaluator's own measurement breaks (judge call fails, embedding call fai
 
 Embeds both outputs and scores the target by cosine similarity to the source (source score pinned at 1.0). Config: `embedding_model` (default `text-embedding-3-small`, an OpenAI model; the Gemini scaffold uses `gemini/gemini-embedding-001`, and the Anthropic and DeepSeek scaffolds borrow OpenAI's or Gemini's — see [Project setup](#project-setup)), `min_similarity` (default 0.9) — below it the pair is flagged `SEMANTIC_REGRESSION`. Within it, drift counts as *equivalent* for policy purposes. The yardstick is the source output, never the suite's `expected` field, so this measures drift rather than correctness: cosine distance can't tell "reworded" from "wrong", which is why the scaffold keeps it advisory (the library default is `blocking: true`; see [FAQ](docs/faq.md#why-does-a-hand-written-config-block-on-semantic-when-init-does-not)) and why correctness belongs to an `llm_judge` criterion.
 
+Presentation is stripped before embedding, so formatting never reads as drift: a code fence wrapping the **whole** output (` ```json … ``` `) is dropped, and an output that is a JSON object or array is re-serialized canonically (sorted keys, uniform separators). A fenced, pretty-printed object and the same object returned compact score 1.0; a fence inside prose is part of the answer and kept. An empty fenced block counts as an empty output. Whether the output is *valid, unfenced* JSON is a separate question — `json_schema` and the judge answer it.
+
 On an agent turn where **both** models answered with tool calls and no prose, there is nothing to embed — the evaluator writes **no record at all** (no provider call) rather than erroring on an empty embedding input or inventing a score. A turn where only *one* side is empty is still scored: a target that went silent where the source answered is exactly the regression this evaluator exists to catch. Since the empty side cannot be embedded (the provider 400s on empty input), the pair scores **0.0 similarity by definition** with no embedding call — still gated by `min_similarity` as usual — and the record carries `empty_side: "source" | "target"` metadata plus an explanation the report shows verbatim. The mirrored case (source silent, target answered) scores identically.
 
 ### LLM-as-judge (`evaluators.llm_judge`, list)

docs/evaluators.md

@@ -55,6 +55,16 @@ equivalent. That is why `evalshift init` ships semantic as advisory
 and why correctness belongs to an `llm_judge` criterion that names the
 property you care about.
 
+**Formatting is not drift.** Both outputs are normalized before
+embedding: a code fence wrapping the whole output (```` ```json … ``` ````)
+is dropped, and a JSON object or array is re-serialized canonically
+(sorted keys, uniform separators). A fenced, pretty-printed object and
+the same object returned compact score 1.0 — without this, layout alone
+pulled such pairs down to ~0.73 and flagged them. A fence inside prose is
+part of the answer and kept; an empty fenced block counts as an empty
+output. Whether an output is valid, unfenced JSON is a job for
+`structural.json_schema` or an `llm_judge` criterion.
+
 Use when:
 * You don't have a clean structural check.
 * You want to detect "wandered off" outputs that still look fine

llms-full.txt

@@ -623,6 +623,9 @@ evaluators:                         # every evaluator config also takes blocking
     # measures drift, not correctness (reworded-but-right reads as drift). Init
     # writes blocking: false for it; library default is true. Judge = correctness.
     # source score pinned 1.0; target = cosine(source, target) clamped [0,1]
+    # both outputs normalized before embedding: a code fence wrapping the WHOLE output is
+    #   dropped; a JSON object/array is re-serialized (sorted keys) -- fenced pretty JSON vs
+    #   the same JSON compact scores 1.0. Fences inside prose are kept. Empty fence = empty.
     # both outputs empty (tool-only turn) -> NO record at all (score returns None),
     #   no provider call. ONE empty side still scores: 0.0 similarity by definition,
     #   no embedding call (provider 400s on empty input); min_similarity gate applies;

src/evalshift_cli/evaluators/semantic.py

@@ -8,6 +8,12 @@
 This deliberately doesn't make symmetric "both halves same score" pairs
 because that would always produce delta=0 and defeat the regression
 detection.
+
+Both outputs are reduced to their meaning before embedding (see
+:func:`_meaning_text`): a code fence wrapping the whole output is dropped
+and JSON is re-serialized canonically. Presentation otherwise dominates
+the embedding — a fenced, pretty-printed object against the same object
+compact scores ~0.73, enough to flag every example of a run.
 """
 
 from __future__ import annotations
@@ -16,6 +22,7 @@
 import json
 import logging
 import math
+import re
 from typing import Any
 
 import litellm
@@ -30,6 +37,10 @@
 #: never collide with completion entries for the same model + text.
 _EMBEDDING_INPUTS: dict[str, Any] = {"kind": "embedding"}
 
+#: A markdown code fence wrapping the *entire* output (optional info
+#: string, e.g. ``json``). A fence inside prose is part of the answer.
+_WHOLE_FENCE = re.compile(r"\A\s*```[^\n`]*\n(.*?)\n?```\s*\Z", re.DOTALL)
+
 
 class CosineSimilarityEvaluator:
     """Cosine similarity between source and target output embeddings."""
@@ -73,8 +84,12 @@ async def score(
         target_output: str,
         history: list[dict[str, str]] | None = None,
     ) -> PairedScore | None:
-        source_silent = not source_output.strip()
-        target_silent = not target_output.strip()
+        # Everything below — the silence guards, the identical shortcut and
+        # the embeddings — sees meaning only, never fences or JSON layout.
+        source_output = _meaning_text(source_output)
+        target_output = _meaning_text(target_output)
+        source_silent = not source_output
+        target_silent = not target_output
         if source_silent and target_silent:
             # Tool-only turn: both models answered with calls, not prose.
             # There is nothing to embed, and an embedding provider will 400
@@ -200,6 +215,33 @@ def _cache_key(self, text: str) -> str:
         )
 
 
+def _meaning_text(text: str) -> str:
+    """Strip presentation from an output, keeping what it says.
+
+    Drops a code fence wrapping the whole output, then, if what remains is
+    a JSON object or array, re-serializes it with sorted keys and default
+    separators — layout and key order are not meaning. Anything else is
+    returned stripped of surrounding whitespace.
+
+    Args:
+        text: A raw model output.
+
+    Returns:
+        The text to embed; empty when the output carries no text.
+    """
+    stripped = text.strip()
+    fenced = _WHOLE_FENCE.match(stripped)
+    if fenced:
+        stripped = fenced.group(1).strip()
+    try:
+        decoded = json.loads(stripped)
+    except json.JSONDecodeError:
+        return stripped
+    if not isinstance(decoded, dict | list):
+        return stripped
+    return json.dumps(decoded, sort_keys=True, ensure_ascii=False)
+
+
 def _decode_embedding(payload: str) -> list[float] | None:
     """Decode a cached embedding, treating anything unreadable as a miss.
 

tests/unit/test_evaluators.py

@@ -521,6 +521,113 @@ async def boom(*_args: Any, **_kwargs: Any) -> Any:
             )
 
 
+class TestCosineEvaluatorFormatNormalization:
+    """Presentation (code fences, JSON layout) is not meaning.
+
+    One model fencing pretty-printed JSON where the other returns it
+    compact used to drag cosine from ~0.97 to ~0.73 — formatting drowned
+    the content signal and flagged every example of a run.
+    """
+
+    @staticmethod
+    def _recording_embed(
+        monkeypatch: pytest.MonkeyPatch,
+    ) -> list[str]:
+        embedded: list[str] = []
+
+        async def fake_embed(model: str, input: list[str]) -> Any:  # noqa: A002
+            embedded.append(input[0])
+            vec = [1.0, 0.0] if len(embedded) == 1 else [0.0, 1.0]
+            return type("R", (), {"data": [{"embedding": vec}]})()
+
+        monkeypatch.setattr(semantic_module.litellm, "aembedding", fake_embed)
+        return embedded
+
+    async def test_fenced_and_compact_json_with_same_content_score_identical(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        embedded = self._recording_embed(monkeypatch)
+        ev = CosineSimilarityEvaluator(embedding_model="x/y")
+        score = await ev.score(
+            prompt_id="p",
+            example_id="e",
+            input_vars={},
+            source_output='```json\n{\n  "b": 2,\n  "a": "x"\n}\n```',
+            target_output='{"a":"x","b":2}',
+        )
+        assert score is not None
+        assert score.target_score == pytest.approx(1.0)
+        assert embedded == ['{"a": "x", "b": 2}']
+
+    async def test_differing_json_is_embedded_in_canonical_form(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        embedded = self._recording_embed(monkeypatch)
+        ev = CosineSimilarityEvaluator(embedding_model="x/y")
+        await ev.score(
+            prompt_id="p",
+            example_id="e",
+            input_vars={},
+            source_output='```json\n{"summary": "café", "category": "bug"}\n```\n',
+            target_output='{"category":"billing","summary":"café"}',
+        )
+        assert sorted(embedded) == [
+            '{"category": "billing", "summary": "café"}',
+            '{"category": "bug", "summary": "café"}',
+        ]
+
+    async def test_fenced_non_json_has_only_the_fence_stripped(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        embedded = self._recording_embed(monkeypatch)
+        ev = CosineSimilarityEvaluator(embedding_model="x/y")
+        score = await ev.score(
+            prompt_id="p",
+            example_id="e",
+            input_vars={},
+            source_output="```python\ndef f():\n    return 1\n```",
+            target_output="def f():\n    return 1",
+        )
+        assert score is not None
+        assert score.target_score == pytest.approx(1.0)
+        assert embedded == ["def f():\n    return 1"]
+
+    async def test_prose_with_an_inner_fence_is_embedded_verbatim(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        # Only a fence wrapping the whole output is presentation; a fence
+        # inside prose is part of the answer.
+        embedded = self._recording_embed(monkeypatch)
+        ev = CosineSimilarityEvaluator(embedding_model="x/y")
+        source = "Run this:\n```bash\nls\n```"
+        await ev.score(
+            prompt_id="p",
+            example_id="e",
+            input_vars={},
+            source_output=source,
+            target_output="Run ls.",
+        )
+        assert source in embedded
+
+    async def test_empty_fence_counts_as_silent(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        # An empty fenced block carries no text; embedding "" would 400.
+        async def boom(*_args: Any, **_kwargs: Any) -> Any:
+            raise AssertionError("the provider must not be called")
+
+        monkeypatch.setattr(semantic_module.litellm, "aembedding", boom)
+        ev = CosineSimilarityEvaluator(embedding_model="x/y")
+        score = await ev.score(
+            prompt_id="p",
+            example_id="e",
+            input_vars={},
+            source_output='{"a": 1}',
+            target_output="```json\n```",
+        )
+        assert score is not None
+        assert score.target_score == 0.0
+        assert score.metadata["empty_side"] == "target"
+
+
 # ---------------------------------------------------------------------------
 # LLM judge
 # ---------------------------------------------------------------------------