Skip to content

fix(semantic): strip code fences and canonicalize JSON before embedding - #48

Merged
babaliauskas merged 1 commit into
mainfrom
fix/semantic-format-normalization
Oct 11, 2026
Merged

babaliauskas merged 1 commit into
mainfrom
fix/semantic-format-normalization

Conversation

@babaliauskas

Copy link
Copy Markdown
Collaborator

Problem

semantic.cosine embedded raw output text, so presentation outweighed meaning. In a real run (r_20261011_triage_9fcdf0), 8/9 source outputs were ```json-fenced pretty-printed JSON and every target was compact JSON: every example scored 0.73–0.79 and was flagged SEMANTIC_REGRESSION. The one unfenced source scored highest (0.89) even though its category actually changed.

Fix

Before embedding, both outputs go through _meaning_text():

  • drop a code fence wrapping the whole output (fences inside prose are part of the answer and kept);
  • re-serialize a JSON object/array canonically (sorted keys, default separators);
  • an empty fenced block counts as an empty output (avoids embedding "").

Always on, no config. Fence/validity checks stay with json_schema and the judge.

Evidence (live text-embedding-3-small, same run)

example before after
cap_c6ebf63e 0.733 0.966
cap_296ecd48 0.749 0.967
cap_8cd2b4c9 0.744 0.971
cap_c2cc1743 (real category change) 0.890 (highest) 0.908 (lowest)

0/9 flagged after the fix.

Tests / docs

  • 5 new tests in TestCosineEvaluatorFormatNormalization (4 red before the fix; 1 guards that inner fences are kept).
  • make ci: 2999 passed, 94.91% coverage; pre-commit run --all-files clean.
  • DOCS.md, llms-full.txt, docs/evaluators.md, CHANGELOG.md (Unreleased → Fixed, notes scores rise on upgrade).

🤖 Generated with Claude Code

A source that fenced pretty-printed JSON against a target returning the
same JSON compact scored ~0.73 cosine and was flagged as semantic drift:
layout dominated the embedding. Both outputs now drop a code fence
wrapping the whole output and re-serialize JSON objects/arrays with
sorted keys before embedding; fences inside prose are kept, and an empty
fenced block counts as an empty output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas
babaliauskas merged commit 8415902 into main Oct 11, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant