Repository navigation
fix(semantic): strip code fences and canonicalize JSON before embedding - #48
Merged
Merged
Conversation
A source that fenced pretty-printed JSON against a target returning the same JSON compact scored ~0.73 cosine and was flagged as semantic drift: layout dominated the embedding. Both outputs now drop a code fence wrapping the whole output and re-serialize JSON objects/arrays with sorted keys before embedding; fences inside prose are kept, and an empty fenced block counts as an empty output. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
semantic.cosineembedded raw output text, so presentation outweighed meaning. In a real run (r_20261011_triage_9fcdf0), 8/9 source outputs were```json-fenced pretty-printed JSON and every target was compact JSON: every example scored 0.73–0.79 and was flaggedSEMANTIC_REGRESSION. The one unfenced source scored highest (0.89) even though itscategoryactually changed.Fix
Before embedding, both outputs go through
_meaning_text():"").Always on, no config. Fence/validity checks stay with
json_schemaand the judge.Evidence (live
text-embedding-3-small, same run)0/9 flagged after the fix.
Tests / docs
TestCosineEvaluatorFormatNormalization(4 red before the fix; 1 guards that inner fences are kept).make ci: 2999 passed, 94.91% coverage;pre-commit run --all-filesclean.DOCS.md,llms-full.txt,docs/evaluators.md,CHANGELOG.md(Unreleased → Fixed, notes scores rise on upgrade).🤖 Generated with Claude Code