feat(ci): working aimock drift detection - #846
Merged
Merged
Conversation
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…from drift
The aimock recorder emits { content: '' } with a 'fixture may be
incomplete' warning when it cannot parse tool-call deltas from a stream.
Observed live: both tool-calling entries in the first record run collapsed
to empty text while the @drift specs passed. A recorder artifact, not
model drift — reported in its own category so it cannot masquerade as a
kind change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
blove
force-pushed
the
blove/aimock-drift-detection
branch
from
August 29, 2026 19:16
d4b2258 to
f59af54
Compare
blove
enabled auto-merge (squash)
August 29, 2026 19:16
Contributor
blove
added a commit
that referenced
this pull request
Aug 29, 2026
* docs: design spec for the fixture-replay blog post Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: tighten the scale claim in the fixture-replay spec Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * docs: implementation plan for the fixture-replay post Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(website): scaffold the fixture-replay post * feat(website): write the fixture-replay post Body across all sections, plus two validation fixes: - Rewrote the transcript-settling sentence, which had 6 eight-grams of overlap with the subgraphs docs guide (same author, two days apart). - Restored Let's transitions and trimmed demonstrative tics. Dated 2026-08-29 (plan said 08-28; the date rolled and siblings date to their merge day). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(website): apply editorial review to the fixture-replay post - Remove a self-contradiction the thesis forbids: the conclusion claimed 'both have a plan' while the body says the drift check cannot see meaning drift. Only one of the two has a plan; say so. - Answer the obvious objection (replay the recorded chunk boundaries): faithful boundaries make the fence failure deterministic, not absent. - Defend the trade by displacement, not documentation — progressive rendering moved to the unit-level variance tables, which do test it. - Name the drift check as the second instance of the law, earning 'every' in the opening line. - Attribute affordability to replacing the model, not to the seam position. - Link the subgraphs post as the source of the worked example; 1 link -> 4. - Register: trim idiom, intensifiers, and doubled phrasing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(website): correct three false claims in the fixture-replay post Spec-compliance review found the post overstated its own thesis. All verified in source: - chunkSize 4096 is a DEFAULT, not a law. A second harness serves the two example apps, and its comment says targeted streaming regressions opt into smaller per-fixture chunks — those fixtures set 3/4/6/23/36 with 25-750ms latency, and there are e2e tests sampling the mid-stream DOM. 'We deleted time' was true of the cockpit tier only. Reframed around which tier opted back in, which is a better question anyway. - 'spawns a real langgraph dev subprocess' is false for 8 of 34 apps; the ag-ui setup spawns uvicorn. The seam generalizes, the server doesn't. - The drift check does not run at all: it shells out to , which is not a declared option under strict parseArgs, so it exits before comparing anything. The post said it re-records and compares. Now says what is true. - Quote the full source comment rather than half of it; the omitted lines were what the following paragraphs presented as original analysis. - Drop the 60ms/56-samples figures — real, but from an ad-hoc session and unverifiable from the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * feat(website): drift section now describes the shipped solution The check went from 'has never run' to merged and proven (#846) between drafts. §5 now carries the real design: a @drift-tagged contract-only subset of the e2e suite runs weekly against the live provider through the record proxy — the assertions are the drift check, so meaning drift is caught by construction. Includes the first run's finding: the differ reported tool-call drift that was actually the recorder failing to parse tool-call deltas — the instrument's own blind spot, now its own category. Conclusion updated to match: both gaps covered by the same move, widening is tagging specs, not new design. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(website): apply re-review findings to the drift section - Keystone stands alone as its own paragraph; third restatement cut. - Anecdote no longer announces itself; actor fixed (the recorder warns, not the check); adds that it has run clean against the live model since. - Conclusion scoped honestly: widening drift = tagging specs plus porting record mode to the shared harness; widening streams = per-fixture chunk opt-in. Both reviews independently flagged the original overclaim. - Tagging rule examples now match what the specs actually assert (prompt-derived patterns, not shape-only). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * style(website): pull the fixture-replay post back to Brian's register - Soften first-person discovery narration to impersonal constructions. The events are real, but they were my discoveries narrated under Brian's byline; the claims now belong to the work, not a narrator. - Restore the flagged opinion on the streaming trade ('I think that's the right trade') — the post's strongest judgment was reading as fact, and siblings carry 1-2 flags. - Fix bold in body prose (italics-only convention). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The drift check had never run.
drift.tsshelled out tollmock --record ... --out, and--outis not a declared CLI option (strictparseArgs; the set isport, host, fixtures, latency, watch, metrics, record, strict), so every invocation died withERR_PARSE_ARGS_UNKNOWN_OPTIONbefore comparing anything. Two deeper problems sat underneath: a fixture entry is onlymatch+response, so it cannot be "re-recorded" in isolation — the request is built by the graph at runtime; and byte-size comparison cannot see meaning drift.Design: docs/superpowers/specs/2026-08-29-aimock-drift-detection-design.md
The two-stage replacement
Stage 1 — the gate. Three contract-only specs tagged
@drift(non-empty reply, research dispatch surfaces a subagent card, interrupt panel renders) run against the live provider through aimock's record-proxy —LLMocksupportsrecord: { providers, fixturePath }natively, no CLI. The assertions are the semantic contract: if the graph's prompts stop eliciting the tool call, a spec fails. Content-asserting specs (canned text, exact counts) are deliberately untagged — they fail live regardless of drift.Stage 2 — the diagnostic. A rewritten differ pairs the run's recordings against committed fixtures by match discriminator and reports tool-name sets, response kind, and a coarse length bucket. Always exit 0 — the specs are the gate; this explains a red run.
Workflow: weekly cron (Mon 09:00 UTC) +
workflow_dispatch, recordings uploaded as an artifact on every run, issue opened on failure with the diff in the body (and the trigger named, fixing the old always-says-scheduled text). Advisory only — never a merge gate. Committed fixtures are never modified.Found while building it
The first record run reported
toolCalls -> texton both tool-calling entries — while the specs proving those tools fired were green. The aimock recorder's stream-collapse couldn't parse our client's tool-call deltas and saved{ content: "" }with its own "fixture may be incomplete" warning. The differ now reports these in a dedicatedincompleteRecordingscategory so a recorder artifact can't masquerade as model drift. (Recordings do preserve real per-chunk timings —interChunkDelaysMs— which may be useful later.)Acceptance (spec requirement: one green run + one artifact before merge)
changed=0 incompleteRecordings=2 unmatchedCommitted=15 unmatchedRecorded=0(unmatched-committed are fixtures for untagged specs — a report, not a failure).aimock-recordingsartifact (5,080 bytes), identical differ output.npx tsx --test e2e/scripts/drift-lib.test.ts).@driftsubset passes in replay (3/3); default mode unchanged.🤖 Generated with Claude Code