Repository navigation
Calibration test: pin the injected system prompt and the cache-write term; say the fix waits for anima-research/context-manager#324 - #2
Merged
Anarchid merged 1 commit intoOct 23, 2026
Conversation
… term in the sample; say the fix waits for context-manager anima-research#324 From our internal review of anima-research#763. - The test's request carried only the agent's configured system prompt, so a sample built from agent.systemPrompt instead of the request's system passed it, and so did a total that dropped cacheCreationTokens (both run against the test as it was). Its module now adds a system-position injection, which the request's system holds and the sample must carry, and its usage event splits the 4900 into input, cache write and cache read. Each of the two variants now fails it. - The changelog fragment says that context-manager 0.20.0 is among the versions that ignore the second argument, so the multiplier learns as before until the host runs a release with context-manager anima-research#324. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
From our internal review of anima-research#763, two things its test lets through. Each was run against the test as it is at
0a0c71f1:agent.systemPromptinstead ofcompiledRequest.systempasses, because the test's request carries only the agent's configured prompt. A module'ssysteminjection would then go out in the request and be left out of the sample.cacheCreationTokenspasses, because the usage event has no cache write.The changelog fragment also reads as if the fix applies on install. With context-manager 0.20.0, which
^0.20.0installs today, the second argument is ignored: 0.20.0'sreportRealInputTokens(realTotal)takes only the total.Changes
system-position injection throughgatherContext. The test asserts the request's system holds both the agent's prompt and the injection, and the existingdeepEqualthen pins the sample to it.What each part is for
Two assertions, one fixture field and one fragment clause. Each test change pins a claim the change makes, that the sample carries the request's own system prompt and its whole input total.
Tests
npm install:test/calibration-request-overhead.test.tspasses at37ee0354.src/framework.ts:20444and:20392, each fail it. Both pass the test as it was at0a0c71f1.Not verified
Written by Esther-2584 (Claude Opus 5.5), from our internal review of anima-research#763. Meg-2646 (Claude Opus 5.5) read the change.
Description checked and posted by Lottie-2661 (Claude Opus 5.5).
🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared
nissa-seruaccount.