Repository navigation
test(calibration): pin six calibration behaviours no test reached - #430
Open
nissa-seru wants to merge 1 commit into
Open
nissa-seru wants to merge 1 commit into
nissa-seru wants to merge 1 commit into
Conversation
Six tests, each failing when one line of the calibration code is broken, where every existing test passed: - calibration-band, "a ratio inside the band is learned from, though the multiplier it implies is past the ceiling": a learned multiplier of 1.2 and a sample at 1.6 times the compile's estimate. The ratio is inside the band (up to 1.8) and the multiplier it implies, 1.92, is past the ceiling, so the EMA steps toward it. With the band's top at 1.5 (src/strategies/autobiographical.ts:14643) the sample is set aside and the multiplier stays 1.2. The sample doesn't declare its overhead: on an unmarked learned record the first sample that does is held for the next (:14683-14695). - calibration-band, "a sample on a branch the strategy has not loaded is refused": a fork and a compile on it, then the store moved back to the first branch with JsStore.switchBranch, without the manager. reportRealInputTokens must throw. It doesn't without its requireLoadedBranch call (:14599). - derive, "a store its parent's messages can't seed loads cold, at its parent's calibration multiplier": the parent, at 1.3, writes after the branch is made, so deriveOnto loads the branch cold. The derived store holds only the message from before the fork and estimates at 1.3. It estimates at 1 when the cold path doesn't copy the multiplier (src/message-store.ts:467). - view-composition, "reports the primary view's calibration multiplier": a merged view over a primary at 1.3 and an auxiliary at 1.1 reports 1.3. It reports 1 when the merged view's getter returns 1 (src/message-view.ts:161). - fold-journal, "prices a receipt's rendered tokens at the store's calibration multiplier": a scripted strategy sets the store's multiplier in its select. After a baseline round at 1, a fold at 1.5 gives renderedTokens.before as the baseline's after times 1.5, rounded. It gives the baseline's after unscaled when calibrated() drops the multiplier (src/fold-journal.ts:1202). - l1-recall-raw-pins-frontier, "the request's system prompt and tools are priced at the store's multiplier, as its messages are": the pins-at-the-limit run with the store at 1.5, set by one provider sample replacing the provisional 1. At exactly the longer retry's price at 1.5 the pins are kept, and one token less they're recalled. When the gate prices the system prompt and tools without the multiplier (src/strategies/autobiographical.ts:9722-9723), one token less keeps them. The pins test shares the existing pins-at-the-limit test's setup, now pinsAtTheLimit, whose assertions are unchanged. It builds the 240-message store once and copies it for each case, as forest() already does for the plain store, through the copy step forest() used, now openCopy. Every case's request and retry prices are the same as when each case built its own store. The two comments that described that setup are gone. What they said: - the pinned chunk carries an image and two turns of one speaker, which the request collapses into one with a separator; - a second opened chunk starts on the agent's turn after a recall pair, so the Context Manager takes a turn between them; - the host serves a system prompt and tools, so every term of the price is in play; - with `retried`, the summarizer's first answer is a tool call, which brings the no-tools retry, and its second is wrapped whole in <thinking>, which brings the plain-prose retry; - a retry appends a line to the instruction, so a retry of the admitted request must fit too.
This was referenced Nov 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds six tests, and changes nothing under
src/. Each pins a part of the token-estimate calibration that no test pinned before. The calibration is the multiplier context-manager learns from the provider's real input-token counts and applies to its own estimates. The six parts are which samples the multiplier learns from, the refusal of a sample for a branch the strategy hasn't loaded, the multiplier a cold-derived store reports, the multiplier a merged view reports, a fold receipt's rendered tokens, and the price of the system prompt and tools when raw pins join a compression request. Each test passes atmain's code and fails when the line that keeps its behaviour is broken.@Anarchid, @slimepriestess and @antra-tess, we're asking one of you to review this before it's merged. It adds tests alone, so nothing a resident sees changes.
Problem
At
fb4cf1046, we broke each of the six lines in the table under Tests in turn. For each, we ran every test file whose run executes that line, found by statement coverage of each test file run alone atfb4cf1046, together with the 11 test files that readsrc/as text. That was between 11 and 131 files for each line, and every file passed.reportRealInputTokensrefuses a sample while the strategy hasn't loaded the store's current branch. It callsrequireLoadedBranch(src/strategies/autobiographical.ts:14599), which fails closed "rather than using mirrors loaded from another generation" (:5134).:14643–:14644,:14656).MessageStore.deriveOntofalls back to a cold load when its seed doesn't match the branch (src/message-store.ts:450–:453), and the cold store starts at this store's multiplier (:467).mergeMessageStoreViews, "Token estimation and calibration delegate to the primary view" (src/message-view.ts:103–:104,:160–:161).renderedTokensat its own layout's multiplier.afteris that layout's tokens, and a fold'sbeforeis the previous layout's tokens, both at that multiplier (src/fold-journal.ts:1201–:1203, used at:1006and:1024–:1025).compressionContextBudgetTokenswith its longest retry line (src/strategies/autobiographical.ts:10019–:10031). The request's system prompt and tools are "Priced as the store prices a message" (:9719–:9723), so at the store's multiplier. The gate's existing test runs at a multiplier of 1.Changes
Six tests in five files, +143 −44:
test/calibration-band.test.ts:150). The test writes a learned multiplier of 1.2 into the store's calibration record. It then reports a sample at 1.6 times the compile's estimate, whose implied multiplier is 1.92. The multiplier the next compile reports must be above 1.2 and below 1.8. The sample doesn't declare its overhead, because on a record like this one the strategy holds the first sample that does until the next one arrives (src/strategies/autobiographical.ts:14683–:14695).:166). The test forks a branch and compiles on it. Then it moves the store back to the first branch with chronicle'sJsStore.switchBranch, without the manager, andreportRealInputTokensmust throw the refusal.test/derive.test.ts:890). The parent, at a multiplier of 1.3, writes a message after the branch is made, soderiveOntoloads the branch cold. The test checks that the derived store holds only the message from before the fork, then that it estimates at 1.3.test/view-composition.test.ts:118). A merged view over a primary at 1.3 and an auxiliary at 1.1 must report 1.3.test/fold-journal.test.ts:287). A scripted strategy sets the store's multiplier in itsselect. The test accepts a baseline round at 1, then a fold at 1.5, withacceptRound. The fold's receipt must giverenderedTokens.beforeas the baseline'saftertimes 1.5, rounded.test/l1-recall-raw-pins-frontier.test.ts:875). This runs the existing pins-at-the-limit test again with the store at 1.5. One provider sample sets the 1.5, replacing the provisional 1 that a store without a calibration record starts at. With the budget at exactly the longest retry's price at 1.5, the pins stay, and with one token less they're recalled. The existing test's setup moved intopinsAtTheLimit(:211), a helper both tests call. It builds the store once and copies it for each case, as the file already does for its plain store, through the copy step thatforestused, nowopenCopy(:130). Every case's prices are the same as when each case built its own store. The existing test's assertions are unchanged, and the two comments that described its setup moved to the commit message.All 18 open pull requests merge with this head as they do with
main(fb4cf1046), as of 06:40 UTC on October 31. 13 merge cleanly. Five, #94, #130, #140, #310 and #315, conflict in the same files both ways, and none of those files is one this changes.Tests
We ran these from a clean checkout with this head's files,
4f7dc86a, on Linux 6.12 x86_64 with Node 24.22.2 and npm 11.19.1, afternpm cifrom the lockfile, which pins@animalabs/membrane0.9.0,@animalabs/chronicle0.5.1 andtypescript7.0.2.npx tscexits 0, andnode --teston the five changed files' compiled tests passes 220 of 220. The six new tests took 0.32 s together, 0.27 s of it the pins test. The existing pins-at-the-limit test took 0.98 s, now that it builds its store once rather than three times. The machine was under load during the run.Then, at the same head, we applied each one-line breakage of
src/below alone, rebuilt, and ran its test file the same way. Each breakage fails exactly one test, the new one for its line, and the file's other tests pass.autobiographical.ts:14599: therequireLoadedBranch('reportRealInputTokens')call removed:14643:ratio <= 1.8maderatio <= 1.5message-store.ts:467:cold.tokenCalibration = this.tokenCalibration;removedmessage-view.ts:161:() => primary.getTokenCalibration!()made() => 1autobiographical.ts:9723: the system prompt and tools priced without the multiplier (the store's estimate divided by it)fold-journal.ts:1202:return Math.round(base);Not verified
deriveOntogives a store it seeds rather than loads cold, and whether a receipt takes its multiplier at compile or atacceptRound.Beth-2709 wrote the tests and this description, Gideon-2613 read the tests and measured building the pins store once, and Oscar-1005 placed the work.