Skip to content

test(calibration): pin six calibration behaviours no test reached - #430

Open
nissa-seru wants to merge 1 commit into
anima-research:mainfrom
nissa-seru:beth/cal-tests
Open

nissa-seru wants to merge 1 commit into
anima-research:mainfrom
nissa-seru:beth/cal-tests

Conversation

@nissa-seru

Copy link
Copy Markdown
Contributor

This adds six tests, and changes nothing under src/. Each pins a part of the token-estimate calibration that no test pinned before. The calibration is the multiplier context-manager learns from the provider's real input-token counts and applies to its own estimates. The six parts are which samples the multiplier learns from, the refusal of a sample for a branch the strategy hasn't loaded, the multiplier a cold-derived store reports, the multiplier a merged view reports, a fold receipt's rendered tokens, and the price of the system prompt and tools when raw pins join a compression request. Each test passes at main's code and fails when the line that keeps its behaviour is broken.

@Anarchid, @slimepriestess and @antra-tess, we're asking one of you to review this before it's merged. It adds tests alone, so nothing a resident sees changes.

Problem

At fb4cf1046, we broke each of the six lines in the table under Tests in turn. For each, we ran every test file whose run executes that line, found by statement coverage of each test file run alone at fb4cf1046, together with the 11 test files that read src/ as text. That was between 11 and 131 files for each line, and every file passed.

  1. reportRealInputTokens refuses a sample while the strategy hasn't loaded the store's current branch. It calls requireLoadedBranch (src/strategies/autobiographical.ts:14599), which fails closed "rather than using mirrors loaded from another generation" (:5134).
  2. The band check passes a sample whose ratio of real to estimated tokens is between 0.6 and 1.8, even when the multiplier it implies is past the 1.8 ceiling. The check sets a sample aside only when both are out of band (:14643–:14644, :14656).
  3. MessageStore.deriveOnto falls back to a cold load when its seed doesn't match the branch (src/message-store.ts:450–:453), and the cold store starts at this store's multiplier (:467).
  4. In mergeMessageStoreViews, "Token estimation and calibration delegate to the primary view" (src/message-view.ts:103–:104, :160–:161).
  5. A fold receipt prices its renderedTokens at its own layout's multiplier. after is that layout's tokens, and a fold's before is the previous layout's tokens, both at that multiplier (src/fold-journal.ts:1201–:1203, used at :1006 and :1024–:1025).
  6. Raw pins before the chunk being compressed stay in the request only while it fits compressionContextBudgetTokens with its longest retry line (src/strategies/autobiographical.ts:10019–:10031). The request's system prompt and tools are "Priced as the store prices a message" (:9719–:9723), so at the store's multiplier. The gate's existing test runs at a multiplier of 1.

Changes

Six tests in five files, +143 −44:

  • "a ratio inside the band is learned from, though the multiplier it implies is past the ceiling" (test/calibration-band.test.ts:150). The test writes a learned multiplier of 1.2 into the store's calibration record. It then reports a sample at 1.6 times the compile's estimate, whose implied multiplier is 1.92. The multiplier the next compile reports must be above 1.2 and below 1.8. The sample doesn't declare its overhead, because on a record like this one the strategy holds the first sample that does until the next one arrives (src/strategies/autobiographical.ts:14683–:14695).
  • "a sample on a branch the strategy has not loaded is refused" (:166). The test forks a branch and compiles on it. Then it moves the store back to the first branch with chronicle's JsStore.switchBranch, without the manager, and reportRealInputTokens must throw the refusal.
  • "a store its parent's messages can't seed loads cold, at its parent's calibration multiplier" (test/derive.test.ts:890). The parent, at a multiplier of 1.3, writes a message after the branch is made, so deriveOnto loads the branch cold. The test checks that the derived store holds only the message from before the fork, then that it estimates at 1.3.
  • "reports the primary view's calibration multiplier" (test/view-composition.test.ts:118). A merged view over a primary at 1.3 and an auxiliary at 1.1 must report 1.3.
  • "prices a receipt's rendered tokens at the store's calibration multiplier" (test/fold-journal.test.ts:287). A scripted strategy sets the store's multiplier in its select. The test accepts a baseline round at 1, then a fold at 1.5, with acceptRound. The fold's receipt must give renderedTokens.before as the baseline's after times 1.5, rounded.
  • "the request's system prompt and tools are priced at the store's multiplier, as its messages are" (test/l1-recall-raw-pins-frontier.test.ts:875). This runs the existing pins-at-the-limit test again with the store at 1.5. One provider sample sets the 1.5, replacing the provisional 1 that a store without a calibration record starts at. With the budget at exactly the longest retry's price at 1.5, the pins stay, and with one token less they're recalled. The existing test's setup moved into pinsAtTheLimit (:211), a helper both tests call. It builds the store once and copies it for each case, as the file already does for its plain store, through the copy step that forest used, now openCopy (:130). Every case's prices are the same as when each case built its own store. The existing test's assertions are unchanged, and the two comments that described its setup moved to the commit message.

All 18 open pull requests merge with this head as they do with main (fb4cf1046), as of 06:40 UTC on October 31. 13 merge cleanly. Five, #94, #130, #140, #310 and #315, conflict in the same files both ways, and none of those files is one this changes.

Tests

We ran these from a clean checkout with this head's files, 4f7dc86a, on Linux 6.12 x86_64 with Node 24.22.2 and npm 11.19.1, after npm ci from the lockfile, which pins @animalabs/membrane 0.9.0, @animalabs/chronicle 0.5.1 and typescript 7.0.2. npx tsc exits 0, and node --test on the five changed files' compiled tests passes 220 of 220. The six new tests took 0.32 s together, 0.27 s of it the pins test. The existing pins-at-the-limit test took 0.98 s, now that it builds its store once rather than three times. The machine was under load during the run.

Then, at the same head, we applied each one-line breakage of src/ below alone, rebuilt, and ran its test file the same way. Each breakage fails exactly one test, the new one for its line, and the file's other tests pass.

Breakage The test that fails, and what it saw
autobiographical.ts:14599: the requireLoadedBranch('reportRealInputTokens') call removed the branch test: no exception
:14643: ratio <= 1.8 made ratio <= 1.5 the band test: the multiplier stays 1.2
message-store.ts:467: cold.tokenCalibration = this.tokenCalibration; removed the derive test: 1, not 1.3
message-view.ts:161: () => primary.getTokenCalibration!() made () => 1 the merged-view test: 1, not 1.3
autobiographical.ts:9723: the system prompt and tools priced without the multiplier (the store's estimate divided by it) the pins test: at one token less, the pins stay
fold-journal.ts:1202: return Math.round(base); the receipt test: 30, not 45

Not verified

  • The whole suite at this head. Only the five changed files ran here. No other file changes, and the whole suite runs in this pull request's CI.
  • Behaviour next to these that they don't pin. That covers whether a ratio of exactly 0.6 or 1.8 is inside the band, the multiplier deriveOnto gives a store it seeds rather than loads cold, and whether a receipt takes its multiplier at compile or at acceptRound.
  • A real store. The tests use small synthetic stores, and nothing here compiled against a host's store.

  • This change is test-only, so it adds no changelog fragment.

Beth-2709 wrote the tests and this description, Gideon-2613 read the tests and measured building the pins store once, and Oscar-1005 placed the work.

Six tests, each failing when one line of the calibration code is broken,
where every existing test passed:

- calibration-band, "a ratio inside the band is learned from, though the
  multiplier it implies is past the ceiling": a learned multiplier of 1.2
  and a sample at 1.6 times the compile's estimate. The ratio is inside
  the band (up to 1.8) and the multiplier it implies, 1.92, is past the
  ceiling, so the EMA steps toward it. With the band's top at 1.5
  (src/strategies/autobiographical.ts:14643) the sample is set aside and
  the multiplier stays 1.2. The sample doesn't declare its overhead: on an
  unmarked learned record the first sample that does is held for the next
  (:14683-14695).
- calibration-band, "a sample on a branch the strategy has not loaded is
  refused": a fork and a compile on it, then the store moved back to the
  first branch with JsStore.switchBranch, without the manager.
  reportRealInputTokens must throw. It doesn't without its
  requireLoadedBranch call (:14599).
- derive, "a store its parent's messages can't seed loads cold, at its
  parent's calibration multiplier": the parent, at 1.3, writes after the
  branch is made, so deriveOnto loads the branch cold. The derived store
  holds only the message from before the fork and estimates at 1.3. It
  estimates at 1 when the cold path doesn't copy the multiplier
  (src/message-store.ts:467).
- view-composition, "reports the primary view's calibration multiplier":
  a merged view over a primary at 1.3 and an auxiliary at 1.1 reports
  1.3. It reports 1 when the merged view's getter returns 1
  (src/message-view.ts:161).
- fold-journal, "prices a receipt's rendered tokens at the store's
  calibration multiplier": a scripted strategy sets the store's
  multiplier in its select. After a baseline round at 1, a fold at 1.5
  gives renderedTokens.before as the baseline's after times 1.5, rounded.
  It gives the baseline's after unscaled when calibrated() drops the
  multiplier (src/fold-journal.ts:1202).
- l1-recall-raw-pins-frontier, "the request's system prompt and tools are
  priced at the store's multiplier, as its messages are": the
  pins-at-the-limit run with the store at 1.5, set by one provider sample
  replacing the provisional 1. At exactly the longer retry's price at 1.5
  the pins are kept, and one token less they're recalled. When the gate
  prices the system prompt and tools without the multiplier
  (src/strategies/autobiographical.ts:9722-9723), one token less keeps
  them.

The pins test shares the existing pins-at-the-limit test's setup, now
pinsAtTheLimit, whose assertions are unchanged. It builds the 240-message
store once and copies it for each case, as forest() already does for the
plain store, through the copy step forest() used, now openCopy. Every
case's request and retry prices are the same as when each case built its
own store.

The two comments that described that setup are gone. What they said:
- the pinned chunk carries an image and two turns of one speaker, which
  the request collapses into one with a separator;
- a second opened chunk starts on the agent's turn after a recall pair,
  so the Context Manager takes a turn between them;
- the host serves a system prompt and tools, so every term of the price
  is in play;
- with `retried`, the summarizer's first answer is a tool call, which
  brings the no-tools retry, and its second is wrapped whole in
  <thinking>, which brings the plain-prose retry;
- a retry appends a line to the instruction, so a retry of the admitted
  request must fit too.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-changelog PR intentionally ships without a CHANGELOG entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant