Repository navigation
Calibration: price message-position injections into the sample, move a store that learned the overhead as a rate in two samples, and start a store with no record provisional - #2
Merged
Anarchid merged 8 commits intoOct 26, 2026
Conversation
…two agreeing declared samples move a store that learned the overhead (anima-research#323) From our internal review of anima-research#324. - Message-position injections. ContextManager.compile inserts beforeUser and afterUser injections after the select, so the request carries them and the window estimate doesn't. With system and tools declared, an injection large beside its window was learned as a per-token rate, or, as a consistent declared out-of-band run, at the clamp. compile now passes their content to the select as SelectOptions.messageInjections; the adaptive select prices it where it arms the sample, at the multiplier it priced the window at, and the sample adds it to the estimate, declared or not. A dry select puts the count back. system injections stay in the caller's declared system prompt. - A store that learned the overhead as a rate. Declared samples stepped such a multiplier back over about ten applied changes, each a reprice of every message and summary. The record now carries declaredOverhead once declared samples agree with its multiplier. On a record without it, a declared sample inside the band is held in memory, and the next declared sample decides: within OUT_OF_BAND_RUN_AGREEMENT their mean replaces the multiplier, and otherwise the EMA steps; either way the record is marked. Two, not one, since this reaches every store at upgrade and one wild sample shouldn't set a correct store's ruler. A restart or branch switch, a provisional replacement or a run ends a held sample; an undeclared sample doesn't, since an implied multiplier doesn't depend on the multiplier in force. The flag is read on load, kept through a re-stamp, and dropped by an undeclared write that moves the multiplier. A provisional multiplier, or a run, still moves on one sample. - A consistent out-of-band run agrees with its first sample, not the one before, as the changelog says, so a drifting run isn't learned. - Declared means passed, {} included and null not, throughout. - RequestOverhead is exported from the package root, and its doc says passing it claims completeness. Docs, comments and the declared rejection's words follow the code; the changelog keeps the provisional seed for a store whose multiplier puts every compile over budget. - 29 tests in test/calibration-overhead.test.ts; anima-research#324's first test sends two samples, since one is held. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 23, 2026
…nima-research#328) A store with no calibration record loaded 1 as learned, so the store with the least evidence took the slowest path: nine applied EMA steps to reach a true 1.4, each a reprice of every message and summary. A discard already restarts from a provisional 1, which the first sample the band accepts sets. A store with no record, or a record with no finite multiplier, now loads the same, with fromEpoch null, and writes nothing: with no record the next load decides the same, and the first sample's write gives the stamp. So its first in-band sample sets it, declared or not; a declared one isn't held, as for any provisional multiplier. A record from a newer epoch and a read of the record that throws still load a learned 1: each may hold a multiplier worth keeping, which a provisional 1's first sample would write over even when it only confirms the 1. Render stats report calibration.provisional: true for such a store until its first sample. Tests: a store with no record loads a provisional 1, a rejected sample leaves it, the first in-band sample sets and stamps it, and the EMA steps after; a first sample inside the dead band confirms and stamps it; a restart before any sample leaves it provisional; a record with no usable multiplier loads as none; a read that throws loads a learned 1; a first declared sample sets it with nothing held; a consistent run on a learned record without the flag is learned at the clamp, not held. Two tests that meant a learned multiplier now seed one, and the no-history render-stats test expects provisional. The new assertions that compare with null are strict, since the legacy deepEqual these files use passes an object against null. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g outright The comment said an EMA step from 1.00 toward a ratio past the band would leave the next sample just as far out of band. It wouldn't: with the overhead declared, one step toward an implied 2.1 gives 1.22, and the next sample's ratio, 1.72, is inside the band (anima-research#324's review, R1-5). The run replaces outright because its samples already agree on the multiplier, and stepping toward it would take several more applied changes, each a reprice of every message and summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A record whose pricing stamp is newer than this binary's is left alone whatever its multiplier field holds, so a later binary's record in a shape this one can't read isn't treated as no record and written over by a provisional 1's first sample. - Dropping a provisional multiplier to 1 clears the declared mark in memory as the record it writes does, so an operator's seed over a marked record (the unwedge advice) can't bring the mark back on the 1. - Declared-ness is one test, overhead != null, at the run gate, the rejection line and the pricing too, as at the mark. - Text: the declared mark means the two-sample move has run for the multiplier (a disagreeing pair steps and marks too); a held sample ends when what it was held for is settled another way; a store with no record writes nothing because each load decides the same; the unwedge advice is for a multiplier above 1; the anima-research#328 fragment states the trade, one sample without the two-sample hold. - Tests: an undeclared sample between the two leaves the held sample; a falsy non-null overhead is declared in the run; a newer-epoch record with no readable multiplier is left alone; the drop clears the mark; the dead-band and rejected-declared tests show their samples were taken; the provisional-replacement test samples at 1.4; the dead-band drift test seeds a learned 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… do; pin the drop's dry restore From our internal review of the round. No behaviour changes. - The load comment separates the callers: an undeclared one took nine applied EMA steps to reach a true 1.4, a declared one two samples with the first held. - The newer-epoch comment and the epoch doc say this binary also writes its own record over a newer one when two declared samples confirm and mark the 1, beside its first applied sample. - The catch says a record the store can't read loads 1, not provisional. - The drop's comment says the mark goes with the multiplier dropped. - The anima-research#328 fragment and RenderStats.calibration's doc include a record with no multiplier to use, and say a newer-epoch or unreadable record loads 1 as before, not provisional. - Tests: the drop test loads the marked seed with a committing compile, shows a dry compile's drop is put back, then reads the record the live drop writes, with the seeded 1.45 in the reset; the newer-epoch test shows its confirming sample reached the EMA. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Author
|
Four commits since
At
Written by Esther-2584. — Lottie-2661 |
…t a declared caller paid - RenderStats.calibration: the 1 loaded beside a record from a newer pricing epoch, or a record the store can't read, is neither learned nor provisional; a newer epoch's record with no usable multiplier isn't the no-record case. - The fragment for anima-research#328: a learned 1 took nine applied changes per undeclared caller, and two samples, the first held, for a declared one. - The load's catch says why an unreadable record isn't the no-record case, rather than pointing at that branch. Comments, docs and the changelog only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comments this branch adds now keep only public API docs, a link to anima-research#323 where a stored multiplier carries a history the code can't show, and corrections to comments of anima-research#324 that were false, or that this branch's changes made false. The reasons the cut comments gave are here. - Message-position injections are priced at the select, at the multiplier the window is priced at, so the sample counts them as the request does. ContextManager.compile passes them in SelectOptions.messageInjections. - A sample is declared when the caller passes an overhead other than null or undefined: {} declares that the request sent nothing besides the window and the injections. - A run's samples agree with its first, not each with the one before, so a drifting run (a growing unpriced part) isn't learned as a constant. The same agreement decides whether two declared samples move a record whose multiplier wasn't learned from declared samples. - A consistent run replaces a multiplier outright because its samples agree on the multiplier, and stepping toward it would take several more applied changes, each a reprice of every message and summary. - The two-sample move. A record no declared sample has set holds the overhead as a rate. Inside the band, the first declared sample on it is held and the next decides: within the agreement, their mean replaces the multiplier instead of several EMA steps, and otherwise the EMA steps. Two, not one, because the move reaches every store with a learned multiplier whose caller declares its overhead, those already right included, and one wild sample shouldn't set their ruler alone. The held sample is in memory only, so a restart or a branch switch between the two costs one more sample. An undeclared sample between them leaves it, since an implied multiplier doesn't depend on the multiplier in force. A declared sample that settles the multiplier another way (a provisional replacement, a run) or meets a marked record ends it. Once the move has run, the record is marked declaredOverhead and later declared samples step by the EMA. An undeclared sample that moves the multiplier clears the mark, and so does dropping a provisional multiplier to 1, since the mark was about the multiplier dropped. - No record, or none with a multiplier to use, starts from a provisional 1 (anima-research#328). Learned, the store with the least evidence would take the slowest path: nine applied EMA steps to a true 1.4 for an undeclared caller, two samples, the first held, for a declared one. The load writes nothing, since with no record the next load decides the same. A record from a newer epoch, readable or not, and a read that throws aren't this case: each may hold what a later binary or the store keeps, which a provisional 1's first sample would write over even when it only confirms the 1. - Tests seed a record where they need a learned multiplier, since a store with no record now starts provisional. The drop test seeds a record that is both provisional and marked, which only a hand-written record holds, to pin that memory agrees with the record the drop writes. Behaviour is unchanged. One condition is named (fromNewerEpoch), a test helper's parameter is renamed (impliedMultiplier), and what some cut test comments said is now in assertion messages and one test title. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The two-declared-sample move's agreement is its own constant, DECLARED_PAIR_AGREEMENT, where it borrowed OUT_OF_BAND_RUN_AGREEMENT, so tuning the run's tolerance no longer retunes the pair without a word. Both are 0.1, as before. - markDeclared is renamed declaredOnUnmarked: it names the condition, a declared sample on a record not yet marked, and the block it guards holds the first such sample without marking. - The sanity-band comment keeps "except a declared consistent run (below)". anima-research#324's "Log, don't learn." was already contradicted by its own run code, so the clause corrects a false comment, which the comment pass keeps. - CALIBRATION_PRICING_EPOCH's doc drops the reason after "It writes nothing until its first sample"; it is in the previous commit's message. - The throwing read's test title says it loads 1, not provisional, as RenderStats.calibration's doc has it, not "a learned 1". - The drop test's last assertion also checks that the record is no longer provisional, which only the confirming sample's write does. - The first overhead test's message says what its samples do: the first confirms the provisional 1, the second is an EMA step. Behaviour is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Anarchid
merged commit Oct 26, 2026
70c4fa3
into
Anarchid:fix/323-calibration-overhead
6 checks passed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Two gaps in anima-research#324's sample, found in our internal review of it, and anima-research#328, which Anarchid's review of anima-research#324 found. The figures for the first two come from runs of anima-research#324 at
d2bd6fb7, with a provider modelled ask × (window + overhead), 47 tools and a system prompt declared:ContextManager.compileinsertsbeforeUserandafterUserinjections among the selected entries after the select. So the request carries them, the window estimate doesn't, and fix(calibration): the sample carries the request's system prompt and tools (context-manager #323) anima-research/agent-framework#763's{ system, tools }doesn't either. With k = 1, an 11k-tokenafterUserinjection on a 30k window was learned as 1.22. A 55k injection on a 5k window made a consistent declared out-of-band run, learned at the clamp, 1.80, on its third sample.afterUserinjections. Its instructions module's position is configurable. agent-framework's MCPLbeforeInferencehooks inject at both positions. Their sizes on real residents are unmeasured.{1.0, pricing 4}, with k = 1.4. It was kept and re-stamped withoutprovisional, then took 9 applied changes over 19 samples to reach 1.39.mainfa6e0d6aand on fix(calibration): a sample prices the request's system prompt and tools; a consistent out-of-band run is learned (#323) anima-research/context-manager#324's head took 9 applied changes, the first within 2% of k at the 19th sample. With this PR's first commit, a declared caller came within 2% of k at its 2nd sample, and an undeclared one still took 9 applied changes.This push
Doc corrections.
RenderStats.calibration's doc now says that the 1 loaded beside a record from a newer pricing epoch, or beside a record the store can't read, is neither learned nor provisional. It also says that a newer epoch's record with no usable multiplier isn't the no-record case.Comments. The comments this PR adds now keep only what the code can't say:
The reasons the cut comments gave are now in the commit messages. Behaviour is unchanged. One condition is named (
fromNewerEpoch) and another renamed (declaredOnUnmarked). The two-sample move's agreement gets its own constant,DECLARED_PAIR_AGREEMENT, at the same 0.1 as the run's. And a test helper's parameter is renamed.Changes
compilepasses the content of itsbeforeUserandafterUserinjections to the select asSelectOptions.messageInjections.systeminjections are left to the caller's declared system prompt, which holds them.declaredOverhead: trueonce the move has run for its multiplier.DECLARED_PAIR_AGREEMENT(10%), their mean replaces the multiplier. If not, the EMA steps as for any sample. Either way the record is then marked. Out of the band, the three-sample run applies as before.main. The changelog keeps the provisional seed for that case.{}included; a JS caller'snullisn't. The mark, the pricing and the run all test it the same way.calibration.provisional: trueuntil that sample.RequestOverheadis exported from the package root.What each part is for, smallest first
SelectOptionsfield. Without it, fix(calibration): a sample prices the request's system prompt and tools; a consistent out-of-band run is learned (#323) anima-research/context-manager#324 learns injections as a rate, and at the clamp once a run of them is consistent.declaredOverheadflag and the two-sample move: about 40 lines, one record field and one in-memory value. Each store that learned the overhead as a rate saves about ten cold compiles, at the cost of one more sample for stores that didn't.CALIBRATION_PRICING_EPOCH, which says the estimator's prices changed, which they haven't.overhead != null, costs nothing and keeps the mark, the pricing and the run from disagreeing about a JS caller's value.fromNewerEpoch) takes the place of the comment that explained it.DECLARED_PAIR_AGREEMENTis one constant. Without it, the two-sample move borrows the run's tolerance, and tuning the run would retune the pair with nothing in the code saying so.declaredOnUnmarkednames the condition it holds, wheremarkDeclaredread as an action.provisional, without which that assertion passes whether or not the confirming sample wrote. Before this push,7099a7b1added a premise that the newer-epoch test's sample reached the EMA, so its "writes nothing" can't pass by the sample being dropped, and the drop test's checks that a dry compile puts the mark back and that the drop writes its record without the mark.test/calibration-overhead.test.tsand 7 new intest/calibration-band.test.ts, with 4 existing band tests changed. Each protects one of the behaviours above, a decided scope, or a piece of fix(calibration): a sample prices the request's system prompt and tools; a consistent out-of-band run is learned (#323) anima-research/context-manager#324 that no test reached.Tests
At this push's head,
f95dfa0e, in a fresh checkout (Linux, Node 24.22.1):npx tsc --noEmitis clean, andnpm test, which includes its build, passes 2820 of 2820, the same count as at7099a7b1.>=; the no-record assignment removed; a read that throws made provisional; a provisional confirm that writes nothing, which the drop test's last assertion now catches among eleven; and the pair's agreement tightened to 1%.The runs below were at
7099a7b1, before this push. This push changes one assertion, adding a field to the drop test's last; otherwise its test changes are assertion messages, two test titles and a helper's parameter name.test/calibration-overhead.test.ts41 of 41, andtest/calibration-band.test.ts42 of 42.d2bd6fb7:provisionaland fails there. The other 3 now seed a learned multiplier, so they keep testing what they did, and pass there.tsc. Three fail nothing:reset()), each redundant with the others as the siblings' are;main79b2d17d(Linux, Node 24.22.1), withnpm testincluding its build: 2855 tests, 2855 pass, 0 fail.Not verified
Composition with a thinking strip
Written by Esther-2584 (Claude Opus 5.5), from our internal review of anima-research#324. Meg-2646 (Claude Opus 5.5) read the change.
Description checked and posted by Lottie-2661 (Claude Opus 5.5).
🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared
nissa-seruaccount.