Skip to content

Calibration: price message-position injections into the sample, move a store that learned the overhead as a rate in two samples, and start a store with no record provisional - #2

Merged
Anarchid merged 8 commits into
Anarchid:fix/323-calibration-overheadfrom
nissa-seru:esther/cm324-house-fixes
Oct 26, 2026
Merged

Anarchid merged 8 commits into
Anarchid:fix/323-calibration-overheadfrom
nissa-seru:esther/cm324-house-fixes

Conversation

@nissa-seru

@nissa-seru nissa-seru commented Oct 23, 2026 •

Copy link
Copy Markdown

Problem

Two gaps in anima-research#324's sample, found in our internal review of it, and anima-research#328, which Anarchid's review of anima-research#324 found. The figures for the first two come from runs of anima-research#324 at d2bd6fb7, with a provider modelled as k × (window + overhead), 47 tools and a system prompt declared:

This push

Changes

  • Injections priced with the window.
    • compile passes the content of its beforeUser and afterUser injections to the select as SelectOptions.messageInjections.
    • The adaptive select prices them where it arms the sample, at the multiplier it priced the window at, and the sample adds them to the estimate.
    • A dry select puts the count back with the other calibration fields.
    • system injections are left to the caller's declared system prompt, which holds them.
  • Two agreeing declared samples move a learned store. The record gains declaredOverhead: true once the move has run for its multiplier.
    • On a record without it, a declared sample inside the band is held in memory, and the next declared sample decides. If the two agree within DECLARED_PAIR_AGREEMENT (10%), their mean replaces the multiplier. If not, the EMA steps as for any sample. Either way the record is then marked. Out of the band, the three-sample run applies as before.
    • Why two: the move reaches every store that samples after the upgrade, including those whose multiplier is already right, and one wild sample shouldn't set their ruler alone.
    • What ends a held sample: a restart or a branch switch, at the cost of one more sample, and any declared sample that takes another path (a provisional replacement, a run). An undeclared sample between the two leaves it, since an implied multiplier doesn't depend on the multiplier in force.
    • Not reached: a store whose multiplier, above 1, puts every compile over budget gets no sample, as on main. The changelog keeps the provisional seed for that case.
    • The flag's lifecycle: it is read on load, kept through a re-stamp, and written by every declared write. It is dropped by an undeclared write that moves the multiplier, and by dropping a provisional multiplier to 1.
    • Unchanged: a provisional multiplier, or a consistent run, is replaced by one sample as before.
    • Declared means the object is passed, {} included; a JS caller's null isn't. The mark, the pricing and the run all test it the same way.
  • A consistent out-of-band run agrees with its first sample, not the one before, as the changelog says. A drifting 2.00, 2.18, 2.38 is no longer learned. The comment beside it no longer gives a reason for replacing outright that the review of fix(calibration): a sample prices the request's system prompt and tools; a consistent out-of-band run is learned (#323) anima-research/context-manager#324 showed false (R1-5): one EMA step from 1.00 toward an implied 2.1 would already bring the next sample inside the band. The run's actual reason is now in the commit message: its samples agree on the multiplier, and stepping toward it would take several more applied changes.
  • A store with no record starts from a provisional 1, as a discard's restart does (A fresh store's calibration multiplier is treated as learned, not provisional anima-research/context-manager#328). That covers no record, or a record with no finite multiplier.
    • Its first in-band sample sets the multiplier, declared or not: one sample, without the two-sample hold, as after a discard.
    • Nothing is written at load: with no record, each load decides the same, and the first sample's write gives the stamp.
    • Render stats report calibration.provisional: true until that sample.
    • Left alone: a record with a newer pricing stamp, whatever its multiplier field holds, and a read of the record that throws. Each still loads 1, not provisional, since each may hold what a later binary or the store keeps, which a provisional 1's first sample would write over even when it only confirms the 1.
  • Text and API:
    • RequestOverhead is exported from the package root.
    • Its doc says passing it claims it's everything the request sent besides the window and the compile's injections.
    • The declared rejection's words no longer call a sample that may join a run "not a window-shaped request".

What each part is for, smallest first

  • The least is the injection pricing alone: about 15 lines and one optional SelectOptions field. Without it, fix(calibration): a sample prices the request's system prompt and tools; a consistent out-of-band run is learned (#323) anima-research/context-manager#324 learns injections as a rate, and at the clamp once a run of them is consistent.
  • The declaredOverhead flag and the two-sample move: about 40 lines, one record field and one in-memory value. Each store that learned the overhead as a rate saves about ten cold compiles, at the cost of one more sample for stores that didn't.
    • Weighed and set aside: replacing on the first declared sample, which is cheaper by about 8 lines but lets one wild sample set a correct store's ruler; and bumping CALIBRATION_PRICING_EPOCH, which says the estimator's prices changed, which they haven't.
  • Comparing a run with its first sample is one line.
  • One test of declared-ness, overhead != null, costs nothing and keeps the mark, the pricing and the run from disagreeing about a JS caller's value.
  • Ending a held sample when a declared sample settles the multiplier another way is one branch. Without it, a held sample survives a provisional replacement or a run, and pairs with a later sample across it.
  • The provisional 1 for a store with no record is one assignment, plus a guard that leaves a newer-epoch record alone. It reuses the provisional state a discard already has, so it adds no new state or record format. It saves a new store about nine cold compiles, at the cost the provisional path already carries: one wild first sample sets its ruler.
  • Clearing the mark when a provisional multiplier is dropped is one line. It keeps memory agreeing with the record the drop writes.
  • The root export and the completeness clause make the type a caller passes importable and say what passing it claims, which fix(calibration): the sample carries the request's system prompt and tools (context-manager #323) anima-research/agent-framework#763 relies on.
  • The wording of the declared rejection keeps the log from calling a sample that may join a run a malformed request.
  • Naming the no-record branch's condition (fromNewerEpoch) takes the place of the comment that explained it.
  • DECLARED_PAIR_AGREEMENT is one constant. Without it, the two-sample move borrows the run's tolerance, and tuning the run would retune the pair with nothing in the code saying so. declaredOnUnmarked names the condition it holds, where markDeclared read as an action.
  • This push's test changes are assertion messages, a title and a parameter name, which carry what the cut test comments said; a second title, which now says the throwing read loads 1, not provisional; and one more field in the drop test's last assertion, the record's provisional, without which that assertion passes whether or not the confirming sample wrote. Before this push, 7099a7b1 added a premise that the newer-epoch test's sample reached the EMA, so its "writes nothing" can't pass by the sample being dropped, and the drop test's checks that a dry compile puts the mark back and that the drop writes its record without the mark.
  • The tests: 33 new in test/calibration-overhead.test.ts and 7 new in test/calibration-band.test.ts, with 4 existing band tests changed. Each protects one of the behaviours above, a decided scope, or a piece of fix(calibration): a sample prices the request's system prompt and tools; a consistent out-of-band run is learned (#323) anima-research/context-manager#324 that no test reached.

Tests

At this push's head, f95dfa0e, in a fresh checkout (Linux, Node 24.22.1):

  • npx tsc --noEmit is clean, and npm test, which includes its build, passes 2820 of 2820, the same count as at 7099a7b1.
  • The two calibration files pass together, 83 of 83.
  • Six variants each fail at least one test: the named condition dropped, or widened to >=; the no-record assignment removed; a read that throws made provisional; a provisional confirm that writes nothing, which the drop test's last assertion now catches among eleven; and the pair's agreement tightened to 1%.

The runs below were at 7099a7b1, before this push. This push changes one assertion, adding a field to the drop test's last; otherwise its test changes are assertion messages, two test titles and a helper's parameter name.

  • The two calibration files pass: test/calibration-overhead.test.ts 41 of 41, and test/calibration-band.test.ts 42 of 42.
  • On fix(calibration): a sample prices the request's system prompt and tools; a consistent out-of-band run is learned (#323) anima-research/context-manager#324's own source, at d2bd6fb7:
    • Overhead file: 28 of the 33 new tests fail there. The 5 that pass are 4 that test fix(calibration): a sample prices the request's system prompt and tools; a consistent out-of-band run is learned (#323) anima-research/context-manager#324's own pieces (an undeclared sample ending a run, a run below the band, one compile counting once, a branch switch restarting a run) and 1 control, a declared record stepping by the EMA.
    • Band file: 5 of the 7 new tests fail there. The 2 that pass pin what stays as it was: a read that throws, and a newer-epoch record with no readable multiplier, each load 1, not provisional.
    • The 4 changed band tests: the no-history render-stats test now expects provisional and fails there. The other 3 now seed a learned multiplier, so they keep testing what they did, and pass there.
  • Removal variants: 46 pieces, each removed or widened alone. 42 fail at least one test, and the root export fails tsc. Three fail nothing:
    • two reset lines that mirror their sibling fields' (the load's start, and reset()), each redundant with the others as the siblings' are;
    • pricing a falsy overhead by truthiness, which prices it at 0 either way.
  • The full suite, on that head merged with main 79b2d17d (Linux, Node 24.22.1), with npm test including its build: 2855 tests, 2855 pass, 0 fail.

Not verified

Composition with a thinking strip


Written by Esther-2584 (Claude Opus 5.5), from our internal review of anima-research#324. Meg-2646 (Claude Opus 5.5) read the change.

Description checked and posted by Lottie-2661 (Claude Opus 5.5).

🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared nissa-seru account.

…two agreeing declared samples move a store that learned the overhead (anima-research#323)

From our internal review of anima-research#324.

- Message-position injections. ContextManager.compile inserts beforeUser
  and afterUser injections after the select, so the request carries them
  and the window estimate doesn't. With system and tools declared, an
  injection large beside its window was learned as a per-token rate, or,
  as a consistent declared out-of-band run, at the clamp. compile now
  passes their content to the select as SelectOptions.messageInjections;
  the adaptive select prices it where it arms the sample, at the
  multiplier it priced the window at, and the sample adds it to the
  estimate, declared or not. A dry select puts the count back. system
  injections stay in the caller's declared system prompt.
- A store that learned the overhead as a rate. Declared samples stepped
  such a multiplier back over about ten applied changes, each a reprice
  of every message and summary. The record now carries declaredOverhead
  once declared samples agree with its multiplier. On a record without
  it, a declared sample inside the band is held in memory, and the next
  declared sample decides: within OUT_OF_BAND_RUN_AGREEMENT their mean
  replaces the multiplier, and otherwise the EMA steps; either way the
  record is marked. Two, not one, since this reaches every store at
  upgrade and one wild sample shouldn't set a correct store's ruler. A
  restart or branch switch, a provisional replacement or a run ends a
  held sample; an undeclared sample doesn't, since an implied multiplier
  doesn't depend on the multiplier in force. The flag is read on load,
  kept through a re-stamp, and dropped by an undeclared write that moves
  the multiplier. A provisional multiplier, or a run, still moves on one
  sample.
- A consistent out-of-band run agrees with its first sample, not the one
  before, as the changelog says, so a drifting run isn't learned.
- Declared means passed, {} included and null not, throughout.
- RequestOverhead is exported from the package root, and its doc says
  passing it claims completeness. Docs, comments and the declared
  rejection's words follow the code; the changelog keeps the provisional
  seed for a store whose multiplier puts every compile over budget.
- 29 tests in test/calibration-overhead.test.ts; anima-research#324's first test sends
  two samples, since one is held.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Esther-2584 and others added 4 commits October 23, 2026 09:10
…nima-research#328)

A store with no calibration record loaded 1 as learned, so the store
with the least evidence took the slowest path: nine applied EMA steps to
reach a true 1.4, each a reprice of every message and summary. A discard
already restarts from a provisional 1, which the first sample the band
accepts sets. A store with no record, or a record with no finite
multiplier, now loads the same, with fromEpoch null, and writes nothing:
with no record the next load decides the same, and the first sample's
write gives the stamp. So its first in-band sample sets it, declared or
not; a declared one isn't held, as for any provisional multiplier.

A record from a newer epoch and a read of the record that throws still
load a learned 1: each may hold a multiplier worth keeping, which a
provisional 1's first sample would write over even when it only confirms
the 1.

Render stats report calibration.provisional: true for such a store until
its first sample.

Tests: a store with no record loads a provisional 1, a rejected sample
leaves it, the first in-band sample sets and stamps it, and the EMA steps
after; a first sample inside the dead band confirms and stamps it; a
restart before any sample leaves it provisional; a record with no usable
multiplier loads as none; a read that throws loads a learned 1; a first
declared sample sets it with nothing held; a consistent run on a learned
record without the flag is learned at the clamp, not held. Two tests that
meant a learned multiplier now seed one, and the no-history render-stats
test expects provisional. The new assertions that compare with null are
strict, since the legacy deepEqual these files use passes an object
against null.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g outright

The comment said an EMA step from 1.00 toward a ratio past the band
would leave the next sample just as far out of band. It wouldn't: with
the overhead declared, one step toward an implied 2.1 gives 1.22, and the
next sample's ratio, 1.72, is inside the band (anima-research#324's review, R1-5). The
run replaces outright because its samples already agree on the
multiplier, and stepping toward it would take several more applied
changes, each a reprice of every message and summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A record whose pricing stamp is newer than this binary's is left alone
  whatever its multiplier field holds, so a later binary's record in a
  shape this one can't read isn't treated as no record and written over
  by a provisional 1's first sample.
- Dropping a provisional multiplier to 1 clears the declared mark in
  memory as the record it writes does, so an operator's seed over a
  marked record (the unwedge advice) can't bring the mark back on the 1.
- Declared-ness is one test, overhead != null, at the run gate, the
  rejection line and the pricing too, as at the mark.
- Text: the declared mark means the two-sample move has run for the
  multiplier (a disagreeing pair steps and marks too); a held sample ends
  when what it was held for is settled another way; a store with no
  record writes nothing because each load decides the same; the unwedge
  advice is for a multiplier above 1; the anima-research#328 fragment states the trade,
  one sample without the two-sample hold.
- Tests: an undeclared sample between the two leaves the held sample; a
  falsy non-null overhead is declared in the run; a newer-epoch record
  with no readable multiplier is left alone; the drop clears the mark;
  the dead-band and rejected-declared tests show their samples were
  taken; the provisional-replacement test samples at 1.4; the dead-band
  drift test seeds a learned 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… do; pin the drop's dry restore

From our internal review of the round. No behaviour changes.

- The load comment separates the callers: an undeclared one took nine
  applied EMA steps to reach a true 1.4, a declared one two samples with
  the first held.
- The newer-epoch comment and the epoch doc say this binary also writes
  its own record over a newer one when two declared samples confirm and
  mark the 1, beside its first applied sample.
- The catch says a record the store can't read loads 1, not provisional.
- The drop's comment says the mark goes with the multiplier dropped.
- The anima-research#328 fragment and RenderStats.calibration's doc include a record
  with no multiplier to use, and say a newer-epoch or unreadable record
  loads 1 as before, not provisional.
- Tests: the drop test loads the marked seed with a committing compile,
  shows a dry compile's drop is put back, then reads the record the live
  drop writes, with the seeded 1.45 in the reset; the newer-epoch test
  shows its confirming sample reached the EMA.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nissa-seru nissa-seru changed the title Calibration: price message-position injections into the sample, and move a store that learned the overhead as a rate in two samples Calibration: price message-position injections into the sample, move a store that learned the overhead as a rate in two samples, and start a store with no record provisional Oct 23, 2026
@nissa-seru

Copy link
Copy Markdown
Author

Four commits since 86984f6f, the head your review of anima-research/context-manager#324 traced. The description is updated to match.

  • 7df5bb6e, for A fresh store's calibration multiplier is treated as learned, not provisional anima-research/context-manager#328: a store with no calibration record, or a record with no finite multiplier, loads a provisional 1, as a discard's restart does, and writes nothing at load. Its first sample inside the band sets the multiplier, declared or not: one sample, without this PR's two-sample hold.
  • e93803d7: the comment beside the run-anchor change now gives the run's actual reason for replacing outright, after R1-5 showed the old one wrong.
  • 0215b1be, from our internal review of those two:
    • a record with a newer pricing stamp is left alone whatever its multiplier field holds;
    • dropping a provisional multiplier to 1 clears the declared mark in memory as the record it writes does;
    • the run gate, the rejection line and the pricing test declared-ness as overhead != null, as the mark does.
  • 7099a7b1: comments, the changelog fragment and test assertions only.

At 7099a7b1:

  • Calibration test files: both pass, 41 of 41 and 42 of 42.
  • Full suite: merged with main 79b2d17d, 2855 of 2855.

Written by Esther-2584.

— Lottie-2661

Esther-2584 and others added 3 commits October 25, 2026 19:43
…t a declared caller paid

- RenderStats.calibration: the 1 loaded beside a record from a newer
  pricing epoch, or a record the store can't read, is neither learned nor
  provisional; a newer epoch's record with no usable multiplier isn't the
  no-record case.
- The fragment for anima-research#328: a learned 1 took
  nine applied changes per undeclared caller, and two samples, the first
  held, for a declared one.
- The load's catch says why an unreadable record isn't the no-record case,
  rather than pointing at that branch.

Comments, docs and the changelog only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comments this branch adds now keep only public API docs, a link to
anima-research#323 where a stored multiplier carries a history the code can't show,
and corrections to comments of anima-research#324 that were false, or that this
branch's changes made false. The reasons the cut comments gave are here.

- Message-position injections are priced at the select, at the
  multiplier the window is priced at, so the sample counts them as the
  request does. ContextManager.compile passes them in
  SelectOptions.messageInjections.
- A sample is declared when the caller passes an overhead other than
  null or undefined: {} declares that the request sent nothing besides
  the window and the injections.
- A run's samples agree with its first, not each with the one before,
  so a drifting run (a growing unpriced part) isn't learned as a
  constant. The same agreement decides whether two declared samples move
  a record whose multiplier wasn't learned from declared samples.
- A consistent run replaces a multiplier outright because its samples
  agree on the multiplier, and stepping toward it would take several
  more applied changes, each a reprice of every message and summary.
- The two-sample move. A record no declared sample has set holds the
  overhead as a rate. Inside the band, the first declared sample on it
  is held and the next decides: within the agreement, their mean
  replaces the multiplier instead of several EMA steps, and otherwise
  the EMA steps. Two, not one, because the move reaches every store with
  a learned multiplier whose caller declares its overhead, those already
  right included, and one wild sample shouldn't set their ruler alone.
  The held sample is in memory only, so a restart or a branch switch
  between the two costs one more sample. An undeclared sample between
  them leaves it, since an implied multiplier doesn't depend on the
  multiplier in force. A declared sample that settles the multiplier
  another way (a provisional replacement, a run) or meets a marked
  record ends it. Once the move has run, the record is marked
  declaredOverhead and later declared samples step by the EMA. An
  undeclared sample that moves the multiplier clears the mark, and so
  does dropping a provisional multiplier to 1, since the mark was about
  the multiplier dropped.
- No record, or none with a multiplier to use, starts from a
  provisional 1 (anima-research#328). Learned, the store with the least evidence would
  take the slowest path: nine applied EMA steps to a true 1.4 for an
  undeclared caller, two samples, the first held, for a declared one.
  The load writes nothing, since with no record the next load decides
  the same. A record from a newer epoch, readable or not, and a read
  that throws aren't this case: each may hold what a later binary or the
  store keeps, which a provisional 1's first sample would write over
  even when it only confirms the 1.
- Tests seed a record where they need a learned multiplier, since a
  store with no record now starts provisional. The drop test seeds a
  record that is both provisional and marked, which only a hand-written
  record holds, to pin that memory agrees with the record the drop
  writes.

Behaviour is unchanged. One condition is named (fromNewerEpoch), a test
helper's parameter is renamed (impliedMultiplier), and what some cut
test comments said is now in assertion messages and one test title.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The two-declared-sample move's agreement is its own constant,
  DECLARED_PAIR_AGREEMENT, where it borrowed OUT_OF_BAND_RUN_AGREEMENT,
  so tuning the run's tolerance no longer retunes the pair without a
  word. Both are 0.1, as before.
- markDeclared is renamed declaredOnUnmarked: it names the condition, a
  declared sample on a record not yet marked, and the block it guards
  holds the first such sample without marking.
- The sanity-band comment keeps "except a declared consistent run
  (below)". anima-research#324's "Log, don't learn." was already contradicted by its
  own run code, so the clause corrects a false comment, which the
  comment pass keeps.
- CALIBRATION_PRICING_EPOCH's doc drops the reason after "It writes
  nothing until its first sample"; it is in the previous commit's
  message.
- The throwing read's test title says it loads 1, not provisional, as
  RenderStats.calibration's doc has it, not "a learned 1".
- The drop test's last assertion also checks that the record is no
  longer provisional, which only the confirming sample's write does.
- The first overhead test's message says what its samples do: the first
  confirms the provisional 1, the second is an EMA step.

Behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Anarchid
Anarchid merged commit 70c4fa3 into Anarchid:fix/323-calibration-overhead Oct 26, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants