Merge main into the calibration fix: the observer's calibration records at its new outcomes

README.md

@@ -84,18 +84,24 @@ How it works, with each option's default in parentheses:
 - With `adaptiveResolution: true` (false), each compile chooses how deeply to fold each part of history to fit the budget. `foldingStrategy` names the solver: `flat-profile` (the default), `oldest-first`, `kv-stable` or `kv-unified`. `kv-unified` needs an explicit `kvUnified` policy.
 
 With a strategy that supports them, as this one does, the manager also offers:
-- `pinRange` and `markDocument` keep messages at their place in the conversation, raw by default. A pin that sets `level` or `maxLevel` holds its messages at that level, or no deeper than it, only with `adaptiveResolution` and the `kv-stable` or `kv-unified` solver. Under the other solvers, and without adaptive resolution, the pin keeps them raw. `listPins` and `unpin` manage the pins.
+- `pinRange` and `markDocument` keep messages at their place in the conversation, raw by default. A pin that sets `level` or `maxLevel` holds its messages at that level, or no deeper than it, only with `adaptiveResolution` and the `kv-stable` or `kv-unified` solver. Under the other solvers, and without adaptive resolution, the pin keeps them raw. Under adaptive resolution, a pin or mark on any part of a long message stored in parts covers the whole message, in what the agent is sent, in compression and in the summarizer's prompt alike, less a part locked with `lockChunk` outside the span its ids name. As with any pin, a chunk recorded before it keeps its messages, so compression can still summarize those parts while the agent is sent them raw. Under `kv-stable`, a pin with `level` (not `maxLevel`) holds whole each summary it reaches, so a locked part that shares a summary with a part the pin covers renders at the level the pin holds that summary at. A compile whose budget can't hold the raw pins beside the rest refuses with `OverBudgetError`; where the adaptive plan's middle is over the budget, its message and `diagnostics.rawPins` name each pin keeping raw tokens in that middle. `listPins` and `unpin` manage the pins.
 - `holdCompression` keeps messages out of compression until `releaseCompression`, or until an optional `timeoutMs` passes, for content you'll still edit. Holds live in memory, so a reopened manager starts with none, and a hold that's never released stalls compression of everything after it.
 - `searchSummaries`, `getSummary` and `getSummariesInRange` read the summaries, for memory tools an agent can call.
 - `setToolDefinitions` and `setSystemPrompt` give the summarizer's calls the agent's own tools and system prompt. A chunk whose messages hold tool calls or results waits to be compressed until tool definitions are set.
 - `getRenderStats` reports how much of the last compile was raw and how much was summaries, with counts for levels 1 to 3. It's `null` until a live compile has run since the strategy loaded its branch, which opening the store, `setStrategy()` and a branch switch each do; a dry compile or preview leaves it as it was. Each compile's result carries its own as `renderStats`, so a dry compile (`dryRun`) shows the composition of the request it built without sending it.
 
+For a model that binds each signed thinking block to the request it was minted in, pass `{ thinkingBinding: { binds: true, dropBlock, participant }, identity }` as compile's third argument, with `participant` the name the model's own replies are stored under and `identity` digests of the request's system text, tools, model and the rest of its head. The compile then leaves out the thinking such a model can no longer take, against a per-branch record of the last request the branch sent, and says what it newly left out in `thinkingStrip`. Once the branch has a record, a reply keeps its thinking at the compile after it's stored only if the host has noted it with `noteRequestSent(rows, identity, compileId)`, naming the `provenance.compileId` of the compile its stream was built from: as it sends each later round of a stream, the ids of the stored messages the request carried after the compiled ones, every round's so far, in order, and once the stream has ended, those and its last reply's, with `{ unstored: true }` for a row it can't vouch was carried or returned as stored. A host that doesn't call it loses each reply's thinking at the compile after it's stored, and is never refused for it. While one of a branch's requests is still streaming, compile for that branch with `binds` only with `dryRun`: a committing compile with `binds` makes that stream's later notes vouch for nothing (see `SelectOptions.thinkingBinding` in `src/types/strategy.ts`).
+
 Every option is documented on `AutobiographicalConfig` in `src/types/strategy.ts`.
 
 ## Branches
 
 Chronicle keeps the store's history, so a conversation can branch. `branchAt(messageId)` creates a branch as of a message, and `switchBranch(name)` moves to it. `fork()` creates a branch at the current head and switches to it. The strategy's state, summaries included, follows the branch.
 
+## Observing a manager's work
+
+Pass `observer` to `ContextManager.open` to be told, as it happens, of each compile (`compile`: compiled, refused or failed, its whole duration, the thrown error, and the compile's render stats), of each refusal context-manager itself makes of compression (`compression`: by a reason from `COMPRESSION_REFUSALS`), and of each calibration sample (`calibration`: by an outcome from `CALIBRATION_SAMPLE_OUTCOMES`). The manager hands the observer to its strategy too. Each callback is optional and called synchronously. What it throws or rejects never reaches the work: the manager logs a callback's first failure once through its diagnostics, under its namespace, and drops the rest, so a compile's result is the same with an observer as without one. Outside that is a returned promise no JavaScript can give a handler, such as one whose `constructor` or species can't be used, or a proxy around one: its rejection goes unhandled, which ends a Node process by default. A record's error and render stats are the objects the caller gets, so treat them as read-only. A refused compile's `OverBudgetError` names its stage in `stageId`, from `OVER_BUDGET_STAGES`. A record can hold text from the store, such as a refusal's message, which names pins, so choose what you keep or send from it.
+
 ## Inspecting and repairing stores
 
 Among other tools, `scripts/` holds ones for inspecting and repairing stores. For example, `inspect-store.ts` writes out a store's messages, summary tree and resolutions, `audit-topology.ts` checks that each summary covers a contiguous run of messages, and `repair-topology.ts` plans a repair for one that doesn't, which it applies with `--apply`. Opening a store takes its lock and can write to it, so run them on a stopped agent's store or on a copy. `scripts/README.md` shows how to compile a script and describes several of them.

changelog.d/323-calibration-request-overhead.fixed.md

@@ -5,16 +5,34 @@
   estimate doesn't, so a tool-heavy agent's multiplier learned its schemas
   as a per-token rate and folded history the budget had room for, or sat at
   1.00 rejecting every sample as out of band. A caller that passes nothing
-  gets the old comparison.
+  gets the old comparison, plus the compile's message-position injections
+  (below).
 - With the overhead declared, three out-of-band calibration samples in a row
-  whose implied multipliers agree within 10% are learned, outright, at the
-  clamp edge, instead of being rejected for good. At 1.00 the ratio and the
-  implied multiplier are the same number, so a store whose true ratio was
-  past 1.8, where every pricing-epoch discard restarts, could never learn.
-  Undeclared samples are rejected as before, and the rejection line says the
-  caller may be leaving out the system prompt and tools it sends.
+  whose implied multipliers agree within 10% of the first are learned,
+  outright, at the clamp edge, instead of being rejected for good. At 1.00
+  the ratio and the implied multiplier are the same number, so a store whose
+  true ratio was past 1.8, where every pricing-epoch discard restarts, could
+  never learn. Undeclared samples are rejected as before, and the rejection
+  line says the caller may be leaving out the system prompt and tools it
+  sends.
+- The sample also counts the compile's message-position injections
+  (`beforeUser` and `afterUser`), which the request carries and the window
+  estimate doesn't: `ContextManager.compile` passes their content to the
+  select as `SelectOptions.messageInjections`, and the select prices it with
+  the window. Left out, an agent whose injections are large beside its window
+  learned them as a per-token rate, or, with the overhead declared, at the
+  clamp.
 - **Host operators:** a multiplier learned before the caller declared its
-  overhead holds that overhead, and steps back down over about ten compiles
-  once samples declare it. To relearn in one sample, seed it as provisional
-  on the stopped store: `setStateJson('<ns>/autobio:calibration',
-  { multiplier, at, pricing, provisional: true })`.
+  overhead holds that overhead. Inside the calibration band, the first two
+  samples that declare it replace the multiplier with their mean when they
+  agree within 10%, and the EMA steps as before when they don't; either way
+  the record then carries `declaredOverhead: true`, so later declared
+  samples step by the EMA. (Out of the band, the three-sample run above
+  applies.) The first of the two is held in memory, so a restart between
+  them costs one more sample. An undeclared sample that moves the multiplier
+  writes the record without the flag. A store whose multiplier, above 1,
+  puts every compile over budget gets no sample to move it, as before: to
+  unwedge it, seed the record as provisional on the stopped store, and the
+  refused compile drops it to 1: `setStateJson('<ns>/autobio:calibration',
+  { multiplier, at, pricing: <current epoch>, provisional: true })`.
+  `RequestOverhead` is exported from the package.

changelog.d/328-fresh-calibration-provisional.fixed.md

@@ -0,0 +1,10 @@
+- A store with no calibration record, or a record with no finite
+  multiplier, starts from a provisional 1, as a discard's restart does
+  (#328). Its first sample inside the calibration band sets the multiplier,
+  declared or not: one sample, without the two-sample hold. Learned, a 1
+  stepped a fifth of the way per undeclared sample (nine applied changes,
+  each a reprice of every message and summary, to reach a true 1.4), and
+  took two declared samples, the first held. Until that
+  sample, render stats report `calibration.provisional: true`. A record
+  from a newer pricing epoch, or a read of the record that throws, still
+  loads 1 as before, not provisional.

changelog.d/context-observer.added.md

@@ -0,0 +1,38 @@
+- `ContextManagerConfig.observer` (`ContextObserver`): a host is told, as each
+  happens, of every compile, of each refusal context-manager itself makes of
+  compression, and of each calibration sample. The manager hands the observer
+  to its strategy in `StrategyContext.observer`. Each callback is optional and
+  called synchronously. What it throws or rejects never reaches the work: the
+  manager logs a callback's first failure once through its diagnostics and
+  drops the rest, so a compile's result is the same with an observer as
+  without one. Outside that is a returned promise no JavaScript can give a
+  handler, such as one whose `constructor` or species can't be used, or a
+  proxy around one: its rejection goes unhandled, which ends a Node process
+  by default. A record's
+  error and render stats are the objects the caller gets, so an observer
+  treats them as read-only.
+  - `compile(record)`, once per `compile()`: the strategy, `dryRun`,
+    `startedAt` on the wall clock (`Date.now()`), the whole compile's
+    `durationMs` on the monotonic clock, `outcome` (`'compiled'`, `'refused'`
+    for one of `COMPILE_REFUSALS`, or `'failed'`), the thrown `error` itself,
+    and a compiled compile's `renderStats`, the object its result holds.
+  - `compression(record)`: a `reason` from `COMPRESSION_REFUSALS`
+    (`'admission-rejected'`, `'topology-violation'`, `'empty'`, `'incomplete'`,
+    `'malformed'`, the split rung's caps, `'chunk-call-cap'` and
+    `'window-call-cap'`, and `'overlap-guard'`, a chunk refused because an L1
+    already covers part of it). A provider's refusal or error on a compression
+    call isn't one of these.
+  - `calibration(record)`, once per sample `reportRealInputTokens` takes: an
+    `outcome` from `CALIBRATION_SAMPLE_OUTCOMES` (`'out-of-band'`,
+    `'deferred'`, `'held'`, `'confirmed'`, `'replaced'`, `'stepped'`), the
+    `ratio`, and the `multiplier` after.
+- `OverBudgetError.stageId` names which stage refused, from the exported
+  `OVER_BUDGET_STAGES`, so a host needn't parse the message. The library's own
+  strategies give it through the constructor's new `stageId` option, whose
+  words open the message when no `stage` is given. The constructor's `stage`
+  is unchanged: a strategy's own words, which open the message exactly as
+  before. Given words alone, `stageId` reads `'strategy-defined'`, a new id;
+  given a `stageId` beside them, it reads that id; given neither, it's
+  `'picker-exhausted'`. Every message reads as before, and
+  `AutobiographicalStrategy`'s protected `overBudgetError` is still given each
+  stage's words as `stage`.

docs/adaptive-resolution-design.md

@@ -373,6 +373,14 @@ This case should be rare in practice with unbounded L_n: the picker can almost a
 > pressure, but any such eviction that would silently lose content instead
 > **refuses the turn** (`UncoveredDropError`) — pressure never ships silent
 > loss. The philosophy stands: surface the overage, let the host decide.
+>
+> *(Raw pins)* The refusal of a plan whose middle is over the budget names
+> each pin that keeps raw tokens there (`OverBudgetDiagnostics.rawPins`), with
+> the tokens it keeps. Under adaptive resolution a pin on part of a message
+> stored in parts covers the whole message, so a mark on a long message can
+> be what doesn't fit: whoever manages the pins can unpin it, which lets
+> compression fold it, or whoever sets the budget can raise it, where the
+> model's window allows.
 
 ### 3.11 Recent-window slide
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@animalabs/context-manager",
-  "version": "0.20.0",
+  "version": "0.22.0",
   "description": "Context management layer for LLM-based agents using Chronicle and Membrane",
   "type": "module",
   "main": "dist/src/index.js",

src/adaptive/index.ts

@@ -151,6 +151,7 @@ export {
   accountFrontier,
   type OverBudgetClearedBy,
   type OverBudgetDiagnostics,
+  type OverBudgetRawPin,
   type PickerInputs,
   type PickerResult,
   type PickerChunk,