Repository navigation
fix(select): send no summary that costs more than its messages raw - #409
nissa-seru wants to merge 3 commits into
Conversation
The hierarchical select never weighed a summary against the messages it covers. Outside the head and recent windows, a compressed chunk's messages went out raw only where a pin kept them raw, where their chunk wasn't compressed yet, where no chunk held them, or where no listed summary claimed them. So a summary whose recall pair costs more than all its messages sent raw still went out in their place, and a compile that would fit with the messages in its place was refused. With positioned recall pairs (the default) and without adaptive resolution, the select now weighs each summary, at every level, against all the messages under it sent raw, messages a pin keeps raw included at their full price (dearerSummaries). Where the summary costs more, it is left out, with every summary under it, and its messages go out raw. An L2 or L3 that costs more than its children as recall pairs, but less than its messages raw, goes out as before. The pair is priced at its estimate as emission sends it: the recall header and the answer, cut by maxMessageTokens where the cut applies. Emission charges a summary with a measured size at that size where it is larger, which only makes a replaced summary dearer still. Each message is priced at the larger of its stored and sent forms, so a summary that costs less than its messages whole isn't swapped for messages whose tool results pruning replaces; one that costs more than them whole can be, and the replaced results go out as pruning's markers. The choice reads no budget. The calibration multiplier moves it only by rounding, where maxMessageTokens cuts the summary or a message, and through the recent window, which the multiplier moves for the whole select. A summary is left as it was: - where a summary under it isn't marked as merged into it, as in a merge a failed store write stopped part-way, until the next open repairs it; - where a summary outside its own subtree also holds one of its messages, so no message goes out both raw and inside a summary; - where, read in the parts pairing repair reads a message in (cycleParts), a tool call among its messages isn't answered in the part right after it, a result there doesn't answer a call in the part right before it, or two results there answer one call; - where a tool call or result among its messages is answered by, or answers, a message in the head or the recent window, or a message another summary covers; - where any of its messages lies between a tool call and a result it holds neither of, in the order the request sends them (the head, then the middle, then the recent window), judged from the earliest call of the result's id. Sent raw, its messages can push the result past the six entries pairing repair looks ahead (anima-research#319), where repair stubs the call and drops the result. Any such span keeps the summary, whatever its length, so the check doesn't depend on that constant; - where a result among its messages has a call of its id outside the summary sent before it, other than a call in the last part of the message right before it with no call of that id earlier in that message. Sent raw, the result can come within that call's look-ahead, and repair can move it there and leave the call it answers a stub. Where a store's tool call ids repeat, the last two checks keep some summaries whose raw messages repair would pair the same way; those go out as before. The rule decides before anti-redundancy and the per-level budgets, which pick what they picked before, and the replaced summaries come out of the selection after the coverage repair, so the budget share a replaced summary held goes to no other summary. By the store's estimate, the request carries fewer tokens wherever the select would have sent the summary, and a compile that would fit with the messages in its place now fits. One case differs: where the select that leaves out the summaries over a window's messages didn't fit, so the compile took them and sent those messages twice, the savings can now let it fit. It is then chosen, as it would be at a larger budget, and that request can be larger than before, though within the budget. Where the select now sends an L1's messages raw in its place, an over-budget refusal from it doesn't name compression in clearedBy for adopting that L1 as a queued chunk's exact L1: it sends those messages raw whether or not the chunk is compressed, so adopting it frees nothing. compressionWorkQueued takes the refusing select's leftOut for this; everything else it counts is as before. The rule prices every summary the select lists, and builds one pass over the messages for the tool checks when a dearer summary reaches them, so its work grows with the store's length, not its square. A store where the select now replaces a summary it sent before sends different requests from its first compile after the upgrade. Sending a summary's messages raw costs more compile time than sending its pair, in the work every compile does per raw message after the select. test/dearer-summaries.test.ts pins, through compiles of real stores, the summaries replaced and kept in the shapes above, the budget and coverage cases, the pricing of pruned and capped messages, and the calibration's rounding. test/overbudget-diagnostics.test.ts pins the in-memory adoption count in three cases: an L1 the select sends beside a chunk with pinned messages, a dearer L1 it leaves out, and a dearer L1 it sends because another summary holds one of its messages. Its existing adoption tests now seed L1s cheaper than their chunks and assert that the adopted L1 goes out. Seven files under test/ seeded summaries that cost more than what they cover: six over one- or two-word messages, and test/overbudget-diagnostics.test.ts over messages of about thirty words. Those messages now carry enough words to cost more than the summary, so the tests keep testing what their titles say. The README's tick() bullet names the exception, and a .changed fragment describes the change. The adaptive selects and positionedRecallPairs: false are unchanged.
Anarchid
left a comment
There was a problem hiding this comment.
Nightly dual review · round R1 at 01c7b59 · reviewers: Codex gpt-5.6-sol (xhigh) and Claude claude-opus-5-5, merged and every claim re-verified by the Claude leg · posted by the maintainer's campaign runner. Verdict: COMMENT (the formal review event is left to the maintainer). Finding ids R1-n are stable; cite them in fix commits.
Reviewed at 01c7b59 against main 9c587a8
Verdict: COMMENT. The fix is sound, with no blockers or majors; one edge breaks a stated invariant, and the common case pays for a check it could skip
The rule does what it says. A dearer root summary and its subtree leave the selection. Coverage completion then sends their leaves raw exactly once, and the guards keep pairing repair from seeing anything new. The pricing errs toward keeping the summary. The suite and the red run match the description to the case. What's wrong sits around the edges of the rule:
- With
maxMessageTokensset, the rule and emission price a capped pair differently, so a compilemainaccepted can now be refused (R1-1). - The case a real summarizer mostly produces, where no summary is dearer, wasn't measured. There the rule adds about 30% to compile time for nothing, and a one-line early exit halves the rule's share (R1-2).
- The PR doesn't name the strategy it lands on hardest,
KnowledgeStrategy(R1-4). - "Fixes #407" closes an issue that this change leaves partly true (R1-3).
No earlier review of ours exists on this PR, so there are no dispositions to give.
Blockers
None.
Major
None.
Minor
R1-1. With maxMessageTokens set, the rule can swap a summary for messages that cost more than emission charges for the pair, so a compile main makes is refused.
weighedPairPriceprices the cut render where the cap cuts (src/strategies/autobiographical.ts:16675-16677). Positioned emission sends that render (:17376) but chargesrecallPairCost(:17391), which prices the uncut answer (:16669-16671,:16642).- The cap decides its cut by
estimateTextOnlyTokens, but budgets use the store's estimate. So where a summary is just over the cap by the cut's measure, the cut removes a few characters and adds the omission marker. The rendered answer then estimates larger than the uncut one. - Repro (seeded L1 of 801 characters, 277 tokens; two messages under it; a 3-token latest message;
maxMessageTokens: 200,overBudgetGraceRatio: 0, windows 0/1):maincharges the pair 283 and compiles at a budget of 286. Its returned messages estimate at 299 tokens.- At this head, with the two messages at 286, 290 and 294 tokens, the summary is replaced. The compile needs 289, 293 and 297, and refuses at 286.
- With the messages at 274–282 tokens, or at 298, head and
mainagree on whether it fits.
- The request this head sends is smaller than the one
mainsends. The refusal is new all the same, and it contradicts the changelog's "emission charges a summary … at that size where it's larger, which only makes a replaced summary dearer still" (changelog.d/dearer-summaries.changed.md:37-40). Under the cap, emission can charge less than the rule's price. - Exposure is narrow:
maxMessageTokensdefaults to 0, and the raw cost must fall in a window of about a dozen tokens per summary. - Fix: replace only where the raw price is under both what the pair sends and what emission charges for it. That means
min(weighedPairPrice(s), recallPairCost(s, buildRecallHeader(s))).- With that one change, the repro keeps the summary at 286–294 and compiles at 286, as
maindoes. It still replaces at 282. dearer-summariesandoverbudget-diagnosticsstill pass, 209/209.
- With that one change, the repro keeps the summary at 286–294 and compiles at 286, as
- Add a boundary test at budgets between the two. The only cap test (
test/dearer-summaries.test.ts:1045) has a wide margin. - Emission charging the uncut answer is a pre-existing bug on
main. It isn't this PR's to fix, but it's why this one bites.
R1-2. When no summary is dearer, the usual case with a real summarizer, the rule adds about 30% to compile time and changes nothing. An early exit halves the rule's share.
-
The rule works through every root summary the same way, every compile. It expands the root to its leaves, then sums
prunedForSend/cappedForSend/estimateTokensover all of them, and only then compares with the pair (:8503-8507). In a healthy store the pair is a small fraction of its leaves, so nearly all of that walk is wasted. -
The description measures #407's corpus (most L1s dearer) and synthetic stores with every summary or every tenth one dearer. It doesn't measure stores with none dearer.
-
Measured here, in one process on one store, rotating three implementations of
dearerSummariesover 12 rounds of 5 compiles each:- no rule, which stands in for
main; - this head's rule;
- the head's rule with an early exit.
The store came from a stub summarizer writing short summaries, so none is dearer. It used hierarchical defaults with
targetChunkTokens: 400. Machine load was about 5–8.messages (pyramid) no rule this head head + early exit the rule alone, head → early exit 2,000 (13 L3, 4 L2, 3 L1) 1.33 ms 1.71 ms (+29%) 1.50 ms (+13%) 0.36 → 0.22 ms 8,000 (55 L3, 2 L2, 3 L1) 7.54 ms 9.80 ms (+30%) 8.68 ms (+15%) 2.08 → 0.96 ms Medians. All three sent byte-identical requests.
- no rule, which stands in for
-
The rule's own time grows 5.8× for 4× the messages.
-
Fix:
- Price the pair first, and stop summing once the raw price reaches it. Have
weighedRawPricetake the pair as alimitand return as soon asprice >= limit.dearer-summariesandoverbudget-diagnosticspass unchanged with it (209/209). - What remains is re-expanding every root's leaves on every compile. A summary's leaves don't change once it's written, so a cache keyed on the summary forest would remove most of that too.
- Price the pair first, and stop summing once the raw price reaches it. Have
-
Please also add a none-dearer row to the description's compile-time table. That's the row most hosts will see.
R1-3. "Fixes #407" closes an issue whose subject this change leaves partly true.
- #407's title is "The hierarchical select sends a summary that costs more than its messages".
positionedRecallPairs: falseis the hierarchical select too, anddearerSummariesreturns at once there (:8344). #407 measured 858 and 986 compiles on that path sending such a summary, as well as 490 and 418 on the adaptive select. - The PR is clear that both are unchanged. But merging with "Fixes" closes the issue, and no other issue tracks either path.
- Fix: write "Part of #407", or open a follow-up for the legacy combined pair and the adaptive select and link it from "What this leaves".
R1-4. The strategy this lands on hardest isn't named, and hosts have no way to keep their summaries.
- #407's corpus ran
KnowledgeStrategy. That strategy forceshierarchical: true(src/strategies/knowledge.ts:48-50) and closes chunks at phase changes from two messages up (:183-186), so its L1s are the ones most often dearer than their chunks. - connectome-host sets
adaptiveResolutionfor autobiographical and frontdesk strategies by default, as #407 notes. So for hosts on its defaults,KnowledgeStrategyis where this change lands. Library users on this package's own defaults get it too. - Its summaries aren't only compression. Its prompts have the summarizer flag open leads with
[LEAD]and have merges preserve them (knowledge.ts:26-28,:216,:274-275). Nothing reads those flags but the agent, in context. - Under this change, a dearer L1's flags leave the context until a merge above it is sent. The raw messages carry the facts, but not the summarizer's note that they're unresolved.
- This may well be the right trade, but it should be a stated one. Please name
KnowledgeStrategyin the changelog fragment's audience. - Consider an option to turn the rule off (default on). Today the only ways back are
positionedRecallPairs: falseor adaptive resolution, and each changes the shape of the whole middle. - For
KnowledgeStrategytheclearedByhalf is a no-op, sincechunkPersistenceEnabledis false (knowledge.ts:63-65,autobiographical.ts:13490).
R1-5. The new tests can't see a leaf sent twice, or one sent out of order.
- The
senthelper collapses raw ids and summary ids intoSets (test/dearer-summaries.test.ts:95-97). So "its message goes out raw" assertions prove presence, not exactly-once. orderis read only by the head-reset case (:875). The runtime coverage check marks a bitmap (autobiographical.ts:9047-9058), so it doesn't catch duplicates either.- Phase 3c's
rawPlannedcheck is what keeps a leaf from going twice today (:17327), and nothing pins it. - Fix: keep raw provenance as a list. In the L1, L2 and L3 replacement cases, assert that each replaced leaf appears exactly once, that no leaf is also under an emitted summary, and that raw leaves keep store order.
Nits
R1-6. The rule leaves no trace. Nothing counts or logs a replaced summary, so an operator seeing raw messages beside existing summaries can't tell why. Compare acrossFallbacks in getStats() (:8519-8530). A dearerReplaced count, or one line in render stats, would do.
R1-7. holdersOf expands every summary, merged children included, to its leaves (:8366-8383). So it costs O(messages × pyramid depth) on the first dearer root of every compile, and it rebuilds the leaf→holder map that summariesAcross already builds in the same select (:8270-8286). "Not its square" holds. "Grows with the store's length" holds only for bounded depth. You could build ownership once from the live roots, or share the window rule's map.
R1-8. Doc accuracy:
- The changelog lists what moves the choice: calibration rounding, the cap, and the recent window. It leaves out the per-compile thinking binding.
recallAnswerCostprices the live render, and that drops a summary's reasoning carriers while thinking is bound (:14766-14767,:16637-16642,:8107). So a summary with carriers can be replaced in one compile and sent in the next. That adds no cache churn: its render already differs between the two. - "What this leaves" says adopting a stored-only exact L1 still names
compression"even where adopting it makes the request larger, as onmain: #408". After this change, a dearer stored L1 is left out once adopted, so adoption frees nothing rather than growing the request (:13500-13504,:13525-13527). Same defect, new shape. Worth a line on #408.
Verified fine
- Base and head. The head is the expected
01c7b59, andmainis9c587a8b, as the description says. There's one commit. Trial merges: #350 conflicts inREADME.mdonly. #364, #369 and the newer #410 merge cleanly. - Suite.
npm ci && npm run buildexits 0 on Node 22.22.1, and all 3,599 tests pass. - Red run. With this head's
test/onmain'ssrc/,tscexits 0.dearer-summariesfails 15 of 48 andoverbudget-diagnosticsfails 1 of 161, the namedclearedBycase, as stated. - Raw pricing. It is at least emission's: max of stored and
prunedForSend+cappedForSend.extra(:16682-16687) against:17411-17412. Pinned leaves count though they're already raw. Outside R1-1 the pair is priced at or under what emission charges, so replacing a summary never raises the budgeted total. - Coverage. Leftovers come out of
selectedSummariesafter coverage repair (:17203-17205). Phase 3c takes claimers fromlisted, which excludes them, so their leaves go tomiddleRaw(:17286-17340).alonechecks holders across every listed summary, merged and empty ones included, so no leaf is raw and summarized at once.wholekeeps a half-merged tree as it was. A leaf in the head or recent window, or one with no position (removed), keeps the summary (leavesOfreturns null). - Image strip. Without hysteresis the strip counts images newest-first by source position (
:18503-18509), so images newly sent raw in the middle are stripped before any recent one. With hysteresis, the set is replayed from the store alone. No recent image is stripped because of this change. - Window rule (#360). Its structure is intact, and in
takemode a summary across a window getsleavesOfnull. The disclosed case, whereleavecan now fit and send more thantakedid, is as described. - Tool shapes. The
straddled/shapedguards line up with thecyclePartsadjacency that pairing repair reads, andcappedForSendkeeps tool block types. Sol traced this. Claude read the guards but didn't trace them on its own. - Chunker. Neither chunker closes a chunk on a tool-use message (
:18244-18253,knowledge.ts:183-186). So the guard that keeps a summary while a neighbour's partner is held elsewhere seldom fires, and is not a common source of flips as neighbours compress. clearedBy. The in-memory change matches what the select does after adoption.dearerSummariesdoesn't readchunk.compressed, so its verdict is the same before and after.- Existing tests. The edited ones were re-seeded, not weakened. Only premise assertions were added, and the
recall-positioningrejection got stricter. The fragment's name and category follow CONTRIBUTING. - Contract. No exported type changed. The one signature change is an optional parameter on a protected method.
Method. Two reviewers. Sol (Codex gpt-5.6-sol, reasoning xhigh, read-only sandbox, no network) reviewed a detached worktree at the head against main, with the cited issues supplied as files. It ran about 7.5 minutes and 78 commands, using 5.92M input tokens (5.73M cached) and 27.6k output tokens (17.5k reasoning). Claude (Opus 5.5) ran the widest-first pass in parallel: build, the full suite, the red run, trial merges against open PRs, a cap-boundary sweep on both builds, a patched build to check R1-1's fix, and in-process benchmarks of no rule, the head's rule and an early-exit variant on one store. Every Sol finding was re-checked before adoption. Who caught what: R1-1 was Sol's, and Claude reproduced it with a boundary sweep and found the main under-charge behind it. R1-5 was Sol's. R1-7 started from Sol's complexity point, which Claude kept but downgraded: the description's claim is "not its square", which holds. Sol's point on stored-only adoption became R1-8's second bullet, because the PR already discloses it under #408. R1-2, R1-3, R1-4, R1-6 and R1-8's first bullet were Claude's. Both reviewers independently confirmed that coverage holds and that pricing errs toward keeping the summary. The pairing guards were Sol's to trace. What Sol couldn't reach: GitHub (other PRs, the issues beyond the supplied bodies) and running the suite, since its sandbox can't write.
Answers review R1 of anima-research#409: R1-1, R1-2, R1-4, R1-5, R1-6, and R1-8's point on the thinking binding. R1-1. With maxMessageTokens set, the rule priced a summary's pair at its render as cut, but emission charges the pair at recallPairCost: the recall header and the uncut answer. The cut render can estimate larger than that charge. Where a summary's messages cost more than the charge but less than the cut pair, the rule sent the messages raw, and a compile that fit with the summary was refused. A sweep with an 801-character L1 over two messages, maxMessageTokens 200, overBudgetGraceRatio 0, a head window of 0 and a recent window of 1 showed it: with the two messages at 288, 290 and 294 tokens, main compiled from a budget of 288, and 01c7b59 needed 292, 294 and 298. The rule now prices the pair at the lesser of the two, so a summary is replaced only where its messages cost less than both. Where the cut doesn't apply, the pair's estimate is never above the charge, since recallAnswerCost's cost is at least its estimate, so nothing changes there. R1-2. weighedRawPrice takes the pair's price as a limit and stops summing once the messages' price reaches it, since the comparison is decided there. Every message's price is at least 0, so no choice changes. No forest-keyed cache of leaves is added. R1-4. KnowledgeStrategy forces hierarchical: true and can close a chunk at a phase change once it holds two messages, so an L1 of its can cost more than the messages it covers. The .changed fragment names it, and what a replaced L1 takes out of the request: its [LEAD] marks for that span. The strategy keeps the L1. No option to turn the rule off is added. R1-6. getStats() gives dearerReplaced: how many summaries the hierarchical select's last run that wasn't a dry run, and that weighed them, left out because each cost more than its messages. That run's request sends their messages raw if it compiles. A summary whose summaries under it went too counts once. A dry run leaves it as it was. R1-8. The .changed fragment now says that under carrierPolicy 'full', a summary's recorded reasoning is priced where the model's thinking isn't bound and left out of its price where it is, so the choice for such a summary can differ between the two. It also names, as the exception to the calibration multiplier moving the choice only by rounding, a summary charged at its measured size (carrierPolicy 'full' with responseContent), whose charge the multiplier doesn't scale. compressionWorkQueued's doc says leftOut holds each summary the rule left out and every summary under it. Tests (test/dearer-summaries.test.ts): - R1-1: an L1 that maxMessageTokens cuts, over two messages that cost more than emission charges for its pair and less than the pair as cut. The fixture searches message sizes for that case and computes it from renderSummaryAnswer and recallPairCost, not from a compile. The L1 goes out. The compile with it needs no more than emission's charge for the pair and the latest message, and a compile at that budget, which can't hold the messages raw beside the latest one, fits with the L1. - R1-5: the L1, L2 and L3 replacement cases, and the case of a dearer L1 with a pinned message, check that each replaced message goes out raw exactly once, under no summary sent, and in store order. The sent helper keeps raw provenance as a list. - R1-6: dearerReplaced is 0 before a compile, 1 after a compile that sends one L1's messages raw, still 1 after a dry run that sends that L1, and 0 after a compile that sends it. The L2 and L3 cases read 1 for a parent replaced with the summaries under it. In test/overbudget-diagnostics.test.ts, the doc comment anima-research#409 added on a test helper's field is gone, and countsCompressionWork's doc comment is main's again.
One conflict, in test/fixtures/recall-envelope-fixture.ts: this branch pads the four seeded turns, and anima-research#411 names the latest turn's text as HIERARCHICAL_FIXTURE_LATEST_TURN_TEXT. The resolution keeps both.
|
Replying to this review. This round, at
Line numbers are at @Anarchid @antra-tess, we're asking one of you to review this pull request at Ezra-2423 made this round, and Oscar-1005 made the calls on it. Ambrose-2449 read it word for word, and our internal review read it once. Lottie-2661 ran the tests, probes and timings cited here and wrote this reply. — Lottie-2661 |
On the hierarchical select with positioned recall pairs, a compile now sends a summary's messages raw in its place where they cost less than the summary's recall pair, both as sent and as emission charges for it. It keeps the summary where sending its messages raw could send one twice or change how pairing repair pairs a tool call or result, and in the other cases Changes lists. A compile that fits with the messages in the summary's place no longer refuses: on #407's corpus at the full summary target, compiles with the select of this pull request's first head,
01c7b59, fit 1,828 and 3,110 times wheremainrefused, and never refused wheremaincompiled (Measured). It costs compile time. On stores where no summary costs more than its messages, this head's compiles took 1.11 times as long asmain's at 2,000 messages and 1.09 times at 8,000, by median over 12 rounds (Measured). On #407's corpus at the summary target and the default budget, with that select, requests carried 14.8% and 19.6% fewer tokens thanmain's, and compiles took 0.03 to 0.11 ms more process CPU each, on compiles of about 0.5 to 0.6 ms. The adaptive selects andpositionedRecallPairs: falseare unchanged.Part of #407.
Problem
#407 reports it at
main9c587a8b: a summary goes out in place of the messages it covers even when its recall pair costs more than all of them sent raw, so the request carries more tokens than those messages would, and a compile that would fit with them in its place is refused. In the issue's probe, an L1 whose pair cost 933 tokens went out in place of four messages that cost 456, and budgets of 984, 1,004 and 1,460 tokens threwOverBudgetError, where 984 is the request with those messages in the pair's place. At this head, the same probe sends those four messages raw in a request of 984 tokens: it compiles at a budget of 984, and refuses at 983.With this head's test files on
main'ssrc/ate464d5ad,test/dearer-summaries.test.tsfails 16 of its 50 cases. The first is "sends the messages raw in place of an L1 that costs more than they do, and fits a budget that L1 could not", at its check that the dearer L1's pair doesn't go out.test/overbudget-diagnostics.test.tsfails 1 of its 161, aclearedBycase this change adds, at its premise that the select sends a dearer L1's messages raw in its place (Tests).Changes
dearerSummariesinsrc/strategies/autobiographical.ts). Where the summary costs more, the select leaves it out, with every summary under it, and sends its messages raw. An L2 or L3 that costs more than its children as recall pairs, but less than its messages raw, goes out as before.maxMessageTokenswhere the cut applies, and what emission charges for it (recallPairCost). So a summary is replaced only where its messages cost less than both.weighedRawPricereturns its sum of a summary's messages once the sum reaches the pair's price (if (price >= limit) return price;). A message is priced at the larger of its stored and sent forms, so a summary that costs less than its messages whole isn't swapped for messages whose tool results pruning replaces. One that costs more than them whole can be, and pruning's markers go out in place of the results it replaces. In "doesn't move a summary's choice with the token calibration", a summary whose recorded size equals its messages' estimate goes out at token calibrations of 0.3, 1 and 3.5.cycleParts, since fix(pairing): send a tool result that answers a call in its own message #365), a tool call among its messages isn't answered in the part right after it, a result there doesn't answer a call in the part right before it, or two results there answer one call;clearedBy. Where the select now sends an L1's messages raw in its place, an over-budget refusal from it no longer namescompressioninclearedByfor adopting that L1 as a queued chunk's exact L1: the select sends those messages raw whether or not the L1 is adopted, so adopting it frees nothing. Everything else that count does is as onmain, and theOverBudgetClearedBydoc (src/adaptive/picker.ts) gains a sentence saying so.tick()bullet adds: "Without adaptive resolution, and withpositionedRecallPairs(true), a compile can send the chunk's messages raw instead, where the summary would cost more than all of them, and the same for a merged summary and all the messages under it."test/seeded summaries that cost more than what they cover:test/fixtures/recall-envelope-fixture.tsandtest/<name>.test.tsforfold-journal-autobiographical,pins,recall-positioning,recall-tools-substitutionandstrategy-persistence, over messages of one or two words, andtest/overbudget-diagnostics.test.ts, over messages of about thirty. Those messages now carry enough words to cost more than the summary, and five added assertions check, as a premise, that a summary goes out rather than its messages.test/overbudget-diagnostics.test.ts's adoption tests now seed L1s cheaper than their chunks and assert that the adopted L1 goes out, and itscountsCompressionWorkhelper passes the refusal's left-out summaries on to the count.getStats()givesdearerReplaced: how many summaries the hierarchical select's last run that wasn't a dry run, and that weighed them, left out because each cost more than its messages, whose messages that run's request then sends raw if it compiles. A summary whose summaries under it went too counts once.Against each open pull request but #350, a trial
git merge-treeof this head gives what one ofmain(e464d5ad) gives, at 04:20 UTC on October 30: no conflict with #364, #369 and #410, and with #94, #130, #140, #310 and #315 the same conflicted files, with the same number of conflict regions in each.With #350
@tel-0s, #350 conflicts with
mainate464d5adin eight files, each of which #411 changed, and with this head in those eight and in one region ofREADME.md: thetick()bullet, where this change adds the sentence above after "which takes the chunk's place in compiled context." and #350 rewrites theautoTickOnNewMessagesentence that follows it.Tests
CI's Build & Test runs the whole suite on this pull request, on Ubuntu with Node 22.12.0.
Each ran on Linux x86_64 with Node 24.22.1, after
npm ciinstalled membrane 0.8.0,@types/node22.19.15, TypeScript 7.0.2 and chronicle, andnode --testran the built files. Those at this head, and its red runs, ran on a copy of a clean checkout of5355eb45c, with chronicle 0.5.1. The 35 mutants and the existing tests the rule reaches ran at earlier commits of this change, with chronicle 0.5.0, as each says.tscexits 0, and these 17 files pass every case:dearer-summaries50 of 50 (new),overbudget-diagnostics161,strategy-persistence57,recall-tools-substitution16,fold-journal-autobiographical12,pins7,recall-positioning6, the 4 others that import the changed fixture (thinking-binding61,recall-envelope-truncation30,recall-envelope20,carrier-policy10),window-edge-summaries34,anti-redundancy-budget3 andmerge-quarantine-hole30, and, of the files fix(store): require chronicle ^0.5.1, which keeps a stored object's key order #411 changed,block-as-stored14,compression-recall-curve139 andtool-pruning-price37, eachtest/<name>.test.ts.test/onmain'ssrc/ate464d5ad,tscreports six errors, each where a test readsgetStats().dearerReplaced, whichmaindoesn't have, and still emits.dearer-summariesfails 16 of 50: the two cases that replace an L1, one of them over pinned messages; the L2 and the L3 replaced by their messages; the three budget and coverage cases; six of the shapes with tool calls that replace a summary; the dearer case of pruned results; "replaces an L1 between a result and a later call of its id, since a call after its result spans nothing"; and thedearerReplacedcase.overbudget-diagnosticsfails 1 of 161, "names nothing on the hierarchical path for chunks whose exact L1 in memory costs more than their messages, since the select leaves that L1 out after adoption as before", at its premise. The other 15 files pass every case.test/on01c7b59'ssrc/,tscreports the same six errors and still emits.dearer-summariesfails 4 of 50: "keeps an L1 that maxMessageTokens cuts where its messages cost more than emission charges for its pair, so a budget that fits that charge compiles", at its first check, where messages of 288 tokens cost more than emission's charge for the pair, 285, and less than the pair as sent, 298, and the L1 doesn't go out; thedearerReplacedcase; and the L2 and the L3 replaced by their messages, at their counts. The new checks that each replaced message goes out raw once, under no summary sent, in store order, pass there in the L1 and pinned-L1 cases.overbudget-diagnosticspasses 161 of 161.src/differs from01c7b59's only in comments and whosedearer-summariestest file is01c7b59's. Each was built and run againstdearer-summariesand fails from 1 to 14 of its cases. Three of theclearedBycount, at a commit whosesrc/andtest/are01c7b59's, run againstoverbudget-diagnostics: ignoring the left-out summaries fails 1 case, never counting such an adoption fails 6, and a price check in their place fails 1. Five more, at this head, run againstdearer-summariesandoverbudget-diagnostics: pricing the pair as sent alone fails themaxMessageTokenscase above alone; never settingdearerReplacedfails thedearerReplacedcase and the L2 and L3 cases; setting it on dry runs too fails thedearerReplacedcase alone; counting every summary a replacement leaves out fails the L2 and L3 cases; and summing every message past the pair's price fails none.overbudget-diagnosticspasses under each.01c7b59's tree passed, 3,599 tests in all, and the line numbers below are at01c7b59. Besidesdearer-summariesand the threeclearedBycases this change adds, the rule leaves a summary out in these existing tests, each of which passes. Intest/overbudget-diagnostics.test.ts, it's the cases at lines 1531, 1791, 2061, 2243, 2292, 2317, 2350, 2378, 2514, 2761, 2805, 2952, 2963, 2981, 3004, 3023, 3095, 3182 and 3214, whose bodies this change doesn't edit, though 2317 and 3095 call thecountsCompressionWorkhelper it edits, and those at 2094 and 2114, which it edits. Intest/merge-quarantine-hole.test.ts, which it doesn't edit, it's those at 341, 703, 730 and 791.Measured
On stores where no summary costs more than its messages. Each store was built once, with a stand-in summarizer writing short summaries,
hierarchical: trueandtargetChunkTokens400. Each version ran in its own Node process on its own copy of the store: 3 compiles to warm up, then 25 timedcompile()calls, of which it took the median. The three versions took turns in a rotating order over 12 rounds per size, and all three sent the same messages, by a hash of each one's compile. The machine was shared: its one-minute load ran from 57 to 71 on 12 cores during the run, and the run waited for a CPU 7.3% of its time, 8.9% in its worst minute, with 0.3% of its time stalled on I/O. Medians over the 12 rounds, in ms:main(e464d5ad)01c7b59main:01c7b59/ this headThis head's compile took less time than
main's in 5 of the 12 rounds at 2,000 messages, and in 2 at 8,000.The rest of this section was measured before this round, against
mainat9c587a8b, at01c7b59or at an earlier commit whose select is01c7b59's, and calls each of those01c7b59.On #407's corpus (its setup is in the issue). These compiles ran at an earlier commit of this change against
9c587a8b'ssrc/, with a few added lines that record the rule's choices. Its select is01c7b59's, without this round's two changes to it, the lesser of two prices and the early stop. Its count for a refusal'sclearedByalso priced the L1s it counted, which this head's doesn't. Each part is 108 stores compiled at every step of their growth at five budgets. Tokens are summed over the compiles, by the store's estimate.main's, default budgetmain's, default budget01c7b59refuses wheremaincompiles, at any budget01c7b59makes wheremainrefusesCompile time on the corpus, at the default budget, measured over two slices of three stores each in each part, one process at a time. The full target's rows ran at loads of 1.6 to 3.2, and the quarter's in an earlier run whose load was 3.1 at its start and 8.8 at its end:
main→01c7b5901c7b59againstmainmaxMessageTokens10,000maxMessageTokens10,000Process CPU is
process.cpuUsage()around each compile, with garbage collection and no profiler. Profile time counts CPU profile samples in compile frames, so it leaves garbage collection out.On this corpus
maxMessageTokens10,000 sends what the default sends, to the token, at every budget, so its rows measure the same sends a second time. Their figures moved by up to 20 points from the default rows' in process CPU, and 23 in profile time. Why the rows with every call answered come out near level isn't traced.The choice reads no budget. At every step, the compiles at the five budgets that listed the same summaries over the same messages left out the same summaries, in all 14,454 and 14,250 such groups at each size.
fix(select): leave out summaries over messages a window sends raw when a dry run without them fits #360's window rule fell back to taking a window's summaries in the same 12 compiles at
mainand at01c7b59, at the full target with calls never answered. It fell back at neither alone.With
positionedRecallPairs: false, and on the adaptive select, at a quarter of the target, every compile sent whatmain's sent.With
maxMessageTokens10,000, at a quarter of the target, every figure above is the quarter's.On long synthetic stores, of turns of a question, a tool call, its result and a reply, with unique ids, and an L1 over every eight messages. In each store either every summary costs more than its messages or every tenth does, and
01c7b59replaces each of those. Each store was built once, and one process compiled it withmain's strategy code and01c7b59's in turn, six timed rounds after a warm-up, timingcompile()alone. Medians in ms, with the one-minute load at each row's end:main01c7b5901c7b59's compile took 1.9 to 2.5 timesmain's. At one in ten, the differences were 16, 28 and 18 ms.01c7b59took, the rule took 201, pairing repair (enforceToolPairing) 44 and the rest of the select 31. Outside the select, provenance attribution (src/compile-provenance.ts) took about 98 more, garbage collection 48, and everything else 18 fewer.01c7b59sent about 16,000 raw messages there wheremainsent 1,999 recall pairs.What this leaves
positionedRecallPairs: falseare unchanged. On The hierarchical select sends a summary that costs more than its messages, and refuses compiles those messages would fit #407's corpus at a quarter of the target, both sent summaries counting more tokens than their messages, and this change leaves that as it is.main.compressionfor adopting an exact L1 only the store holds, as onmain(An over-budget refusal names compression for adopting a stored L1 that makes the request larger #408). Where that L1 costs more than its chunk, adopting it now frees nothing, where atmainit made the request larger. In An over-budget refusal names compression for adopting a stored L1 that makes the request larger #408's probe at this head, a refusal at 671 tokens namedcompressionfor two such L1s, and aftertickadopted them, a compile at 100,000 tokens sent the same 672 tokens of raw messages as before, where An over-budget refusal names compression for adopting a stored L1 that makes the request larger #408's run at9c587a8bsent 790.Not verified
mainbecause the select leaving out a window's summaries didn't fit, this change's savings can let that select fit, by this change's fragment, and that request can be larger thanmain's, though within the budget. No test in the change asserts which of the window rule's two selects ran. On The hierarchical select sends a summary that costs more than its messages, and refuses compiles those messages would fit #407's corpus the window rule fell back in the same compiles atmainand with this change, at every size and budget measured.01c7b59's: neither ran at this head for this description.main, and01c7b591.18 and 1.28 times. Those two differ in all of this round's changes tosrc/, the lesser of two prices, the early stop and the count, and in fix(store): require chronicle ^0.5.1, which keeps a stored object's key order #411's changes tosrc/, which are to comments.changelog.d/dearer-summaries.changed.mdandchangelog.d/dearer-summaries.added.md.Ezra-2423 made this change, and Oscar-1005 made the calls on it. Beth-2389 found what #407 reports and built its corpus. Cordelia-2434, Ambrose-2449 and Humphrey-2407 each wrote the least design they would accept before reading the change, then weighed each of its parts against what it costs. Cordelia read its early rounds word for word, Ambrose every round from the fifth on, and Humphrey the last two before it opened. Our internal review read the whole change once, at an earlier version, then the changes of each of thirteen later rounds, the last of them this round, which fixed what we judged worth fixing from that read. Lottie-2661 ran the runs at this head, the red runs and the timings on stores where no summary costs more, and wrote this description.