Repository navigation
feat(kv-unified): callerCacheMarkers, for cache markers the caller's request carries outside the messages - #422
Open
nissa-seru wants to merge 5 commits into
Conversation
added 5 commits
October 29, 2026 21:53
…utside the messages membrane's formatters doc tells a cm-owned caller to mark its system prompt in a beforeRequest hook. kv-unified couldn't tell that marker from its own: its accept keeps a call's cache reference only when the wire receipt's marker count equals the compile's boundaries, so with the hook's marker no call kept a reference and every compile weighed no cache costs; and once kv-unified's own four markers were in use, the hook's was a fifth, which membrane refuses under cm-owned. kvUnified.callerCacheMarkers (an integer from 0 to 3, 0 unless set) is how many markers the caller's request carries outside the messages. Only the caller knows what its hook adds, and a request at four of kv-unified's markers is refused before any receipt could say so, so the count is declared, not learned. - placeCacheMarkers places 4 minus that many: 3 - r history markers at the legal boundaries nearest to i/(3 - r) of the history, the last at its last legal boundary, and the tail end. At 0 that is the thirds and the tail end, as before. - reportKvUnifiedAccepted keeps the reference when the receipt carries the compile's boundaries plus the declared count. The line it logs once on a mismatch names the option. - The strategy refuses a value that isn't an integer from 0 to 3 when it is built, and each use reads it the same way. At 0 the placement, the accept and the guard are what they were, so a caller that declares nothing sends the same bytes.
…uilt The placement and the accept each read kvUnified.callerCacheMarkers from the live config, which holds the caller's kvUnified object by reference. So a change to that object between a compile and the accept of its request compared the receipt against a count the compile didn't place for, and a preview whose overrides replace kvUnified placed its markers as if none were declared, though the caller's request is the same. The strategy now reads the count when it is built, checks it there, and keeps it; placement and accept read what it kept. An unset count is 0, and null reads as unset, as for the other kvUnified options. The reading function that sat between AutobiographicalStrategy's doc comment and the class is gone, so the class has its doc again. Texts the option made false now say what holds: kv-unified places at most 4 minus the declared count, since targets can share a boundary and short history has fewer; it keeps the reference only when the wire carries its markers plus the declared ones; and the comments on placeCacheMarkers, the mismatch branch, reconcileKvUnifiedMarkerIndices and the render record no longer say kv-unified owns all four slots or that every wire marker is its own. The changelog says membrane's formatters doc gives the fifth marker's refusal, rather than stating it as shown. Tests: - The placement table's history entries differ in size, so its expectations follow the history's tokens rather than its entry count. - A declared count keeps the reference for 1, 2 and 3 declared, in both scenarios, and the reference's markers sit where their messages end. - Changing the config object after the strategy is built changes neither placement nor acceptance. - NaN, Infinity and a string are refused, beside 4, -1 and 1.5.
…the preview path The refusal formatted the value with JSON.stringify, which prints NaN and Infinity as null and throws on a bigint before the option's message is built. It now prints a string quoted and anything else as String gives it. The count is read when the strategy is built, and a preview's overrides don't change it, since it describes the caller's request; the types doc now says so. A test previews with overrides that replace kvUnified, without the count, with 2 and with 4, and each places as the built count does. The declared-count cases also check where the marks sit: the history's last boundary, while a history marker remains, and the tail end, as with none declared. null is taken as unset, as for the other kvUnified options, and a test says so. The changelog now says membrane refused the fifth marker's request, which a run with a hook marking the system prompt showed; the doc comments on placeCacheMarkers and the render record say what they mean.
…s tokens The fragment said evenly spaced, where placement puts each on the legal boundary nearest an even fraction of the history's tokens.
The kv-unified design's slot bullet and its configuration table said CM owns all four cache slots under cm-owned. A caller that declares markers with kvUnified.callerCacheMarkers holds some of them, so both now say so.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
kv-unified gains
kvUnified.callerCacheMarkers: the number of cache markers the caller's own request carries outside the messages, such as a membranebeforeRequesthook's marker on the system prompt. kv-unified then places at most 4 minus that many markers of its own, and keeps a call's cache reference when the request carries its markers plus that many. A caller that sets nothing gets the same placement and the same acceptance as before. Withmain9c587a8ba'ssrc, this branch's 11 new tests fail, and the other 10 in their file pass. With this branch'ssrc, all 21 pass. On a simulation of a host whose hook marks the system prompt,mainat9c587a8bakept no cache reference, and in a long conversation membrane refused its requests 61 times for a fifth marker. With the option set to 1, no request was refused, and every accepted call kept its reference (Tests).@antra-tess, we're asking you to review this before it's merged, since it changes how kv-unified places cache markers in the requests that carry a resident's context, where the caller sets
kvUnified.callerCacheMarkers.Problem
membrane's
docs/formatters.mdtells acm-ownedcaller to mark its system prompt in abeforeRequesthook ("ThroughMembrane, whose request'ssystemis a string, acm-ownedcaller marks its system prompt in abeforeRequesthook",:129at943b8812). agent-framework sendscacheMarkers: 'cm-owned'for kv-unified (src/agent.ts:2056atfd6d6a362c). A caller that follows that advice meets two things atmain9c587a8ba:src/strategies/autobiographical.ts:14839at9c587a8ba). The hook's marker is one more, so no call keeps a reference, and every compile weighs no cache costs.:15558–15560and the tail end, at9c587a8ba), and the hook's is a fifth. membrane refuses such a request undercm-owned.On our simulation with a hook (Tests),
mainat9c587a8balogged the mismatch once, and no call kept a reference. In a long conversation, membrane refused the requests 61 times as carrying 5 markers (maximum 4).Only the caller knows what its hook adds, and a request at four of kv-unified's markers is refused before any receipt reaches kv-unified. So the caller declares the count, rather than kv-unified learning it.
We know of no host that sets such a hook today. connectome-host builds its
Membranewith nohooks(src/index.ts:1086–1094atfab6ab93c6).Changes
src/types/strategy.ts:1390):KvUnifiedConfig.callerCacheMarkers, an integer from 0 to 3, 0 when unset (undefined or null), with its contract in the doc comment.src/strategies/autobiographical.ts:2547–2556): the strategy reads the count when it's built and keeps it, so a later change to the config object, or a preview's overrides, changes neither placement nor acceptance. Anything but an integer from 0 to 3 is refused there, with aRangeErrornaming the option, the value as given and the range.:15548–15575): kv-unified places 3 − r history markers, at the legal boundaries nearest to i/(3 − r) of the history's rendered tokens, the last at the history's last legal boundary, and marks the tail end. At r = 0 that's the same 1/3, 2/3, last boundary and tail end as atmain9c587a8ba. r = 1 gives 1/2, the last boundary and the tail end, r = 2 the last boundary and the tail end, and r = 3 the tail end alone. Targets can share a boundary, and short history has fewer, so these are at most. The guard against more markers than the budget allows checks against 4 − r (:15587–15591).reportKvUnifiedAccepted,:14850): a call keeps its reference when the receipt's markers number the compile's boundaries plus r. The reference holds kv-unified's own markers. The line kv-unified logs once for a mismatch now nameskvUnified.callerCacheMarkersand the r it expected (:14876–14883).placeCacheMarkers' doc and its kv-unified branch, the mismatch branch,reconcileKvUnifiedMarkerIndices' doc and the render record's doc) now say what holds.test/kv-unified-marker-boundaries.test.ts:345–463, 11 new cases): a placement table for r = 0 to 3 over history entries of different sizes, so it follows tokens rather than entry count (:350); 1, 2 and 3 declared markers keeping the reference, in two scenarios, with kv-unified's markers where their messages end (:372); the count read when the strategy is built, against a later change to the config object (:392) and a preview's overrides (:409); a count other than the declared one keeping no reference, with the line naming the option (:424); and 4, −1, 1.5, NaN, Infinity and a string each refused with aRangeErrornaming the option and the range, NaN and the string named as given, and 0, 3 and null taken (:451). The r = 0 cases' offset check moved into a helper,markerOffsets, which the declared cases share.docs/unified-solve-design.md): §5A.3's "CM owns all four slots" (:794) and the configuration table'scacheMarkersrow (:952) now say less any the caller declares.changelog.d/caller-cache-markers.added.md.mainnow: at 20:28 UTC on October 30,mainis atfb4cf1046, 8 commits past9c587a8ba, andgit merge-treeof this head with it gives no conflict.git merge-treeof fix(kv-unified): only a compile's first accepted call sets its cache reference #410 withmain9c587a8bagives no conflict, and with this head it gives three conflict regions. One is insrc/strategies/autobiographical.ts, the comment that opensplaceCacheMarkers' kv-unified branch, which both rewrite. Two are intest/kv-unified-marker-boundaries.test.ts, where both add adescribeat the file's end. fix(kv-unified): only a compile's first accepted call sets its cache reference #410 also adds a sentence to the slot-contract comment aboveplaceCacheMarkers, that kv-unified "holds all four slots on a compile's own request". That sentence merges without a conflict, though with this change kv-unified holds the four slots less the caller's declared count. On our simulation (Tests), with fix(kv-unified): only a compile's first accepted call sets its cache reference #410 merged into this change and membrane at feat(cache): float a marker on cm-owned tool-loop calls that ask for it antra-tess/membrane#239's70c4d2b0, a caller declaring one marker sent no request with more than four. Where the hook's marker and kv-unified's three filled the four slots, membrane's float on a tool loop's later calls took the slot of kv-unified's earliest marker. At feat(cache): float a marker on cm-owned tool-loop calls that ask for it antra-tess/membrane#239's head at 06:30 UTC on October 30,f184181640, membrane places that float undercm-ownedonly when the request or membrane's config asks for it (antra-tess/membrane'ssrc/membrane.ts:1538–1539there).git merge-treeof each with this head gives what one withmain9c587a8bagives: no conflict with feat(presentation): a window per manager, which the strategy view reads (#291 step 1) #350, feat(view): blocks with an audience, and the reader a manager compiles for #364, fix(autobiographical): a removed message is in no compression or merge request built after it leaves, and a summary that lost an end message keeps its place #369 and fix(select): send no summary that costs more than its messages raw #409, and with feat: folding-strategy migration utility (kv-stable ⇄ kv-unified) #94, RenderStats.nesting; the kv-overlap warning stops claiming a double render #130, chore(deprecate): ContextInjection and compile(injections) #140, fix(shards): a sharded body keeps each block in place and goes out as given, and window edges keep a tool call with its results #310 and fix(reset): send store order while a reset head fills, and keep a reset head's messages whole #315 the same conflicted files, with the same number of conflict regions in each.Tests
On Linux with Node 24.22.1, installed with
npm cifrom the lockfile, in a clean checkout of81cc29d49, this branch's third commit. This head,b91f1696f, changes only the changelog fragment anddocs/unified-solve-design.mdafter it.npx tscexited 0.81cc29d49:kv-unified-marker-boundariespassed 21 of 21.kv-unified-render-state(55),kv-unified-strategy-integration(33),strategy-persistence(57),derive(26),compression-recall-curve(139) andmerge-raw-pin-seam(13) passed whole.main9c587a8ba'ssrc/strategies/autobiographical.tsandsrc/types/strategy.tslaid over,kv-unified-marker-boundariesfailed exactly its 11 new tests and passed the other 10.Our simulation of a host. These runs used a rig of ours, not part of this pull request: connectome-host
191d4f2cthrough agent-framework8145085d06and membrane493eb153to a stand-in provider, with no billing. A preload standing in for abeforeRequesthook marked the system prompt at a one-hour lifetime. Recipes came from connectome-host's own kv-unified settings, withcallerCacheMarkers: 1added where set.main9c587a8ba, with the hook: in a short conversation, 19 requests, kv-unified logged its mismatch line once, and none of 18 accepted calls kept a reference. In a long conversation, membrane refused requests 61 times with "streamOnce: cache_control limit exceeded: 5 markers (maximum 4)", 10 requests reached the stand-in, and none of 5 accepted calls kept a reference.3280e9ad9, with the hook and the option set to 1: in the short conversation, 19 requests, all 18 accepted calls kept a reference, and 9 of 10 compiles weighed one. In the long conversation, 44 requests, none was refused and none carried more than 4 markers. 30 carried four, the system marker and three of kv-unified's. All 34 accepted calls kept a reference, and 31 of 32 compiles weighed one.3280e9ad9, with the hook and the option unset: as atmain, with the mismatch line now naming the option.4a15a8cbcwith fix(kv-unified): only a compile's first accepted call sets its cache reference #410's16e08f0c2merged in, keeping both sides' tests, joining the two comments where they conflict, and adding the caller's declared markers to fix(kv-unified): only a compile's first accepted call sets its cache reference #410's slot-contract sentence, with membrane at feat(cache): float a marker on cm-owned tool-loop calls that ask for it antra-tess/membrane#239's70c4d2b0. Therenpx tscexited 0, andkv-unified-marker-boundariespassed 26 of 26, this change's 21 and fix(kv-unified): only a compile's first accepted call sets its cache reference #410's 5. With the hook and the option set to 1, in the long conversation, 44 requests, none was refused and none carried more than 4 markers. Each of the 8 later calls of a tool loop carried the system marker, kv-unified's later two and one on its newest message, where the turn's first call carried the system marker and kv-unified's three. All 34 accepted calls kept a reference. In the short conversation, 19 requests, none carried more than 4. In its two turns with later calls, the first call carried the system marker and kv-unified's two, and each later call carried those and one on its newest message. With the hook and the option unset, membrane refused requests 61 times as atmain.main9c587a8baagainst this branch's first commit,a9ba3896e: all 158 requests were byte-identical, apart from the session's start time and a clock line, which differ between two runs of one commit too. context-manager's compile decisions were identical once timestamps were masked.From
3280e9ad9to this head,srcchanges only how the refusal names the value and three doc comments, and the tests, the fragment and the design doc change too. Froma9ba3896e,srcalso changes from reading the count at each use to reading it once, the refusal's wording, and more comments.Not verified
src/types/strategy.ts:1380–1389). No run measured what that costs.src/strategies/autobiographical.ts:14850keeps the reference. No test has one.changelog.d/caller-cache-markers.added.md.Desmond-2579 made the change and ran it, Oscar-1005 made its design calls and read the last fix round, and Beth-2709 checked that each part is needed. Gordon-2414 found the behaviour, on Nigel-2405's simulation. Fiona-2757 wrote this description.