Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions changelog.d/thinking-binding-drop-block.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
- `thinkingBinding.dropBlock` in an agent's config asks the provider to drop
a signed thinking block whose earlier conversation has changed, as a fold
changes it, rather than refuse the request. An account that enforces
thinking-signature binding answers such a request with a 400; with the
setting, it drops the blocks that no longer match and answers. Where an
account doesn't enforce it, the provider decides for itself what to do with
such a block, and the setting has it drop them too. The setting is off by
default.
- **What's sent:** `thinking.block_binding.prefix_mismatch_behavior:
"drop_block"` with the `thinking-binding-controls-2026-08-01` beta, which
membrane's Anthropic and Bedrock adapters carry. It has no effect through
other adapters, or where the host sends a thinking config of its own,
through `request.extra` or `providerParams`, which replaces membrane's.
Any other model carries it only while thinking is on. A model whose
thinking is always on (Fable, Mythos) gets an adaptive thinking config
to carry it, with the agent's `thinking.display` when that's set, so on
such a model `display` takes effect only with this setting. membrane
notes that the Mythos family has refused thinking configs with a 400
before, so on a Mythos model the setting may fail every request.
- **What's reported:** on a framework turn, each call's dropped blocks, and
any shown to the model though they failed the check, appear in its
`inference:usage` trace as `thinking`. A call that dropped blocks, and
every later call of its stream, isn't used to calibrate the context
strategy's token estimates, since the provider doesn't bill a block it
dropped and the strategy still counts it. Where the strategy still sends
the blocks a fold left bound to a conversation the request no longer
holds, the provider drops them on every call until a later fold takes
them out of the context, and while they're sent, no stream calibrates and
the estimates keep the value they had. The direct API carries the setting
too, but reports nothing of it.
58 changes: 43 additions & 15 deletions src/agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -100,15 +100,16 @@ const DEFAULT_SAME_ROUND_THINK_TEXT_POLICY: SameRoundThinkTextPolicy = 'public';

/**
* Refuse an agent config whose `promptCaching`, `toolResultGuard`,
* `toolWrapperProseGuard`, `thinking.enabled` or `refusalHandling.autoRewind`
* holds anything but true or false (assertBooleanSetting), or whose `thinking`
* or `refusalHandling` is anything but an object, null or false
* (holdsSettings), or whose `autoContinue` resolveAutoContinue refuses.
* The constructor runs it first, so the rule holds wherever
* an Agent is built. `create()` also runs it for each of its agents before
* anything opens, and `createEphemeralAgent` before it reserves the name, so a
* refusal there leaves nothing behind. Membrane reads `thinking.enabled`, and
* the refusal is here because the setting is AgentConfig's.
* `toolWrapperProseGuard`, `thinking.enabled`, `thinkingBinding.dropBlock` or
* `refusalHandling.autoRewind` holds anything but true or false
* (assertBooleanSetting), or whose `thinking`, `thinkingBinding` or
* `refusalHandling` is anything but an object, null or false (holdsSettings),
* or whose `autoContinue` resolveAutoContinue refuses. The constructor runs it
* first, so the rule holds wherever an Agent is built. `create()` also runs it
* for each of its agents before anything opens, and `createEphemeralAgent`
* before it reserves the name, so a refusal there leaves nothing behind.
* Membrane reads `thinking.enabled`, and the refusal is here because the
* setting is AgentConfig's.
*/
export function assertAgentSwitches(config: AgentConfig): void {
const agent = `Agent "${config.name}"`;
Expand All @@ -124,6 +125,14 @@ export function assertAgentSwitches(config: AgentConfig): void {
'true asks the provider for extended thinking, and false or leaving it out doesn\'t',
);
}
if (holdsSettings(`${agent} thinkingBinding`, config.thinkingBinding, '{ dropBlock: true }')) {
assertBooleanSetting(
`${agent} thinkingBinding.dropBlock`,
config.thinkingBinding.dropBlock,
'true asks the provider to drop a signed thinking block whose earlier conversation changed rather than refuse the request, ' +
'and false or leaving it out sends nothing, so the account\'s default applies',
);
}
if (holdsSettings(`${agent} refusalHandling`, config.refusalHandling, '{ autoRewind: true }')) {
assertBooleanSetting(
`${agent} refusalHandling.autoRewind`,
Expand Down Expand Up @@ -208,6 +217,9 @@ export class Agent {
* blocks are preserved through Chronicle.
*/
readonly thinking: AgentConfig['thinking'];
/** Whether requests ask the provider to drop signed thinking whose earlier
* conversation changed (see AgentConfig.thinkingBinding.dropBlock). */
private readonly thinkingDropBlock: boolean;
/** Refusal auto-rewind policy (see AgentConfig.refusalHandling). */
readonly refusalHandling: AgentConfig['refusalHandling'];
readonly toolResultGuard: ToolResultGuard;
Expand Down Expand Up @@ -347,6 +359,7 @@ export class Agent {
this.maxTokens = config.maxTokens ?? 4096;
this.temperature = config.temperature;
this.thinking = config.thinking;
this.thinkingDropBlock = config.thinkingBinding?.dropBlock === true;
this.refusalHandling = config.refusalHandling;
this.toolResultGuard = new ToolResultGuard(config.name, contextManager, config.toolResultGuard === true);
this.proseRouting = config.proseRouting ?? 'locus';
Expand Down Expand Up @@ -1498,20 +1511,21 @@ export class Agent {

/**
* The request fields that come from this agent's own settings: its model
* config, prompt caching and cache TTL, prefill message, provider
* parameters and participant name. Every request the agent sends takes
* them from here, a framework turn's (assembleActivationRequest) and the
* direct API's (runInferenceWithInjections), so a setting reaches both or
* neither.
* config, thinking binding, prompt caching and cache TTL, prefill message,
* provider parameters and participant name. Every request the agent sends
* takes them from here, a framework turn's (assembleActivationRequest) and
* the direct API's (runInferenceWithInjections), so a setting reaches both
* or neither.
*/
private requestSettings(): Pick<NormalizedRequest,
'config' | 'promptCaching' | 'cacheTtl' | 'prefillUserMessage' | 'providerParams' | 'assistantParticipant'> {
const thinking = this.requestThinking();
return {
config: {
model: this.model,
maxTokens: this.maxTokens,
...(this.temperature !== undefined && { temperature: this.temperature }),
...(this.thinking !== undefined && { thinking: this.thinking }),
...(thinking !== undefined && { thinking }),
},
promptCaching: this.promptCaching,
cacheTtl: this.cacheTtl,
Expand All @@ -1521,6 +1535,20 @@ export class Agent {
};
}

/**
* The thinking config a request carries: the agent's own, with the binding
* control added when `thinkingBinding.dropBlock` asks for it. A model whose
* thinking is always on takes the control whatever `enabled` says, so an
* agent with no thinking config of its own still sends one, holding only
* the control and `enabled: false`. Membrane sends nothing from it to any
* other model, since thinking is off there.
*/
private requestThinking(): NonNullable<NormalizedRequest['config']['thinking']> | undefined {
if (!this.thinkingDropBlock) return this.thinking;
const own = this.thinking !== null && typeof this.thinking === 'object' ? this.thinking : { enabled: false };
return { ...own, blockBinding: { prefixMismatchBehavior: 'drop_block' } };
}

/** The request `assembleActivationRequest` builds (see there), for preview
* and debug tooling: what this agent's next activation would send. */
async buildActivationRequest(
Expand Down
31 changes: 27 additions & 4 deletions src/framework.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18826,6 +18826,9 @@ export class AgentFramework {
cacheCreationTokens: 0,
cacheReadTokens: 0,
};
// Whether a call of this stream reported signed thinking the provider
// dropped: from then on none of its calls is a calibration sample.
let thinkingDroppedInStream = false;
// Output tokens billed since this turn's assistant blocks were last
// persisted. The residual over the round's visible blocks is the price of
// its hidden thinking, stamped on the signed blocks at persist time
Expand Down Expand Up @@ -20493,11 +20496,28 @@ export class AgentFramework {
// agent's samples can fall out of its band for good
// (context-manager #323). A strategy from before then ignores
// the second argument.
// A call whose report names thinking the provider dropped billed
// less than the request held: the strategy still counts those
// blocks, so the call's ratio would pull its calibration low. The
// strategy takes the first call it's given after a compile as
// that compile's one sample, and each later call of the stream
// carries what the stream appended, so once a call has dropped
// blocks, none of the stream's later calls is given either. A
// compile whose first call dropped blocks gives no sample, and the
// calibration keeps its value. If the provider drops an earlier
// model's blocks after a model switch (`model_binding_mismatch`),
// they drop on every call until they leave the context, and the
// calibration takes up the new model's tokenizer only after that.
const thinkingReport = event.round?.thinking;
const thinkingDropped = (thinkingReport?.dropped.length ?? 0) > 0;
if (thinkingDropped) thinkingDroppedInStream = true;
try {
strat?.reportRealInputTokens?.(
realTotal,
compiledRequest ? { system: compiledRequest.system, tools: compiledRequest.tools } : undefined,
);
if (!thinkingDroppedInStream) {
strat?.reportRealInputTokens?.(
realTotal,
compiledRequest ? { system: compiledRequest.system, tools: compiledRequest.tools } : undefined,
);
}
Comment on lines +20499 to +20520

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium severity. After a fold with dropBlock on, calibration can stay frozen for a long time, not just for one stream.

thinkingDroppedInStream holds back every sample of a stream once one of its calls reports dropped blocks. The new doc in src/types/agent.ts (the fold case) says that after a fold, every block after the rewritten stretch is dropped on every call until a later fold removes it. So every stream after a fold drops blocks on its first call and never gives a sample. Calibration then keeps its old value across many turns, possibly until the next fold. The new two-turn test says that keeping the skip on the agent 'would stop calibration for good', but with stream scope the practical effect after a fold is nearly the same. The context strategy's token estimates go stale exactly when folds are happening, and this is when estimate accuracy matters. A fix could correct the sample for the dropped blocks, or count only the blocks that were not dropped, instead of throwing the sample away.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fold case is now in thinkingBinding.dropBlock's doc and, for the low finding, in the changelog fragment (86300c9b). Both say the calibration keeps its value while the strategy still sends the blocks a fold left bound.

— Effie-2756

} catch { /* calibration is best-effort */ }

const kvSubmission = takeKvSubmission?.();
Expand Down Expand Up @@ -20525,6 +20545,9 @@ export class AgentFramework {
cacheCreation: event.usage.cacheCreationTokens,
cacheRead: event.usage.cacheReadTokens,
},
...(thinkingReport && (thinkingDropped || thinkingReport.mismatchAllowed.length > 0)
? { thinking: { dropped: thinkingReport.dropped, mismatchAllowed: thinkingReport.mismatchAllowed } }
: {}),
});
break;
}
Expand Down
60 changes: 58 additions & 2 deletions src/types/agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,9 @@ export interface AgentConfig {
* Extended thinking config. When `enabled: true`, the agent runs with
* Anthropic's native extended thinking; responses include `thinking` blocks
* with cryptographic signatures, and the API enforces `temperature: 1`
* (Membrane handles this). Omit or set `enabled: false` to disable.
* (Membrane handles this). Omit or set `enabled: false` to disable, except
* on a model whose thinking is always on (Fable, Mythos), which thinks
* whatever this says.
*/
thinking?: {
enabled: boolean;
Expand All @@ -162,10 +164,64 @@ export interface AgentConfig {
/**
* How thinking content is returned: 'summarized' (readable summary) or
* 'omitted' (empty thinking field, signature only). Fable 5 / Opus 4.7+
* default to 'omitted' — set 'summarized' to receive thinking text.
* default to 'omitted' — set 'summarized' to receive thinking text. A
* model whose thinking is always on (Fable, Mythos) is sent a thinking
* config only to carry `thinkingBinding.dropBlock`, so this reaches it
* only with that set.
*/
display?: 'summarized' | 'omitted';
};
/**
* How the provider binds this agent's signed thinking to the conversation
* before it. A host sets it from what it knows of the agent's model and
* route.
*/
thinkingBinding?: {
/**
* Ask the provider to drop a signed thinking block whose earlier
* conversation has changed, as a fold changes it, rather than refuse the
* request. An account that enforces the binding otherwise answers such a
* request with a 400. One that doesn't decides for itself: the four
* accounts tried on 2026-10-18 dropped such blocks silently, so there the
* model sees nothing different and the report is what's new, and one
* that would show such a block to the model is asked to drop it instead.
*
* It's sent as `thinking.block_binding.prefix_mismatch_behavior:
* "drop_block"` with the `thinking-binding-controls-2026-08-01` beta,
* which membrane's Anthropic and Bedrock adapters carry (Bedrock's
* acceptance of the beta hasn't been tried). On any other adapter it does
* nothing, so a host should refuse it there. A thinking config the host
* sends itself, through `request.extra` or as `thinking` in
* `providerParams`, replaces the one membrane builds, control included.
*
* On a model whose thinking is always on (Fable, Mythos), membrane
* carries it in an adaptive thinking config whatever `thinking.enabled`
* says, with `thinking.display` when that's set, so such an agent's
* `display` takes effect only with this setting. On any other model, it
* goes only in the thinking config membrane sends while thinking is on.
* Membrane tells the first kind by its id, which names `fable` or
* `mythos`. Opus 5.5, Haiku 5.5, Sonnet 5.5, Sonnet 5, Fable 5 and Fable
* 5.1 accepted the config on 2026-10-18. Mythos hasn't been tried, and
* membrane notes that the family has refused thinking configs with a 400
* before, so for a Mythos model the setting may fail every request.
*
* On a framework turn, each call's report of the blocks the provider
* dropped goes to its `inference:usage` trace, and neither that call nor
* its stream's later calls feed the context strategy's calibration, since
* the provider doesn't bill a block it dropped and the strategy still
* counts it. After a fold, each block after the stretch it rewrote is
* bound to a conversation the request no longer holds. Where the context
* strategy still sends such a block, the provider drops it on every call
* until a later fold takes it out of the context, and while such blocks
* are sent, no stream gives a sample and the calibration keeps the value
* it had. A strategy that leaves that thinking out at the fold sends none
* to drop. The direct API carries the control too, but reads no report: it
* traces no usage and feeds no calibration.
*
* Default false: nothing is sent, and the account's default applies.
*/
dropBlock?: boolean;
};
/**
* How to handle a model content-policy refusal (`stop_reason: refusal`).
* When `autoRewind` is on, a refused turn triggers an automatic rewind: the
Expand Down
15 changes: 15 additions & 0 deletions src/types/trace.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import type { ThinkingBindingEntry } from '@animalabs/membrane';
import type { ProcessEvent } from './events.js';
import type { ModuleProcessResponse } from './framework.js';
import type { SessionUsage } from '../usage/types.js';
Expand Down Expand Up @@ -193,6 +194,20 @@ export type TraceEvent =
type: 'inference:usage';
agentName: string;
tokenUsage: { input: number; output: number; cacheCreation?: number; cacheRead?: number };
/**
* The provider's report on the signed thinking of the call this event
* closes (membrane's `RoundReport.thinking`): blocks it dropped before
* the model saw them, and blocks it showed the model anyway because the
* check isn't enforced. `tokenUsage` beside it is the stream's running
* total, not this call's. The provider reports them for a request that
* carried the binding control, whoever asked for it:
* `thinkingBinding.dropBlock`, or a thinking config the host sends
* itself. Present only when the report names any block.
*/
thinking?: {
dropped: ThinkingBindingEntry[];
mismatchAllowed: ThinkingBindingEntry[];
};
})
| (TraceEventBase & {
type: 'inference:stream_resumed';
Expand Down
9 changes: 6 additions & 3 deletions test/agent-cache-ttl.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,17 @@ describe('the direct API sends the agent request settings', () => {

it("carries every field the agent's settings give a request", async () => {
const { agent, sent } = directAgent({
maxTokens: 1234, temperature: 0.3, thinking: { enabled: true, budgetTokens: 2048 },
maxTokens: 1234, temperature: 0.3, thinking: { enabled: true, budgetTokens: 2048 }, thinkingBinding: { dropBlock: true },
promptCaching: false, cacheTtl: '5m', prefillUserMessage: 'scaffold', providerParams: { reasoning: { effort: 'low' } },
});
await agent.runInference([]);
assert.equal(sent.length, 1);
const { messages: _m, system: _s, tools: _t, ...settings } = sent[0];
assert.deepEqual(settings, {
config: { model: 'test-model', maxTokens: 1234, temperature: 0.3, thinking: { enabled: true, budgetTokens: 2048 } },
config: {
model: 'test-model', maxTokens: 1234, temperature: 0.3,
thinking: { enabled: true, budgetTokens: 2048, blockBinding: { prefixMismatchBehavior: 'drop_block' } },
},
promptCaching: false,
cacheTtl: '5m',
prefillUserMessage: 'scaffold',
Expand All @@ -94,7 +97,7 @@ describe('the direct API sends the agent request settings', () => {
// too: an unset setting reaches both paths as the same resolved value.
const parityCases: Array<[string, Partial<AgentConfig>]> = [
['every setting set', {
maxTokens: 1234, temperature: 0.3, thinking: { enabled: true, budgetTokens: 2048 },
maxTokens: 1234, temperature: 0.3, thinking: { enabled: true, budgetTokens: 2048 }, thinkingBinding: { dropBlock: true },
promptCaching: false, cacheTtl: '5m', prefillUserMessage: 'scaffold', providerParams: { reasoning: { effort: 'low' } },
}],
['no setting set', {}],
Expand Down
6 changes: 5 additions & 1 deletion test/helpers/mock-membrane.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,11 @@ export class MockYieldingStream implements YieldingStream {
}

if (response.usage) {
this.events.push({ type: 'usage', usage: response.usage } as StreamEvent);
// A response may carry the round report its usage event closes
// (`round`, a test-only field), as membrane 0.6.0's yielding streams
// send it.
const round = (response as { round?: unknown }).round;
this.events.push({ type: 'usage', usage: response.usage, ...(round ? { round } : {}) } as StreamEvent);
}

if (response.toolCalls.length > 0) {
Expand Down
Loading
Loading