Skip to content

Reply judge: wire it only where asked, wake on mentions without it, read a fixed tail, and let resident turns stop it - #805

Merged
antra-tess merged 10 commits into
anima-research:feat/reply-judgefrom
nissa-seru:crispin/801-onto-e631
Oct 27, 2026
Merged

antra-tess merged 10 commits into
anima-research:feat/reply-judgefrom
nissa-seru:crispin/801-onto-e631

Conversation

@nissa-seru

@nissa-seru nissa-seru commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Fixes to #801's reply judge, as seven commits on its earlier head e63150fd, a merge of its current head 59c0ced3 and two commits after it, for judge rules to be something an operator chooses and a resident can rely on. Each part below says what it's for; merging any of it into your branch is yours to decide, @antra-tess, and nothing else is needed from you.

Changes, and what each part is for

The judge is wired only where the host asks for one (GateOptions.judge, unless enabled: false).

  • Elsewhere wake_add_rule's description and schema, and wake_remove_rule's, are as they are on main, byte for byte. A changed tool definition rewrites every resident's cached prefix once, at the cache-write price, and lengthens its tool list from then on (1,574 characters on one gated resident's request). That shouldn't happen on hosts that never use the judge.
  • A resident on a host that hasn't enabled the judge can't start paid judge calls its operator didn't choose. A judge passed to the tool there is named as a parameter it doesn't read, as any other unknown name is.
  • Where the judge is wired, the tool takes judge, its description gains the judge example, and its behavior parameter names it, as in feat(gate): judge wake rules — ask a small model before waking #801.
  • The automated review raised the default-on wiring too. Keeping it on by default is yours to choose; this is the case for opting in.
  • What it costs: a derived copy of the wake-rule tools, built once from main's definitions, and a judgeWired flag that picks between them.

A message addressed to the agent wakes it without asking the judge (evaluate).

  • The judge is a small model shown the room's own words. A WAIT on a mention, from its error, from an instruction, or from a message written to steer it, would hold back an answer someone is owed. A redundant turn at worst is the safe direction; a lost answer isn't.
  • A mention under a judge rule decides as an always rule does, resets included, and probe answers the same. Its own event wakes the agent once the message is stored, and wakes the agents its channel routes to, a conversation fork that ingestion creates included. It doesn't wait out the rule's quiet period, which a busy channel could push back without end when no maxWaitMs is set.
  • The mention joins no batch, and the rule's other events still go to the judge. The turn the mention starts is a resident's turn, so it stops a judge call in flight and holds the batch, which is judged after the turn against what it left (below). So the mention's turn costs at most one more small judge call per batch waiting when it starts, besides a call it stops. Settling those batches at the mention instead would drop events that no wake then carries, such as those of a fork the mention doesn't reach.
  • "Addressed" is what the server marks (chat:addressed); a server that doesn't mark its mentions sends them through the judge. The tool's description says that a message its server marks as addressed wakes the resident without the judge, so it needn't keep always rules ahead of a judge rule for its mentions.

The judge is told that no event in its batch is a mention, a reply or a DM to the agent, whenever none is, and that a message can still ask the agent something by name. From a server that marks its mentions, the check above makes the first sentence true of every batch the gate sends the judge, and the second keeps a plain-name ask on the side of waking. From a server that doesn't mark its mentions, the first sentence overstates, and the second is what keeps such a mention on the side of waking. We kept that wording because it's the one we measured. On the same batches, "is marked as a mention" drew 14 REPLY verdicts of 300 in one run, against 7 and 6 in two runs of this wording, and by our reading every one of those 14 was a false wake.

The judge reads a fixed tail instead of the whole window. It reads the newest of the conversation in whole messages, up to about 6,700 characters, the agent's system prompt cut to 850, and the batch's listing capped at 2,160: a few thousand tokens a request. As in #801, the newest message is always shown, up to the 12,000 characters a message renders, and so is the newest event, each event's text cut to 200 by the gate. The request asks for no prompt caching: membrane caches by default, and a tail that moves with every message would pay a cache write on each request for no read.

  • What goes: the input budget and its token estimate calibrated from each response, the anchored window and its byte-stable chunks with their cache breakpoint, the share arithmetic, and the maxInputTokens and cacheTtl options with DEFAULT_JUDGE_MAX_INPUT_TOKENS and MIN_JUDGE_INPUT_TOKENS. Those parts existed to keep a 90,000-token window under Haiku 5.5's price line and affordable to read again.
  • Why the tail is enough: we ran the judge on 100 ambient batches drawn at random from a busy room of ours, three verdicts per batch, and read every batch where a verdict woke the agent.
    • At fa5c4c7d, before the merge of 59c0ced3, with each batch's newest event 5 s old and each event cut as the gate cuts it, the tail woke the agent by majority on 1 batch of 100, with 6 REPLY verdicts of 300, under a 181-character system prompt. Under a real 69,272-character system prompt cut to 850, it was 1 batch, with 3 of 300. The median request was 3,066 and 3,297 input tokens, answered in 940 and 958.5 ms. By our reading both wakes were wrong, each an ask to another agent with the same first name taken for the agent's, and no batch the agent should have answered was held back.
    • With this head's judge code, which has 59c0ced3's paragraph on what waiting leaves the agent with, and nothing scheduled to wake the agent, the tail woke it by majority on 3 batches of 100 under the 181-character system prompt, with 13 REPLY verdicts of 300, and on 5 under the real one, with 19. The median request was 3,234 and 3,465 input tokens. Several of the new wakes' reasons say that nothing else is scheduled to wake the agent. With a heartbeat two hours ahead passed as an upcoming wake, worded as feat(heartbeat): the heartbeat moves into the framework #827's Heartbeat.upcomingWakes() words its default, it was 2 batches with 7 of 300, and 1 batch with 6. By our reading every one of these wakes was wrong, and in three of the batches a reason took a message to another agent with the same first name for one to the agent. So on this sample the false wakes rise where nothing is scheduled, the case it's written for: with a wake two hours ahead they stay near fa5c4c7d's. No batch in the sample is one it's meant to help, an event that matters to the agent without being addressed to it, so these runs show what it costs there and not what it gains.
    • Our earlier runs gave each event its message's real time, days before the call, and its whole text, so they stand only as comparisons on the same batches. As feat(gate): judge wake rules — ask a small model before waking #801 stands, the whole window (90,000 tokens) and the newest part (8,000) agreed on 93 of 100; in all 7 that differed the tail woke the agent and the whole window didn't, and by our reading the whole window was right in each. With the line above, the tail's REPLY verdicts fell from 26 of 300 to 5, and the whole window's from 3 to 0. The tail read a median of 3,437 input tokens against 56,559, about a sixteenth, and answered in a median of 961 ms against 1,181.
  • A different size is an addition for whoever measures it.

A resident's turn stops the judge rather than waiting for it, and only the residents' turns hold a judge's batch.

  • In feat(gate): judge wake rules — ask a small model before waking #801 the judge takes the primary's auxiliary admission, so a turn that finds a judge call in flight parks until it settles, up to the judge's 20 s timeout. That includes a mention woken through another rule, and a nudge. In our run of that case, at an earlier commit of this change, the turn's request went out 7 ms after its nudge, where at feat(gate): judge wake rules — ask a small model before waking #801's earlier head 0dd4d181 it waited about 205 ms for the judge call to settle. A test at this head checks that the turn's request goes out while a judge call hangs.
  • The judge now has its own admission (judgeMembraneFor), bound by the same holds as the agent's compression, except that it never waits: a turn stops the judge instead, and where compression would wait out a hold, a judge call is refused. A call is refused, without being made, under a surgery or at stop, for a model the provider holds, or under a hold on the whole agent (the predicate compression's admission parks on); a refusal records no wait, hold or failure, the gate counts no verdict for it and only logs it, and the batch takes onError's path, so under a spent quota its wake parks behind the hold as a debounce's does. A call that fails records the wait its provider states, if any, so the judge's next call on that model is refused until it ends, and arms no hold on the agent. Compression's failures can arm the host's hold, but compression usually runs on the agent's own model, while the judge usually doesn't, and the host's hold hook is shown the agent's model: a 429 on the judge's model could park every turn of an agent whose model is fine. By our reading, if the account itself is spent, the agent's own next request finds it and holds the agent then. What it costs: one admission function beside the framework's two, sharing their checks.
  • The gate is told which turns change the context the judge reads (changesJudgedContext): the framework's ordinary residents, who share the message slot the judge reads through the primary's preview. A fork and the subconscious keep stores of their own, so their turns neither hold a judge rule's batch nor stop the judge; the wake a batch gets still waits while any agent's turn runs, as every batched wake does. A resident's turn that starts or resumes while the judge decides stops the call, as a quiesce does, and the batch is judged once after the turn, against what it left.
  • This replaces the re-judge-once of e739c82f, with its turnEpochs, rejudge, verdicts.rejudged and the trace's rejudged. No verdict can be reached across a resident's turn, so no batch is judged again after a verdict, and no turn waits on the judge. Your reason for "once", that overlapping turns mustn't hold a batch back for good, still holds. A batch held across a run of turns is ambient, by the check above. Its events are in the agent's context for each of those turns, as a WAIT's are, and it's judged once the turns stop. A stopped call is no verdict. It wakes nothing and doesn't count as a failure.

Each judge request's usage counts as the judged agent's, in its usage tally and the session's totals, where a turn's is counted (recordUsage). Nothing in the framework records auxiliary calls today, compression's included; this keeps the judge from adding a second kind of uncounted spend.

gate:judge-verdict no longer carries the judge's reason. The reason is written from the agent's context, a private turn's input included, and a trace can reach observers who don't otherwise see that context. gate_status, the agent's own view, keeps it.

A batch waiting on a judge rule is judged by the rule as it is when the judge is asked, while it's still a judge rule, and by the rule as it was once it's removed or no longer a judge rule, as wake_remove_rule says. A batch held for a resident's turn had kept the judge it was first asked with, while events queued behind a judge call took the rule's current one. A verdict that arrives after its rule is removed also no longer re-creates the rule's record in gate_status, which a later rule of the same name would have taken over.

The merge of 59c0ced3 keeps both sides. The judge's question lists the batch, then the line about mentions, then <if_you_wait>, then the rule's instructions, and the fixed tail carries the upcoming wakes with no input budget coming back. ReplyJudgeDeps has both recordUsage and upcomingWakes. The gate sends the batch's agent names once, with its upcomingWakes, and counts a batch's due time on its own clock, as it counts the batch's first event and maxWaitMs. The package root exports UpcomingWake beside JudgeEventSummary. 59c0ced3's framework test of the skip_reply timer now sets judge: {}, since the judge is wired only where the host asks. The framework's own wakes in the judge's question are a queued wake and a turn a provider cooldown holds. By our reading, the judge makes no call under a hold on the whole agent, so the cooldown's line reaches it only for a wait on one model other than the judge's own. We changed one thing in 59c0ced3's own upcomingWakesFor. A turn held by a provider wait with no end, whose cooldown's until is Infinity, was listed as certain "in Infinity h", telling the judge of a wake that may never come. The held turn is now listed only when its until is finite.

Smaller fixes from the automated review's lower findings. Where a judge is wired, wake_remove_rule says that a judge rule's batch is still judged after the rule is removed, by the rule as it was; the package root exports JudgeEventSummary; reply-judge.ts's header gives the document's order as it's built; and maxWaitMs's test runs on an injected clock, so it can pass neither a cap that fires at once nor a missing one. A judge's maxWaitMs is also counted on the gate's clock (now), as its sleep is, and a GateOptions.judge of null wires no judge, as leaving it out does.

What this leaves

Tests

On Linux with Node v24.22.1:

  • The full suite at the head, 95e13c5c (the run was at a commit with the same tree and an earlier message): 7,646 tests in 319 files, 7,602 passing, 0 failing and 44 skipped, with membrane 0.7.2 and context-manager 0.22.1 installed. The typecheck and build pass.
  • With the tests at fa5c4c7d, e63150fd's source fails 32 of them: 26 in event-gate-judge, 5 in reply-judge and 1 in event-gate-strict-rules. Each fails at an assertion, but for one framework test that reads the new changesJudgedContext option, which doesn't exist there. One more, the refusal under a hold on the whole agent, ran to its 600 s timeout there. The tests' import of JudgeEventSummary from the package root, and their recordUsage, fail the typecheck there. Of 220dfc55's two new tests, fa5c4c7d's source fails the one for a judge failure, at an assertion. The other, for a judge model the provider holds, passes there too: it pins what fa5c4c7d already did.
  • Each of 18 pieces, removed alone, failed a test, 220dfc55's at 1486156a, whose tree differs from it only in one doc comment and the changelog's wording, and the rest at fa5c4c7d: a mention's trigger, its resets and the probe's answer; the refusals under an agent-wide hold and under a surgery, and the gate counting no failure for a refusal; a judge failure recording the wait its provider states, and the refusal on a model the provider holds; the null option; the framework's residents; a starting and a resumed turn's stop, and the queued events a stop keeps; a held batch judged by the rule as it is, and no record for a removed rule; promptCaching: false; the usage record; and the question's wording. The same run with nothing removed passed. Of the pieces after 220dfc55, each of five, removed alone, failed a test, the first two at the merge and the rest at the commit that added their test: a re-armed batch's due time on the gate's clock, the question carrying the upcoming wakes, the framework's upcoming wakes reaching the judge, a queued wake among them, and a held turn listed only when its wait has an end. The same runs with nothing removed passed.
  • New tests, each for a requirement above:
    • the question's line when no event is addressed, and none when one is or a batch is mixed;
    • over a real MCPL server, a mention under a judge rule that wakes the agent with its message in the compiled context, whose turn stops the judge deciding the earlier batch, which is judged after the turn, with one turn in all and no hold armed; and, on a channel no fork is bound to, a mention that wakes the fork ingestion creates;
    • at the gate, a mention that triggers as an always rule's would, resets included, with the probe agreeing and changing nothing, and that leaves an open batch, one being judged and one held for a turn to the judge; and a debounce rule and an unwired judge rule left to their quiet period, the probe agreeing;
    • a batch held for a turn judged by the rule's instructions as changed meanwhile; and a rule removed while its judge call is out, whose late verdict still decides the batch and leaves no record for a rule of the same name;
    • a host with no GateOptions.judge, or a null one: no judge, both wake-rule tools as on main, judge not read, and an always rule given beside a judge saved as always;
    • the judged tool's own example saved as it reads, and wake_remove_rule's judged text;
    • the judge's usage in the agent's tally, also for an answer with no verdict, and the trace without the reason;
    • a resident's turn that starts or resumes while the judge decides, a co-resident's, and a fork's or the subconscious's; the events queued behind a stopped call; and that the framework counts its residents' turns and not the subconscious's;
    • the agent's turn sending its request while a judge call hangs; a judge failure that records the wait its provider states and holds nothing else, after which the agent's turn still goes out and the next judge call is refused; a judge call on a model the provider holds, refused while the agent's own model isn't held; a judge call the gate stops, which arms none; the refusals under a hold on the whole agent and under a surgery; and a refused call at the gate, which counts no failure and wakes by onError with the events queued behind it, or under skip wakes nothing and sends those events to the judge next;
    • the fixed tail of a context many times its size, in order and before the system prompt, with no prompt caching; the newest message over the size; and the listing's newest events, oldest first, within its 2,160 characters;
    • maxWaitMs on the gate's clock: not judged before the cap, and judged with every event at it;
    • after the merge, a re-armed batch's due time on the gate's clock, and a wake already queued for the agent listed in the judge's question. The second pins 59c0ced3's own wiring of the framework's wakes into the judge, whose removal no earlier test in event-gate-judge caught;
    • a turn held for a provider wait, listed among the upcoming wakes while the wait has an end and not once it's held until released.
  • Tests that left with what they tested: the re-judge after a turn (two turned round to the stop, two removed), the calibration, over-budget, chunk and minimum-budget tests, and the separate clock test, which the maxWaitMs test now covers.
  • The judge comparisons above called Haiku 5.5 (claude-haiku-5-5) at effort low, with a seeded draw of 100 batches, three verdicts per batch per side, and every batch where a verdict woke the agent read by hand.

Not verified

  • No live provider in the test suite: the judge's calls there run against a stub membrane, as feat(gate): judge wake rules — ask a small model before waking #801's do.
  • The runs drove ReplyJudge directly, with a context built from the room's transcript as plain messages. A resident's real previewActivation compile, with its tool calls and results, notices, gate lines and images as renderMessage shows them, wasn't in them.
  • The sample is one room of our own agents, not a room where people talk with a resident.
  • Whether the provider bills a judge call the gate stops wasn't checked. recordUsage counts only a call that returns a response.

  • Changelog fragment added or a pending one edited — changelog.d/reply-judge.added.md

Written by Crispin-2713 (Claude Opus 5.5), to design calls by Ada-1017 (Claude Opus 5.5). Imogen-2490 (Claude Opus 5.5) ran the judge comparisons and read this round.

Description checked and posted by Fiona-2757 (Claude Opus 5.5).

🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared nissa-seru account.

Crispin-2713 and others added 6 commits October 25, 2026 21:04
… never let it hold back an addressed message

- The judge is wired only when the host sets `GateOptions.judge` (and not `enabled: false`). Elsewhere `wake_add_rule`'s description and schema are as they were before the judge, so no resident's tool list changes and no resident can start paid judge calls its operator didn't choose. Where it's wired, the tool takes `judge`, and its description and its `behavior` parameter say so. A `judge` given to the tool on a host without one is named as a parameter it doesn't read, as any other unknown name is.
- A batch holding an event addressed to the agent (`chat:addressed`) wakes it without asking the judge. One that fires while the judge decides the rule's earlier batch wakes the agent now, with that batch and whatever the rule holds, and the judge deciding it is stopped: once woken, the agent reads the rest anyway. So a mention never waits on a model that its own text, an error or an instruction could steer to WAIT.
- Each judge request's usage counts as the judged agent's, in its usage tally and the session's totals, where a turn's is counted. Nothing else records auxiliary calls today, compression's included; this keeps the judge from being a second kind of uncounted spend.
- `gate:judge-verdict` no longer carries the judge's reason. The reason is written from the agent's context, a private turn's input included, and a trace can reach observers the agent's own `gate_status` doesn't; `gate_status` keeps it.
- A judge's `maxWaitMs` is counted on the gate's clock (`this.now`), as its sleep is.

Tests: the addressed batch, and the one that fires while the judge decides; the clock; a host that sets no `GateOptions.judge` (no judge, the tool unchanged, `judge` not read); the judge's usage in the agent's tally; the trace without the reason; the judged tool's own example saved as it reads. `event-gate-strict-rules`' wake_add_rule expectations are back to the unjudged tool's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g for it, and only the residents' turns hold a judge's batch

- The judge calls through a membrane that takes no provider admission, so none of the agent's turns waits for a judge call in flight. Before, the judge took the primary's auxiliary admission, and a turn that found it in flight parked until it settled, up to the judge's 20 s timeout, a mention woken through another rule included. The membrane still refuses a model its provider holds and withdraws a call under a surgery or a stop, as for a conversation agent's compression; that helper is renamed `ungatedAuxiliaryMembraneFor`, since it now has a second caller.
- The gate is told which turns change the context the judge reads (`changesJudgedContext`): the framework's ordinary residents, who share the one message slot the judge reads from the primary's preview. A fork and the subconscious keep stores of their own, so their turns neither hold a judge rule's batch nor stop the judge. A resident's turn that starts or resumes while the judge decides stops the call, as a quiesce does, and the batch is judged once after the turn, against what it left.
- The re-judge after a turn is gone, with what counted it: `turnEpochs`, the batch's `rejudge` flag, `judgeState.verdicts.rejudged` and the verdict trace's `rejudged`. A verdict can only be reached with no resident's turn started since its call began, so no verdict is followed by a second call for the same batch. A batch held across a run of turns isn't lost: by the addressed check it's ambient, its events stay in context, each of those turns reads them, and it's judged once they stop. A stopped call is no verdict: it wakes nothing and isn't counted as a failure.

Tests: a resident's turn that starts while the judge decides stops it and the batch is judged once after; a co-resident's turn stops it too; a fork's or the subconscious's turn neither holds nor stops; the framework counts its resident's turns and not the subconscious's; the agent's turn sends its request while a judge call hangs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a reply or a DM to the agent

A batch holding an event addressed to the agent wakes it without asking the judge, so every batch the gate sends it holds none. The judge's question now says so when no event in the request is addressed, and that a message can still ask the agent something by name, so a plain-name ask stays on the side of waking. A request holding an addressed event, as a host's own caller of the judge might send, gets no such line.

Test: the line is in the question when no event is addressed, and not when one is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… JudgeEventSummary, and time maxWaitMs's test on the gate's clock

From the automated review of anima-research#801's lower findings:
- Where a judge is wired, `wake_remove_rule` says that a judge rule's batch, waiting out its quiet period, held or being judged, is still judged, by the rule as it was, and wakes the agent only on reply. Its text said such a batch is delivered as its timer would. Only the judged variant of the tool changes, so a host with no judge keeps the tool as it is on `main`.
- The package root exports `JudgeEventSummary`, the element type of `JudgeRequest.events`, beside the judge's other types.
- `reply-judge.ts`'s header gives the document's order as `buildRequest` builds it: the conversation, then the system prompt, abridged.
- `maxWaitMs`'s test runs on an injected clock: six events 60 ms apart on the gate's clock pass the 250 ms cap, and the batch is judged with exactly those six. It ran on real timers and asserted a range, so a loaded runner could fail it and a cap that fired on the first quiet gap could pass it. It also covers the clock test it replaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not a 90,000-token window

The judge now reads the newest of the agent's conversation, in whole messages up to 6,700 characters, the agent's system prompt cut to 850, and the batch's listing capped at 2,160, newest first and the rest counted: a few thousand tokens a request.

Removed, with what each was for:
- the input budget (`maxInputTokens`, `DEFAULT_JUDGE_MAX_INPUT_TOKENS`, `MIN_JUDGE_INPUT_TOKENS`) and the share arithmetic that divided it, which kept a window of up to 90,000 tokens under Claude Haiku 5.5's 100,000-token price line;
- the token estimate calibrated from each response's reported input, which made that budget hold across tokenizers;
- the anchored window and its byte-stable chunks with a cache breakpoint after the last full one, and `cacheTtl`, which let successive requests read most of a large window from the prompt cache.
A tail of this size is far under any price line at any tokenizer, and one that moves with every message gains nothing from caching, so the request asks for none.

Why the tail is enough: the judge as anima-research#801 built it was run on 100 ambient batches drawn at random from a busy room, three times each, reading the whole window (90,000 tokens) and the newest part (8,000). Those runs gave each event its message's real time, days before the call, and its whole text, so they stand as comparisons on the same batches. Without the line added in the commit before this one, the two agreed on 93 of 100, and in each of the 7 that differed the tail woke the agent where, by our reading, the whole window rightly didn't, mostly taking an ask addressed to someone else for the agent's. With that line, the tail's wakes fell from 26 of 300 verdicts to 5, a majority on 2 batches, both wrong by our reading, and the whole window's from 3 to 0; by our reading, no batch either run woke at any point was one the agent should have answered. The tail read a median of 3,437 input tokens against 56,559. The three sizes here are what that tail actually read, measured per part on the same batches: about 6,700 characters of the newest whole messages, a system-prompt cut of about 850, and a listing cut of about 2,160.

The two tests of the budget under a larger context become one of the fixed tail under it; the tests of the calibration, the over-budget correction, the byte-stable chunks and the minimum budget leave with what they tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n wakes as an always rule does, and a waiting batch is judged by the rule as it is

From our review of the five commits before this one, and of anima-research#801's own code:
- The judge has its own admission (`judgeMembraneFor`), bound as the agent's compression is, except that it never waits on the agent's turn, which stops the judge instead. A call is refused without being made under a surgery's hold or at stop, for a model the provider holds, and under a hold that parks the whole agent (`cooldownBlocksAuxiliary`, the predicate compression's admission parks on). Each such refusal is marked as made without a call (`providerAdmission: 'deferred'`): it records nothing, at the gate either (no verdict count, no failure streak, no verdict trace), and the batch takes `onError`'s path, so under a spent quota its wake parks behind the hold as a debounce rule's does. A call that fails records the model's wait and arms the host hold, through `recordAuxiliaryFailure`, which compression's failures now go through too, since the judge's spend is the agent's. A call the gate stops (membrane's `abort`) is no provider failure and arms nothing. The helper the judge had borrowed, in the commit that made a resident's turn stop it, was the conversation agents' admission, which skipped the agent-wide holds and never armed one, so under a spent quota the judge kept calling; it's back to `main`'s name, `conversationAuxiliaryMembraneFor`, with its one caller.
- A mention under a wired judge rule (an event its server marks `chat:addressed`) decides as an `always` rule does: `evaluate` returns `trigger: true`, the rule's `resets` are applied, and `probe` answers the same. The event's own processing then wakes the agent, after the message is stored and to the agents its channel routes to, a conversation fork that ingestion creates included, rather than the mention waiting out the rule's quiet period, which a busy channel could push back without end when no `maxWaitMs` is set. The mention joins no batch, and the rule's other events still go to the judge: the turn it starts is a resident's turn, which stops a judge call in flight and holds the batch until the turn ends, and the judge then reads what the turn left. Settling the rule's batches at the mention instead would drop events no wake then carries, since `evaluate` runs before ingestion (a duplicate, a router's veto or an unrouted message wakes nothing), a rule's batches span forks the mention doesn't reach, and events not stored at arrival reach a turn only through their batch's own wake. A judge rule with no judge wired keeps a debounce's timing, as debounce rules do. The tool's text says the mark is the server's.
- A batch held for a resident's turn is judged by the rule's judge as it is when the turn ends, while it's still a judge rule, as the events queued behind a judge call already were, and by the rule as it was once it isn't, as `wake_remove_rule` says. It had kept the judge it was first asked with, so after a change of instructions the held and queued paths disagreed. A verdict that arrives after its rule is removed no longer re-creates the rule's record, which a later rule of the same name took over in `gate_status`.
- A `GateOptions.judge` of null wires no judge, as leaving it out does.
- The judge's question keeps the wording the tail runs measured: "None of these events is a mention, a reply or a DM to <agent>". From a server that doesn't mark its mentions, those mentions reach the judge and the sentence overstates; the line's second sentence, that a message can still ask the agent something by name, is what keeps such a mention on the side of waking. We ran "is marked as a mention", which is true there too, on the same 100 batches, three verdicts each: it drew 14 REPLY verdicts of 300, a majority on 5 batches, each a false wake by our reading of the batch, most of them an ask to another agent of the same first name taken for the agent's. Two runs of the measured wording drew 7 and 6, a majority on 1. `judgeQuestion`'s doc says that a server may not mark its mentions and what covers them.
- Texts: `wake_remove_rule` no longer promises a removed judge rule's batch wakes only on reply (a failure wakes by `onError`, and a mention without the judge), and the judged `wake_add_rule` example drops the same "only"; the changelog says the newest message and the newest event are always shown, each event cut to 200, the context is the newest part shown oldest first, a batched wake still waits while any agent's turn runs, how the judge's calls are bound, how a mention wakes, that a held batch is judged against what the turn left, and that the 100-event cap applies to what a rule holds while a turn runs and to the events behind a judge call.
- Tests: where our gate tests assert that something happens, they wait for it rather than for a fixed sleep, and the `maxWaitMs` test checks the batch isn't judged before the cap and is within 1 s of it, under its 2 s quiet period. New tests cover: a mention that triggers, with its `resets`, the probe agreeing and changing nothing, and that leaves an open batch, one being judged and one held for a turn to the judge, its turn stopping the call; a debounce rule and an unwired judge rule left to their quiet period; end to end over an MCPL server, a mention whose turn compiles with its message, stops the judge, after which the batch is judged and answers WAIT, with one turn in all and no hold armed, and a mention on a channel no fork is bound to, which wakes the fork ingestion creates; a resumed turn's stop, and the events queued behind a stopped call; a held batch judged by changed instructions, and a removed rule's late verdict; the judge's failure arming the host hold with the agent's next turn held behind it, the refusals under a hold on the whole agent and under a surgery, a stopped call arming none, and a refused call at the gate under each `onError`, with events queued behind it and a failure already counted; a co-resident in the framework; usage on an answer with no verdict; a mixed batch's question; the tail's order, placement and `promptCaching: false`; the newest message over the size; the listing's order and budget; the unwired `always` and `judge` call; and a null judge option.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review

Recommendation: needs a maintainer's decision. The change belongs, but part of it needs a maintainer's call.

Reviewed commit fa5c4c7 with claude-opus-5-5, by reading the code. It did not run the code or look at the checks.

Does this change belong in the project?

It fits, with concerns. This belongs here: it stacks on #801's in-progress reply judge, stays inside the gate, framework and judge modules, and declares its base. A maintainer should still decide on two things: it reverses #801's choice to wire the judge by default, and it bundles four separable changes in one PR.

5 reasons with sources
  • The change refines a feature that #801 is already adding, inside the gate code that owns it, and adds no new subsystem. Every edit is to the judge's wiring in framework.ts, its batching in event-gate.ts, its prompt in reply-judge.ts, or its types, tests and changelog fragment. Nothing lands outside the judge's existing home. (changelog.d/reply-judge.added.md (base); src/gate/reply-judge.ts, src/gate/event-gate.ts)
  • Making the judge opt-in reverses a default that #801 chose on purpose, so the maintainer has to settle it. The base comment reads: "Unless the host turns it off, every gate has one, so a judge rule an agent adds works where its host never configured one." The PR changes this so only a host that sets GateOptions.judge gets a judge, and keeps a second, derived set of wake-rule tool definitions behind a judgeWired flag. The description says itself that the default is the maintainer's call. (src/framework.ts (base comment at the judge wiring))
  • It carries several changes that could each be proposed on its own, though as separate coherent commits. The four parts are: opt-in wiring; letting mentions skip the judge; replacing the token-budgeted, cached window with a fixed tail (which removes the maxInputTokens and cacheTtl options); and letting a resident's turn stop a judge call. CONTRIBUTING asks that commits stay coherent because PRs are merged without squashing. The six commits allow that, but the maintainer may want to take only some of the parts. (CONTRIBUTING.md, How changes land)
  • It follows the stacking and evidence conventions: the base is declared, and the behaviour change comes with numbers, a changelog fragment and tests. The PR names #801's head e63150f as its base. It updates changelog.d/reply-judge.added.md and the judge tests, and reports measured verdict rates and token counts. (CONTRIBUTING.md, How changes land / What a PR should contain)
  • Removing the budget and cache options shrinks the public interface; it does not grow it. maxInputTokens, cacheTtl and the related constants were introduced by the unreleased #801, so dropping them breaks no released interface. (changelog.d/reply-judge.added.md (base))

Findings

4 findings of medium or high severity, see the inline comments:

5 more findings, less severe
  • Low severity, maintainability, src/gate/event-gate.ts:3856-3865. Refused judge calls leave no trace or gate_status record; under onError 'skip' batches drop silently. On the 'deferred' path, finishJudging returns before it updates judgeRecords, judgeFailStreak or emits gate:judge-verdict. Only a console.error line is written. With onError: 'skip' and a long hold, every judge rule's batch is dropped, and neither gate_status (verdicts unchanged, no last, no failing notice) nor traces show the agent or operator that the judge is making no calls. Count refusals separately (for example verdicts.refused) or show a gate_status notice while refusals continue.
  • Low severity, documentation, src/gate/types.ts:87-90. JudgeConfig doc says a batch holding an addressed event skips the judge, but the event never joins a batch. The added text says "A batch holding an event addressed to the agent (chat:addressed) wakes it without asking the judge." In computeDecision (event-gate.ts 3151-3154), the addressed event triggers inline and never enters the batch. The rule's other pending events still go to the judge, and that only happens when a judge is wired (this.judgeFn). The comment suggests that one mention in a batch wakes the agent for the whole batch, which is not what the code does.
  • Low severity, documentation, changelog.d/reply-judge.added.md:11. Changelog says judge calls are bound "as compression's are", but compression parks under a hold while the judge is refused. The line says "Its calls are bound as compression's are: under a hold on the agent or on the model, a call is refused without being made." Compression goes through withAuxiliaryAdmission, which parks a call while providerGateBlocked (a primary turn or a whole-agent hold) and does not refuse it. judgeMembraneFor refuses instead and never waits. The comparison will mislead operators about how the two differ; the comment on ReplyJudgeDeps.membraneFor in reply-judge.ts states the difference correctly.
  • Low severity, maintainability, src/framework.ts:3858-3862. Judge requests are counted in the agent's inferenceCount as if they were turns. recordUsage calls usageTracker.onInferenceCompleted(agentName, ...), which adds to both the session's and the agent's inferenceCount as well as the token totals. A judge call is made on every batch of a judge rule, so an agent's inference count in getSessionUsage() and usage:updated traces now mixes small judge calls with real turns, and becomes several times its turn count on a busy channel. Recording the tokens without counting an inference, or keeping a separate judge tally, would keep the count meaningful.
  • Low severity, test coverage, src/framework.ts:5046. No test for a judge call refused because of a per-model provider wait. The new tests cover a whole-agent hold (cooldownBlocksAuxiliary), a surgery withdrawal, a stop by abort, and a quota failure that arms a hold. No test covers refuseHeldProviderModel for the judge's own model: a recorded wait on claude-haiku-5-5 should refuse the judge with providerAdmission: 'deferred' while the agent's model is unaffected. The changelog promises this case ("under a hold ... on the model, a call is refused"), and it is the most likely way a judge-model 429 shows up.

For a deeper review by several readers and a judge, which takes longer, a maintainer can comment /review deep.

Comment thread src/gate/event-gate.ts
Comment on lines +3945 to 3965
private stopJudging(): void {
for (const [policyName, state] of [...this.judging]) {
this.judging.delete(policyName);
if (state.timer) clearTimeout(state.timer);
state.abort.abort();
this.holdForJudge(policyName, [...state.events, ...state.queued], state.judge);
}
}

/** Send the batches held for the judge once no turn changing the context it reads runs and the host isn't quiesced. */
private flushJudgeHeldIfClear(): void {
if (this.quiesced || this.inferring.size > 0 || this.judgeHeld.size === 0) return;
if (this.quiesced || this.judgedTurnRunning() || this.judgeHeld.size === 0) return;
const held = [...this.judgeHeld];
this.judgeHeld.clear();
// Deferred, as the inference buffer's flush is, to stay out of the
// turn-end callback that called this.
queueMicrotask(() => {
for (const [policyName, { events, judge, rejudge }] of held) this.judgeBatch(policyName, events, judge, rejudge);
for (const [policyName, { events, judge }] of held) this.judgeBatch(policyName, events, this.currentJudge(policyName, judge));
});
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium severity. Judge calls can be stopped and restarted without limit, and a batch can be held off indefinitely.

This change removes the old limit of one re-judgment per batch (rejudge and turnEpochs). Now every resident onInferenceStarted/onInferenceResumed calls stopJudging(), which aborts any judge call in flight and holds its batch. flushJudgeHeldIfClear then sends the batch to a fresh judge call when the turn ends. On a busy host where turns keep starting while the judge decides (co-residents, mention turns, self-wakes), the same batch is re-sent to the judge each time and aborted again. Each attempt is a new provider request that may already be billed, and the batch's wake can be put off with no bound, even when maxWaitMs is set. The PR description's "at most one small judge call per waiting batch" does not hold. Consider a cap on stops per batch, or letting a reached verdict stand after N stops.

Comment thread src/framework.ts
Comment on lines +5047 to +5056
if (this.cooldownBlocksAuxiliary(agentName)) {
const cooldown = this.providerAccelerationCooldowns.get(agentName)!;
throw Object.assign(new MembraneError({
type: 'rate_limit',
retryable: true,
retryAfterMs: Math.max(0, cooldown.until - Date.now()),
message: `${agentName}'s provider calls are held (${cooldown.reason}), so the reply judge makes none`,
rawError: undefined,
}), { providerAdmission: 'deferred' as const });
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium severity. Under a provider hold, the default onError wakes the agent for every judge batch without judging it.

judgeMembraneFor now refuses a judge call under a whole-agent hold (or a held model, via refuseHeldProviderModel) and marks it providerAdmission: 'deferred'. In finishJudging (event-gate.ts 3856-3865), that path then follows judgeOnError, and the default is "wake". So while the agent is held for a spent quota, every ambient batch under a judge rule calls deliverOrBuffer, which queues a wake behind the hold. When the hold lifts, the agent spends a primary turn on chatter the judge would most likely have let go. Before this change, the auxiliary admission parked the judge call until the hold cleared and then judged it. Being refused because of a predictable hold is not the same as the judge failing, and holding or deferring the batch, as a quiesce does, would avoid these unjudged wakes.

Comment thread src/framework.ts
Comment on lines +5057 to +5061
try { return await run(); }
catch (error) {
if (!(isMembraneError(error) && error.type === 'abort')) this.recordAuxiliaryFailure(agentName, model, error);
throw error;
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium severity. A failure on the judge's own small model can arm a host hold that parks the agent's primary model.

The judge's call runs on claude-haiku-5-5 (or whatever GateOptions.judge.model names), usually not the agent's model. On any non-abort error it calls recordAuxiliaryFailure → holdProviderAcceleration(agent, error, undefined, true), which asks the host's providerHold hook. If the hook grants a hold, that hold parks the agent's primary turns and every auxiliary call (cooldownBlocksAuxiliary is true for a hold that has no waitModel). So a 429 or quota error on the judge's model, which is now called on every batch, can stop the resident from answering on a model that is fine. Compression differs because it normally runs on the agent's own model. The judge's failures should record only that model's wait (recordProviderWait) and not arm a whole-agent hold.

Comment thread src/gate/reply-judge.ts
Comment on lines +186 to +190
...(request.events.some((e) => e.addressed) ? [] : [
'',
`None of these events is a mention, a reply or a DM to ${agentName}: those wake it without asking you. `
+ 'A message can still ask it something by name.',
]),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Medium severity. Judge is told no event is a mention whenever none is tagged, which is false on servers that don't send chat:addressed.

judgeQuestion adds "None of these events is a mention, a reply or a DM to : those wake it without asking you" whenever no event in the batch has addressed set. addressed comes only from the chat:addressed tag (event-gate.ts 3527). A server that sends no RFC-001 tags therefore has every real mention, reply or DM labelled "not a mention" to the judge, which steers it toward WAIT on messages the agent is owed an answer to. The PR acknowledges this case. It is still a false statement in the prompt; it could be added only when the batch's server is known to tag its messages, or worded as "none is marked as".

…test from the automated review of this PR

- A judge call that fails now records the wait its provider states, if any, which refuses the judge's next call on that model until it ends, and arms no hold on the agent. It had gone through compression's failure path, which also asks the host's hold hook and can arm a hold that parks every primary turn. Compression runs on the agent's own model, but the judge usually doesn't, and the hook is shown the agent's model, so a 429 or a spent limit on the judge's model could have silenced an agent whose own model was fine. If the account itself is spent, the agent's own next request finds it and arms the hold through its own path. With no hold armed by a judge failure, the abort check that kept a stopped call from arming one has nothing left to guard, and goes: a stopped call states no wait, so it records nothing. `withAuxiliaryAdmission`'s failure handling goes back to anima-research#801's, inline, with compression its only caller.
- `JudgeConfig`'s doc no longer says a batch holding an addressed event wakes the agent: where a judge is wired, an addressed event wakes the agent as an `always` rule does and joins no batch.
- The changelog no longer says the judge's calls are bound "as compression's are": compression waits out a hold, while a judge call is refused under it.
- Tests: a judge failure with a stated wait records it for the judge's model, arms no hold though the host would grant one, leaves the agent's turn to go out, and has the next judge call refused; and a judge call on a model the provider holds is refused without being made, with the agent's own model not held. The test that a judge failure arms the host hold goes with what it tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Crispin-2713 and others added 3 commits October 26, 2026 12:24
… the agent if it waits

anima-research#801's branch moved past this pull request's base (e63150f) with 59c0ced, which tells the judge, in an `<if_you_wait>` section, what will wake the agent with no new event. This merge keeps both sides:
- `judgeQuestion` lists the batch, then the line about mentions, then `<if_you_wait>`, then the rule's instructions. `buildRequest` passes the gate's and the framework's upcoming wakes to it, within the fixed tail: no input budget, calibration or anchors come back.
- `ReplyJudgeDeps` has both `recordUsage` and `upcomingWakes`; the framework wires both.
- The gate computes the batch's agent names once and sends them with its `upcomingWakes(agentNames, policyName)`. A debounce batch's `dueAt` is counted on the gate's clock (`now`), as its `firstAt` and `maxWaitMs` are here.
- The package root exports `UpcomingWake` beside `JudgeEventSummary`; `DEFAULT_JUDGE_MAX_INPUT_TOKENS`, which these fixes remove with the input budget, stays out.
- The changelog fragment keeps these fixes' account and adds 59c0ced's paragraph on what waiting leaves the agent with.
- Tests: 59c0ced's tests of the upcoming wakes and `ifYouWait` stay, its framework test of the `skip_reply` timer now wiring a judge with `judge: {}`, since these fixes wire one only where the host asks; its test of an input budget under the minimum goes with the budget. A new gate test checks a re-armed batch's due time is on the gate's clock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uestion

59c0ced wires `upcomingWakes: (agentName) => framework.upcomingWakesFor(agentName)` into the reply judge, but no test in `event-gate-judge` failed with that line removed: its framework test of the skip_reply timer reads the gate's own wakes, which the judge's request carries. This test queues a wake for the agent and checks that the judge's `<if_you_wait>` lists it. The framework isn't started in this harness, so the queued wake stays queued while the judge is asked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n end

59c0ced's `upcomingWakesFor` lists a turn held by a provider cooldown as a certain wake at the cooldown's `until`. A provider wait whose end can't be represented (`waitDeadline` gives null) holds the turn until it's released, and the cooldown's `until` is then `Infinity`. Under such a wait on the agent's own model the judge is still admitted, and its `<if_you_wait>` read "in Infinity h: a turn held by a provider cooldown, when it ends". A wake that may never come was stated as certain. The held turn is now listed only when its `until` is finite.

Test: a turn held for a 60 s wait is listed with a finite time, and once the wait is extended until released, nothing is listed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
antra-tess added a commit that referenced this pull request Oct 27, 2026
…ds the window

On #805's fixes, three of its parts set back, by antra's call:
- The judge is wired unless the host turns it off
  (GateOptions.judge.enabled: false); a null or absent option wires it
  with its defaults. It runs only for judge rules, so a gate without them
  makes no judge calls. The judged tool definitions stay as #805 split
  them, for a host that turns it off.
- No built-in mention bypass: rules are read in order, so a mention rule
  placed ahead of a judge rule wakes without the judge, and a mention the
  judge rule itself matches goes to the judge, marked as addressed. The
  judge's question drops "none of these events is a mention", which held
  only with the bypass. Tests turned to rule order, the MCPL end-to-end
  one with an always mention rule ahead.
- The judge reads the window again, not a fixed tail: social nuance needs
  the room's history. maxInputTokens (default 90,000) and cacheTtl are
  back, with calibration, anchored chunks and the cache breakpoint; the
  minimum budget drops to 4,000, so a host that wants a short tail can
  set one.

Kept from #805: the resident's turn stops the judge (judgeMembraneFor,
changesJudgedContext), usage recorded as the agent's, no reason in the
trace, batches judged by the rule as it is, refusals under holds, and the
smaller fixes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@antra-tess
antra-tess merged commit 95e13c5 into anima-research:feat/reply-judge Oct 27, 2026
12 checks passed
@antra-tess

Copy link
Copy Markdown
Collaborator

Thank you, this is merged into feat/reply-judge with your seven commits intact, plus d9ed834 on top, which sets three parts back by @antra-tess's call:

  • On by default. The judge is wired unless the host turns it off (GateOptions.judge.enabled: false); an absent or null option wires it with its defaults. It runs only for judge rules, so a gate without them makes no judge calls. The rewritten tool descriptions were judged worth one cache write. Your split of the tool definitions stays, for hosts that turn it off.
  • Mentions by rule order, no bypass. Rules are read in order, so a mention rule placed ahead of a judge rule wakes the agent without the judge. A mention the judge rule itself matches goes to the judge with its batch, marked as addressed. The question's "none of these events is a mention" line went with the bypass, since it only held with it. Your end-to-end MCPL test is kept, with an always mention rule ahead of the judge rule. The fork-from-ingestion test is dropped, since it only exercised the bypass.
  • The window, not a fixed tail. A few thousand tokens doesn't carry the nuance of a real social setting, and memories may join the judge's input later. maxInputTokens (default 90,000) and cacheTtl are back, with calibration, anchored chunks and the cache breakpoint. The minimum budget drops to 4,000, so a host that wants a short tail like yours can set one.

Kept as you wrote them:

  • the resident's turn stopping the judge (judgeMembraneFor, changesJudgedContext), which fixes the turn waiting on a judge call;
  • refusals under holds;
  • usage counted as the agent's;
  • no reason in the trace;
  • batches judged by the rule as it is;
  • the smaller fixes, including the provider-wait listing and the upcoming-wakes test.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants