Repository navigation
Reply judge: wire it only where asked, wake on mentions without it, read a fixed tail, and let resident turns stop it - #805
Conversation
… never let it hold back an addressed message - The judge is wired only when the host sets `GateOptions.judge` (and not `enabled: false`). Elsewhere `wake_add_rule`'s description and schema are as they were before the judge, so no resident's tool list changes and no resident can start paid judge calls its operator didn't choose. Where it's wired, the tool takes `judge`, and its description and its `behavior` parameter say so. A `judge` given to the tool on a host without one is named as a parameter it doesn't read, as any other unknown name is. - A batch holding an event addressed to the agent (`chat:addressed`) wakes it without asking the judge. One that fires while the judge decides the rule's earlier batch wakes the agent now, with that batch and whatever the rule holds, and the judge deciding it is stopped: once woken, the agent reads the rest anyway. So a mention never waits on a model that its own text, an error or an instruction could steer to WAIT. - Each judge request's usage counts as the judged agent's, in its usage tally and the session's totals, where a turn's is counted. Nothing else records auxiliary calls today, compression's included; this keeps the judge from being a second kind of uncounted spend. - `gate:judge-verdict` no longer carries the judge's reason. The reason is written from the agent's context, a private turn's input included, and a trace can reach observers the agent's own `gate_status` doesn't; `gate_status` keeps it. - A judge's `maxWaitMs` is counted on the gate's clock (`this.now`), as its sleep is. Tests: the addressed batch, and the one that fires while the judge decides; the clock; a host that sets no `GateOptions.judge` (no judge, the tool unchanged, `judge` not read); the judge's usage in the agent's tally; the trace without the reason; the judged tool's own example saved as it reads. `event-gate-strict-rules`' wake_add_rule expectations are back to the unjudged tool's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g for it, and only the residents' turns hold a judge's batch - The judge calls through a membrane that takes no provider admission, so none of the agent's turns waits for a judge call in flight. Before, the judge took the primary's auxiliary admission, and a turn that found it in flight parked until it settled, up to the judge's 20 s timeout, a mention woken through another rule included. The membrane still refuses a model its provider holds and withdraws a call under a surgery or a stop, as for a conversation agent's compression; that helper is renamed `ungatedAuxiliaryMembraneFor`, since it now has a second caller. - The gate is told which turns change the context the judge reads (`changesJudgedContext`): the framework's ordinary residents, who share the one message slot the judge reads from the primary's preview. A fork and the subconscious keep stores of their own, so their turns neither hold a judge rule's batch nor stop the judge. A resident's turn that starts or resumes while the judge decides stops the call, as a quiesce does, and the batch is judged once after the turn, against what it left. - The re-judge after a turn is gone, with what counted it: `turnEpochs`, the batch's `rejudge` flag, `judgeState.verdicts.rejudged` and the verdict trace's `rejudged`. A verdict can only be reached with no resident's turn started since its call began, so no verdict is followed by a second call for the same batch. A batch held across a run of turns isn't lost: by the addressed check it's ambient, its events stay in context, each of those turns reads them, and it's judged once they stop. A stopped call is no verdict: it wakes nothing and isn't counted as a failure. Tests: a resident's turn that starts while the judge decides stops it and the batch is judged once after; a co-resident's turn stops it too; a fork's or the subconscious's turn neither holds nor stops; the framework counts its resident's turns and not the subconscious's; the agent's turn sends its request while a judge call hangs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a reply or a DM to the agent A batch holding an event addressed to the agent wakes it without asking the judge, so every batch the gate sends it holds none. The judge's question now says so when no event in the request is addressed, and that a message can still ask the agent something by name, so a plain-name ask stays on the side of waking. A request holding an addressed event, as a host's own caller of the judge might send, gets no such line. Test: the line is in the question when no event is addressed, and not when one is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… JudgeEventSummary, and time maxWaitMs's test on the gate's clock From the automated review of anima-research#801's lower findings: - Where a judge is wired, `wake_remove_rule` says that a judge rule's batch, waiting out its quiet period, held or being judged, is still judged, by the rule as it was, and wakes the agent only on reply. Its text said such a batch is delivered as its timer would. Only the judged variant of the tool changes, so a host with no judge keeps the tool as it is on `main`. - The package root exports `JudgeEventSummary`, the element type of `JudgeRequest.events`, beside the judge's other types. - `reply-judge.ts`'s header gives the document's order as `buildRequest` builds it: the conversation, then the system prompt, abridged. - `maxWaitMs`'s test runs on an injected clock: six events 60 ms apart on the gate's clock pass the 250 ms cap, and the batch is judged with exactly those six. It ran on real timers and asserted a range, so a loaded runner could fail it and a cap that fired on the first quiet gap could pass it. It also covers the clock test it replaces. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, not a 90,000-token window The judge now reads the newest of the agent's conversation, in whole messages up to 6,700 characters, the agent's system prompt cut to 850, and the batch's listing capped at 2,160, newest first and the rest counted: a few thousand tokens a request. Removed, with what each was for: - the input budget (`maxInputTokens`, `DEFAULT_JUDGE_MAX_INPUT_TOKENS`, `MIN_JUDGE_INPUT_TOKENS`) and the share arithmetic that divided it, which kept a window of up to 90,000 tokens under Claude Haiku 5.5's 100,000-token price line; - the token estimate calibrated from each response's reported input, which made that budget hold across tokenizers; - the anchored window and its byte-stable chunks with a cache breakpoint after the last full one, and `cacheTtl`, which let successive requests read most of a large window from the prompt cache. A tail of this size is far under any price line at any tokenizer, and one that moves with every message gains nothing from caching, so the request asks for none. Why the tail is enough: the judge as anima-research#801 built it was run on 100 ambient batches drawn at random from a busy room, three times each, reading the whole window (90,000 tokens) and the newest part (8,000). Those runs gave each event its message's real time, days before the call, and its whole text, so they stand as comparisons on the same batches. Without the line added in the commit before this one, the two agreed on 93 of 100, and in each of the 7 that differed the tail woke the agent where, by our reading, the whole window rightly didn't, mostly taking an ask addressed to someone else for the agent's. With that line, the tail's wakes fell from 26 of 300 verdicts to 5, a majority on 2 batches, both wrong by our reading, and the whole window's from 3 to 0; by our reading, no batch either run woke at any point was one the agent should have answered. The tail read a median of 3,437 input tokens against 56,559. The three sizes here are what that tail actually read, measured per part on the same batches: about 6,700 characters of the newest whole messages, a system-prompt cut of about 850, and a listing cut of about 2,160. The two tests of the budget under a larger context become one of the fixed tail under it; the tests of the calibration, the over-budget correction, the byte-stable chunks and the minimum budget leave with what they tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n wakes as an always rule does, and a waiting batch is judged by the rule as it is From our review of the five commits before this one, and of anima-research#801's own code: - The judge has its own admission (`judgeMembraneFor`), bound as the agent's compression is, except that it never waits on the agent's turn, which stops the judge instead. A call is refused without being made under a surgery's hold or at stop, for a model the provider holds, and under a hold that parks the whole agent (`cooldownBlocksAuxiliary`, the predicate compression's admission parks on). Each such refusal is marked as made without a call (`providerAdmission: 'deferred'`): it records nothing, at the gate either (no verdict count, no failure streak, no verdict trace), and the batch takes `onError`'s path, so under a spent quota its wake parks behind the hold as a debounce rule's does. A call that fails records the model's wait and arms the host hold, through `recordAuxiliaryFailure`, which compression's failures now go through too, since the judge's spend is the agent's. A call the gate stops (membrane's `abort`) is no provider failure and arms nothing. The helper the judge had borrowed, in the commit that made a resident's turn stop it, was the conversation agents' admission, which skipped the agent-wide holds and never armed one, so under a spent quota the judge kept calling; it's back to `main`'s name, `conversationAuxiliaryMembraneFor`, with its one caller. - A mention under a wired judge rule (an event its server marks `chat:addressed`) decides as an `always` rule does: `evaluate` returns `trigger: true`, the rule's `resets` are applied, and `probe` answers the same. The event's own processing then wakes the agent, after the message is stored and to the agents its channel routes to, a conversation fork that ingestion creates included, rather than the mention waiting out the rule's quiet period, which a busy channel could push back without end when no `maxWaitMs` is set. The mention joins no batch, and the rule's other events still go to the judge: the turn it starts is a resident's turn, which stops a judge call in flight and holds the batch until the turn ends, and the judge then reads what the turn left. Settling the rule's batches at the mention instead would drop events no wake then carries, since `evaluate` runs before ingestion (a duplicate, a router's veto or an unrouted message wakes nothing), a rule's batches span forks the mention doesn't reach, and events not stored at arrival reach a turn only through their batch's own wake. A judge rule with no judge wired keeps a debounce's timing, as debounce rules do. The tool's text says the mark is the server's. - A batch held for a resident's turn is judged by the rule's judge as it is when the turn ends, while it's still a judge rule, as the events queued behind a judge call already were, and by the rule as it was once it isn't, as `wake_remove_rule` says. It had kept the judge it was first asked with, so after a change of instructions the held and queued paths disagreed. A verdict that arrives after its rule is removed no longer re-creates the rule's record, which a later rule of the same name took over in `gate_status`. - A `GateOptions.judge` of null wires no judge, as leaving it out does. - The judge's question keeps the wording the tail runs measured: "None of these events is a mention, a reply or a DM to <agent>". From a server that doesn't mark its mentions, those mentions reach the judge and the sentence overstates; the line's second sentence, that a message can still ask the agent something by name, is what keeps such a mention on the side of waking. We ran "is marked as a mention", which is true there too, on the same 100 batches, three verdicts each: it drew 14 REPLY verdicts of 300, a majority on 5 batches, each a false wake by our reading of the batch, most of them an ask to another agent of the same first name taken for the agent's. Two runs of the measured wording drew 7 and 6, a majority on 1. `judgeQuestion`'s doc says that a server may not mark its mentions and what covers them. - Texts: `wake_remove_rule` no longer promises a removed judge rule's batch wakes only on reply (a failure wakes by `onError`, and a mention without the judge), and the judged `wake_add_rule` example drops the same "only"; the changelog says the newest message and the newest event are always shown, each event cut to 200, the context is the newest part shown oldest first, a batched wake still waits while any agent's turn runs, how the judge's calls are bound, how a mention wakes, that a held batch is judged against what the turn left, and that the 100-event cap applies to what a rule holds while a turn runs and to the events behind a judge call. - Tests: where our gate tests assert that something happens, they wait for it rather than for a fixed sleep, and the `maxWaitMs` test checks the batch isn't judged before the cap and is within 1 s of it, under its 2 s quiet period. New tests cover: a mention that triggers, with its `resets`, the probe agreeing and changing nothing, and that leaves an open batch, one being judged and one held for a turn to the judge, its turn stopping the call; a debounce rule and an unwired judge rule left to their quiet period; end to end over an MCPL server, a mention whose turn compiles with its message, stops the judge, after which the batch is judged and answers WAIT, with one turn in all and no hold armed, and a mention on a channel no fork is bound to, which wakes the fork ingestion creates; a resumed turn's stop, and the events queued behind a stopped call; a held batch judged by changed instructions, and a removed rule's late verdict; the judge's failure arming the host hold with the agent's next turn held behind it, the refusals under a hold on the whole agent and under a surgery, a stopped call arming none, and a refused call at the gate under each `onError`, with events queued behind it and a failure already counted; a co-resident in the framework; usage on an answer with no verdict; a mixed batch's question; the tail's order, placement and `promptCaching: false`; the newest message over the size; the listing's order and budget; the unwired `always` and `judge` call; and a null judge option. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Automated review
Recommendation: needs a maintainer's decision. The change belongs, but part of it needs a maintainer's call.
Reviewed commit fa5c4c7 with claude-opus-5-5, by reading the code. It did not run the code or look at the checks.
Does this change belong in the project?
It fits, with concerns. This belongs here: it stacks on #801's in-progress reply judge, stays inside the gate, framework and judge modules, and declares its base. A maintainer should still decide on two things: it reverses #801's choice to wire the judge by default, and it bundles four separable changes in one PR.
5 reasons with sources
- The change refines a feature that #801 is already adding, inside the gate code that owns it, and adds no new subsystem. Every edit is to the judge's wiring in framework.ts, its batching in event-gate.ts, its prompt in reply-judge.ts, or its types, tests and changelog fragment. Nothing lands outside the judge's existing home. (changelog.d/reply-judge.added.md (base); src/gate/reply-judge.ts, src/gate/event-gate.ts)
- Making the judge opt-in reverses a default that #801 chose on purpose, so the maintainer has to settle it. The base comment reads: "Unless the host turns it off, every gate has one, so a judge rule an agent adds works where its host never configured one." The PR changes this so only a host that sets GateOptions.judge gets a judge, and keeps a second, derived set of wake-rule tool definitions behind a judgeWired flag. The description says itself that the default is the maintainer's call. (src/framework.ts (base comment at the judge wiring))
- It carries several changes that could each be proposed on its own, though as separate coherent commits. The four parts are: opt-in wiring; letting mentions skip the judge; replacing the token-budgeted, cached window with a fixed tail (which removes the maxInputTokens and cacheTtl options); and letting a resident's turn stop a judge call. CONTRIBUTING asks that commits stay coherent because PRs are merged without squashing. The six commits allow that, but the maintainer may want to take only some of the parts. (CONTRIBUTING.md, How changes land)
- It follows the stacking and evidence conventions: the base is declared, and the behaviour change comes with numbers, a changelog fragment and tests. The PR names #801's head e63150f as its base. It updates changelog.d/reply-judge.added.md and the judge tests, and reports measured verdict rates and token counts. (CONTRIBUTING.md, How changes land / What a PR should contain)
- Removing the budget and cache options shrinks the public interface; it does not grow it. maxInputTokens, cacheTtl and the related constants were introduced by the unreleased #801, so dropping them breaks no released interface. (changelog.d/reply-judge.added.md (base))
Findings
4 findings of medium or high severity, see the inline comments:
- Medium severity. Judge calls can be stopped and restarted without limit, and a batch can be held off indefinitely.
- Medium severity. Under a provider hold, the default onError wakes the agent for every judge batch without judging it.
- Medium severity. A failure on the judge's own small model can arm a host hold that parks the agent's primary model.
- Medium severity. Judge is told no event is a mention whenever none is tagged, which is false on servers that don't send chat:addressed.
5 more findings, less severe
- Low severity, maintainability,
src/gate/event-gate.ts:3856-3865. Refused judge calls leave no trace or gate_status record; under onError 'skip' batches drop silently. On the 'deferred' path,finishJudgingreturns before it updatesjudgeRecords,judgeFailStreakor emitsgate:judge-verdict. Only a console.error line is written. WithonError: 'skip'and a long hold, every judge rule's batch is dropped, and neither gate_status (verdicts unchanged, nolast, no failing notice) nor traces show the agent or operator that the judge is making no calls. Count refusals separately (for exampleverdicts.refused) or show a gate_status notice while refusals continue. - Low severity, documentation,
src/gate/types.ts:87-90. JudgeConfig doc says a batch holding an addressed event skips the judge, but the event never joins a batch. The added text says "A batch holding an event addressed to the agent (chat:addressed) wakes it without asking the judge." IncomputeDecision(event-gate.ts 3151-3154), the addressed event triggers inline and never enters the batch. The rule's other pending events still go to the judge, and that only happens when a judge is wired (this.judgeFn). The comment suggests that one mention in a batch wakes the agent for the whole batch, which is not what the code does. - Low severity, documentation,
changelog.d/reply-judge.added.md:11. Changelog says judge calls are bound "as compression's are", but compression parks under a hold while the judge is refused. The line says "Its calls are bound as compression's are: under a hold on the agent or on the model, a call is refused without being made." Compression goes throughwithAuxiliaryAdmission, which parks a call whileproviderGateBlocked(a primary turn or a whole-agent hold) and does not refuse it.judgeMembraneForrefuses instead and never waits. The comparison will mislead operators about how the two differ; the comment onReplyJudgeDeps.membraneForin reply-judge.ts states the difference correctly. - Low severity, maintainability,
src/framework.ts:3858-3862. Judge requests are counted in the agent's inferenceCount as if they were turns.recordUsagecallsusageTracker.onInferenceCompleted(agentName, ...), which adds to both the session's and the agent'sinferenceCountas well as the token totals. A judge call is made on every batch of a judge rule, so an agent's inference count ingetSessionUsage()andusage:updatedtraces now mixes small judge calls with real turns, and becomes several times its turn count on a busy channel. Recording the tokens without counting an inference, or keeping a separate judge tally, would keep the count meaningful. - Low severity, test coverage,
src/framework.ts:5046. No test for a judge call refused because of a per-model provider wait. The new tests cover a whole-agent hold (cooldownBlocksAuxiliary), a surgery withdrawal, a stop by abort, and a quota failure that arms a hold. No test coversrefuseHeldProviderModelfor the judge's own model: a recorded wait onclaude-haiku-5-5should refuse the judge withproviderAdmission: 'deferred'while the agent's model is unaffected. The changelog promises this case ("under a hold ... on the model, a call is refused"), and it is the most likely way a judge-model 429 shows up.
For a deeper review by several readers and a judge, which takes longer, a maintainer can comment /review deep.
| private stopJudging(): void { | ||
| for (const [policyName, state] of [...this.judging]) { | ||
| this.judging.delete(policyName); | ||
| if (state.timer) clearTimeout(state.timer); | ||
| state.abort.abort(); | ||
| this.holdForJudge(policyName, [...state.events, ...state.queued], state.judge); | ||
| } | ||
| } | ||
|
|
||
| /** Send the batches held for the judge once no turn changing the context it reads runs and the host isn't quiesced. */ | ||
| private flushJudgeHeldIfClear(): void { | ||
| if (this.quiesced || this.inferring.size > 0 || this.judgeHeld.size === 0) return; | ||
| if (this.quiesced || this.judgedTurnRunning() || this.judgeHeld.size === 0) return; | ||
| const held = [...this.judgeHeld]; | ||
| this.judgeHeld.clear(); | ||
| // Deferred, as the inference buffer's flush is, to stay out of the | ||
| // turn-end callback that called this. | ||
| queueMicrotask(() => { | ||
| for (const [policyName, { events, judge, rejudge }] of held) this.judgeBatch(policyName, events, judge, rejudge); | ||
| for (const [policyName, { events, judge }] of held) this.judgeBatch(policyName, events, this.currentJudge(policyName, judge)); | ||
| }); | ||
| } | ||
|
|
There was a problem hiding this comment.
Medium severity. Judge calls can be stopped and restarted without limit, and a batch can be held off indefinitely.
This change removes the old limit of one re-judgment per batch (rejudge and turnEpochs). Now every resident onInferenceStarted/onInferenceResumed calls stopJudging(), which aborts any judge call in flight and holds its batch. flushJudgeHeldIfClear then sends the batch to a fresh judge call when the turn ends. On a busy host where turns keep starting while the judge decides (co-residents, mention turns, self-wakes), the same batch is re-sent to the judge each time and aborted again. Each attempt is a new provider request that may already be billed, and the batch's wake can be put off with no bound, even when maxWaitMs is set. The PR description's "at most one small judge call per waiting batch" does not hold. Consider a cap on stops per batch, or letting a reached verdict stand after N stops.
| if (this.cooldownBlocksAuxiliary(agentName)) { | ||
| const cooldown = this.providerAccelerationCooldowns.get(agentName)!; | ||
| throw Object.assign(new MembraneError({ | ||
| type: 'rate_limit', | ||
| retryable: true, | ||
| retryAfterMs: Math.max(0, cooldown.until - Date.now()), | ||
| message: `${agentName}'s provider calls are held (${cooldown.reason}), so the reply judge makes none`, | ||
| rawError: undefined, | ||
| }), { providerAdmission: 'deferred' as const }); | ||
| } |
There was a problem hiding this comment.
Medium severity. Under a provider hold, the default onError wakes the agent for every judge batch without judging it.
judgeMembraneFor now refuses a judge call under a whole-agent hold (or a held model, via refuseHeldProviderModel) and marks it providerAdmission: 'deferred'. In finishJudging (event-gate.ts 3856-3865), that path then follows judgeOnError, and the default is "wake". So while the agent is held for a spent quota, every ambient batch under a judge rule calls deliverOrBuffer, which queues a wake behind the hold. When the hold lifts, the agent spends a primary turn on chatter the judge would most likely have let go. Before this change, the auxiliary admission parked the judge call until the hold cleared and then judged it. Being refused because of a predictable hold is not the same as the judge failing, and holding or deferring the batch, as a quiesce does, would avoid these unjudged wakes.
| try { return await run(); } | ||
| catch (error) { | ||
| if (!(isMembraneError(error) && error.type === 'abort')) this.recordAuxiliaryFailure(agentName, model, error); | ||
| throw error; | ||
| } |
There was a problem hiding this comment.
Medium severity. A failure on the judge's own small model can arm a host hold that parks the agent's primary model.
The judge's call runs on claude-haiku-5-5 (or whatever GateOptions.judge.model names), usually not the agent's model. On any non-abort error it calls recordAuxiliaryFailure → holdProviderAcceleration(agent, error, undefined, true), which asks the host's providerHold hook. If the hook grants a hold, that hold parks the agent's primary turns and every auxiliary call (cooldownBlocksAuxiliary is true for a hold that has no waitModel). So a 429 or quota error on the judge's model, which is now called on every batch, can stop the resident from answering on a model that is fine. Compression differs because it normally runs on the agent's own model. The judge's failures should record only that model's wait (recordProviderWait) and not arm a whole-agent hold.
| ...(request.events.some((e) => e.addressed) ? [] : [ | ||
| '', | ||
| `None of these events is a mention, a reply or a DM to ${agentName}: those wake it without asking you. ` | ||
| + 'A message can still ask it something by name.', | ||
| ]), |
There was a problem hiding this comment.
Medium severity. Judge is told no event is a mention whenever none is tagged, which is false on servers that don't send chat:addressed.
judgeQuestion adds "None of these events is a mention, a reply or a DM to : those wake it without asking you" whenever no event in the batch has addressed set. addressed comes only from the chat:addressed tag (event-gate.ts 3527). A server that sends no RFC-001 tags therefore has every real mention, reply or DM labelled "not a mention" to the judge, which steers it toward WAIT on messages the agent is owed an answer to. The PR acknowledges this case. It is still a false statement in the prompt; it could be added only when the batch's server is known to tag its messages, or worded as "none is marked as".
…test from the automated review of this PR - A judge call that fails now records the wait its provider states, if any, which refuses the judge's next call on that model until it ends, and arms no hold on the agent. It had gone through compression's failure path, which also asks the host's hold hook and can arm a hold that parks every primary turn. Compression runs on the agent's own model, but the judge usually doesn't, and the hook is shown the agent's model, so a 429 or a spent limit on the judge's model could have silenced an agent whose own model was fine. If the account itself is spent, the agent's own next request finds it and arms the hold through its own path. With no hold armed by a judge failure, the abort check that kept a stopped call from arming one has nothing left to guard, and goes: a stopped call states no wait, so it records nothing. `withAuxiliaryAdmission`'s failure handling goes back to anima-research#801's, inline, with compression its only caller. - `JudgeConfig`'s doc no longer says a batch holding an addressed event wakes the agent: where a judge is wired, an addressed event wakes the agent as an `always` rule does and joins no batch. - The changelog no longer says the judge's calls are bound "as compression's are": compression waits out a hold, while a judge call is refused under it. - Tests: a judge failure with a stated wait records it for the judge's model, arms no hold though the host would grant one, leaves the agent's turn to go out, and has the next judge call refused; and a judge call on a model the provider holds is refused without being made, with the agent's own model not held. The test that a judge failure arms the host hold goes with what it tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the agent if it waits anima-research#801's branch moved past this pull request's base (e63150f) with 59c0ced, which tells the judge, in an `<if_you_wait>` section, what will wake the agent with no new event. This merge keeps both sides: - `judgeQuestion` lists the batch, then the line about mentions, then `<if_you_wait>`, then the rule's instructions. `buildRequest` passes the gate's and the framework's upcoming wakes to it, within the fixed tail: no input budget, calibration or anchors come back. - `ReplyJudgeDeps` has both `recordUsage` and `upcomingWakes`; the framework wires both. - The gate computes the batch's agent names once and sends them with its `upcomingWakes(agentNames, policyName)`. A debounce batch's `dueAt` is counted on the gate's clock (`now`), as its `firstAt` and `maxWaitMs` are here. - The package root exports `UpcomingWake` beside `JudgeEventSummary`; `DEFAULT_JUDGE_MAX_INPUT_TOKENS`, which these fixes remove with the input budget, stays out. - The changelog fragment keeps these fixes' account and adds 59c0ced's paragraph on what waiting leaves the agent with. - Tests: 59c0ced's tests of the upcoming wakes and `ifYouWait` stay, its framework test of the `skip_reply` timer now wiring a judge with `judge: {}`, since these fixes wire one only where the host asks; its test of an input budget under the minimum goes with the budget. A new gate test checks a re-armed batch's due time is on the gate's clock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…uestion 59c0ced wires `upcomingWakes: (agentName) => framework.upcomingWakesFor(agentName)` into the reply judge, but no test in `event-gate-judge` failed with that line removed: its framework test of the skip_reply timer reads the gate's own wakes, which the judge's request carries. This test queues a wake for the agent and checks that the judge's `<if_you_wait>` lists it. The framework isn't started in this harness, so the queued wake stays queued while the judge is asked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n end 59c0ced's `upcomingWakesFor` lists a turn held by a provider cooldown as a certain wake at the cooldown's `until`. A provider wait whose end can't be represented (`waitDeadline` gives null) holds the turn until it's released, and the cooldown's `until` is then `Infinity`. Under such a wait on the agent's own model the judge is still admitted, and its `<if_you_wait>` read "in Infinity h: a turn held by a provider cooldown, when it ends". A wake that may never come was stated as certain. The held turn is now listed only when its `until` is finite. Test: a turn held for a 60 s wait is listed with a finite time, and once the wait is extended until released, nothing is listed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ds the window On #805's fixes, three of its parts set back, by antra's call: - The judge is wired unless the host turns it off (GateOptions.judge.enabled: false); a null or absent option wires it with its defaults. It runs only for judge rules, so a gate without them makes no judge calls. The judged tool definitions stay as #805 split them, for a host that turns it off. - No built-in mention bypass: rules are read in order, so a mention rule placed ahead of a judge rule wakes without the judge, and a mention the judge rule itself matches goes to the judge, marked as addressed. The judge's question drops "none of these events is a mention", which held only with the bypass. Tests turned to rule order, the MCPL end-to-end one with an always mention rule ahead. - The judge reads the window again, not a fixed tail: social nuance needs the room's history. maxInputTokens (default 90,000) and cacheTtl are back, with calibration, anchored chunks and the cache breakpoint; the minimum budget drops to 4,000, so a host that wants a short tail can set one. Kept from #805: the resident's turn stops the judge (judgeMembraneFor, changesJudgedContext), usage recorded as the agent's, no reason in the trace, batches judged by the rule as it is, refusals under holds, and the smaller fixes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Thank you, this is merged into
Kept as you wrote them:
🤖 Generated with Claude Code |
Fixes to #801's reply judge, as seven commits on its earlier head
e63150fd, a merge of its current head59c0ced3and two commits after it, for judge rules to be something an operator chooses and a resident can rely on. Each part below says what it's for; merging any of it into your branch is yours to decide, @antra-tess, and nothing else is needed from you.Changes, and what each part is for
The judge is wired only where the host asks for one (
GateOptions.judge, unlessenabled: false).wake_add_rule's description and schema, andwake_remove_rule's, are as they are onmain, byte for byte. A changed tool definition rewrites every resident's cached prefix once, at the cache-write price, and lengthens its tool list from then on (1,574 characters on one gated resident's request). That shouldn't happen on hosts that never use the judge.judgepassed to the tool there is named as a parameter it doesn't read, as any other unknown name is.judge, its description gains the judge example, and itsbehaviorparameter names it, as in feat(gate): judge wake rules — ask a small model before waking #801.main's definitions, and ajudgeWiredflag that picks between them.A message addressed to the agent wakes it without asking the judge (
evaluate).alwaysrule does,resetsincluded, andprobeanswers the same. Its own event wakes the agent once the message is stored, and wakes the agents its channel routes to, a conversation fork that ingestion creates included. It doesn't wait out the rule's quiet period, which a busy channel could push back without end when nomaxWaitMsis set.chat:addressed); a server that doesn't mark its mentions sends them through the judge. The tool's description says that a message its server marks as addressed wakes the resident without the judge, so it needn't keepalwaysrules ahead of a judge rule for its mentions.The judge is told that no event in its batch is a mention, a reply or a DM to the agent, whenever none is, and that a message can still ask the agent something by name. From a server that marks its mentions, the check above makes the first sentence true of every batch the gate sends the judge, and the second keeps a plain-name ask on the side of waking. From a server that doesn't mark its mentions, the first sentence overstates, and the second is what keeps such a mention on the side of waking. We kept that wording because it's the one we measured. On the same batches, "is marked as a mention" drew 14 REPLY verdicts of 300 in one run, against 7 and 6 in two runs of this wording, and by our reading every one of those 14 was a false wake.
The judge reads a fixed tail instead of the whole window. It reads the newest of the conversation in whole messages, up to about 6,700 characters, the agent's system prompt cut to 850, and the batch's listing capped at 2,160: a few thousand tokens a request. As in #801, the newest message is always shown, up to the 12,000 characters a message renders, and so is the newest event, each event's text cut to 200 by the gate. The request asks for no prompt caching: membrane caches by default, and a tail that moves with every message would pay a cache write on each request for no read.
maxInputTokensandcacheTtloptions withDEFAULT_JUDGE_MAX_INPUT_TOKENSandMIN_JUDGE_INPUT_TOKENS. Those parts existed to keep a 90,000-token window under Haiku 5.5's price line and affordable to read again.fa5c4c7d, before the merge of59c0ced3, with each batch's newest event 5 s old and each event cut as the gate cuts it, the tail woke the agent by majority on 1 batch of 100, with 6 REPLY verdicts of 300, under a 181-character system prompt. Under a real 69,272-character system prompt cut to 850, it was 1 batch, with 3 of 300. The median request was 3,066 and 3,297 input tokens, answered in 940 and 958.5 ms. By our reading both wakes were wrong, each an ask to another agent with the same first name taken for the agent's, and no batch the agent should have answered was held back.59c0ced3's paragraph on what waiting leaves the agent with, and nothing scheduled to wake the agent, the tail woke it by majority on 3 batches of 100 under the 181-character system prompt, with 13 REPLY verdicts of 300, and on 5 under the real one, with 19. The median request was 3,234 and 3,465 input tokens. Several of the new wakes' reasons say that nothing else is scheduled to wake the agent. With a heartbeat two hours ahead passed as an upcoming wake, worded as feat(heartbeat): the heartbeat moves into the framework #827'sHeartbeat.upcomingWakes()words its default, it was 2 batches with 7 of 300, and 1 batch with 6. By our reading every one of these wakes was wrong, and in three of the batches a reason took a message to another agent with the same first name for one to the agent. So on this sample the false wakes rise where nothing is scheduled, the case it's written for: with a wake two hours ahead they stay nearfa5c4c7d's. No batch in the sample is one it's meant to help, an event that matters to the agent without being addressed to it, so these runs show what it costs there and not what it gains.A resident's turn stops the judge rather than waiting for it, and only the residents' turns hold a judge's batch.
0dd4d181it waited about 205 ms for the judge call to settle. A test at this head checks that the turn's request goes out while a judge call hangs.judgeMembraneFor), bound by the same holds as the agent's compression, except that it never waits: a turn stops the judge instead, and where compression would wait out a hold, a judge call is refused. A call is refused, without being made, under a surgery or at stop, for a model the provider holds, or under a hold on the whole agent (the predicate compression's admission parks on); a refusal records no wait, hold or failure, the gate counts no verdict for it and only logs it, and the batch takesonError's path, so under a spent quota its wake parks behind the hold as a debounce's does. A call that fails records the wait its provider states, if any, so the judge's next call on that model is refused until it ends, and arms no hold on the agent. Compression's failures can arm the host's hold, but compression usually runs on the agent's own model, while the judge usually doesn't, and the host's hold hook is shown the agent's model: a 429 on the judge's model could park every turn of an agent whose model is fine. By our reading, if the account itself is spent, the agent's own next request finds it and holds the agent then. What it costs: one admission function beside the framework's two, sharing their checks.changesJudgedContext): the framework's ordinary residents, who share the message slot the judge reads through the primary's preview. A fork and the subconscious keep stores of their own, so their turns neither hold a judge rule's batch nor stop the judge; the wake a batch gets still waits while any agent's turn runs, as every batched wake does. A resident's turn that starts or resumes while the judge decides stops the call, as a quiesce does, and the batch is judged once after the turn, against what it left.e739c82f, with itsturnEpochs,rejudge,verdicts.rejudgedand the trace'srejudged. No verdict can be reached across a resident's turn, so no batch is judged again after a verdict, and no turn waits on the judge. Your reason for "once", that overlapping turns mustn't hold a batch back for good, still holds. A batch held across a run of turns is ambient, by the check above. Its events are in the agent's context for each of those turns, as a WAIT's are, and it's judged once the turns stop. A stopped call is no verdict. It wakes nothing and doesn't count as a failure.Each judge request's usage counts as the judged agent's, in its usage tally and the session's totals, where a turn's is counted (
recordUsage). Nothing in the framework records auxiliary calls today, compression's included; this keeps the judge from adding a second kind of uncounted spend.gate:judge-verdictno longer carries the judge's reason. The reason is written from the agent's context, a private turn's input included, and a trace can reach observers who don't otherwise see that context.gate_status, the agent's own view, keeps it.A batch waiting on a judge rule is judged by the rule as it is when the judge is asked, while it's still a judge rule, and by the rule as it was once it's removed or no longer a judge rule, as
wake_remove_rulesays. A batch held for a resident's turn had kept the judge it was first asked with, while events queued behind a judge call took the rule's current one. A verdict that arrives after its rule is removed also no longer re-creates the rule's record ingate_status, which a later rule of the same name would have taken over.The merge of
59c0ced3keeps both sides. The judge's question lists the batch, then the line about mentions, then<if_you_wait>, then the rule's instructions, and the fixed tail carries the upcoming wakes with no input budget coming back.ReplyJudgeDepshas bothrecordUsageandupcomingWakes. The gate sends the batch's agent names once, with itsupcomingWakes, and counts a batch's due time on its own clock, as it counts the batch's first event andmaxWaitMs. The package root exportsUpcomingWakebesideJudgeEventSummary.59c0ced3's framework test of theskip_replytimer now setsjudge: {}, since the judge is wired only where the host asks. The framework's own wakes in the judge's question are a queued wake and a turn a provider cooldown holds. By our reading, the judge makes no call under a hold on the whole agent, so the cooldown's line reaches it only for a wait on one model other than the judge's own. We changed one thing in59c0ced3's ownupcomingWakesFor. A turn held by a provider wait with no end, whose cooldown'suntilisInfinity, was listed as certain "in Infinity h", telling the judge of a wake that may never come. The held turn is now listed only when itsuntilis finite.Smaller fixes from the automated review's lower findings. Where a judge is wired,
wake_remove_rulesays that a judge rule's batch is still judged after the rule is removed, by the rule as it was; the package root exportsJudgeEventSummary;reply-judge.ts's header gives the document's order as it's built; andmaxWaitMs's test runs on an injected clock, so it can pass neither a cap that fires at once nor a missing one. A judge'smaxWaitMsis also counted on the gate's clock (now), as its sleep is, and aGateOptions.judgeof null wires no judge, as leaving it out does.What this leaves
Tests
On Linux with Node v24.22.1:
95e13c5c(the run was at a commit with the same tree and an earlier message): 7,646 tests in 319 files, 7,602 passing, 0 failing and 44 skipped, with membrane 0.7.2 and context-manager 0.22.1 installed. The typecheck and build pass.fa5c4c7d,e63150fd's source fails 32 of them: 26 inevent-gate-judge, 5 inreply-judgeand 1 inevent-gate-strict-rules. Each fails at an assertion, but for one framework test that reads the newchangesJudgedContextoption, which doesn't exist there. One more, the refusal under a hold on the whole agent, ran to its 600 s timeout there. The tests' import ofJudgeEventSummaryfrom the package root, and theirrecordUsage, fail the typecheck there. Of220dfc55's two new tests,fa5c4c7d's source fails the one for a judge failure, at an assertion. The other, for a judge model the provider holds, passes there too: it pins whatfa5c4c7dalready did.220dfc55's at1486156a, whose tree differs from it only in one doc comment and the changelog's wording, and the rest atfa5c4c7d: a mention's trigger, itsresetsand the probe's answer; the refusals under an agent-wide hold and under a surgery, and the gate counting no failure for a refusal; a judge failure recording the wait its provider states, and the refusal on a model the provider holds; the null option; the framework's residents; a starting and a resumed turn's stop, and the queued events a stop keeps; a held batch judged by the rule as it is, and no record for a removed rule;promptCaching: false; the usage record; and the question's wording. The same run with nothing removed passed. Of the pieces after220dfc55, each of five, removed alone, failed a test, the first two at the merge and the rest at the commit that added their test: a re-armed batch's due time on the gate's clock, the question carrying the upcoming wakes, the framework's upcoming wakes reaching the judge, a queued wake among them, and a held turn listed only when its wait has an end. The same runs with nothing removed passed.alwaysrule's would,resetsincluded, with the probe agreeing and changing nothing, and that leaves an open batch, one being judged and one held for a turn to the judge; and a debounce rule and an unwired judge rule left to their quiet period, the probe agreeing;GateOptions.judge, or a null one: no judge, both wake-rule tools as onmain,judgenot read, and analwaysrule given beside ajudgesaved asalways;wake_remove_rule's judged text;onErrorwith the events queued behind it, or underskipwakes nothing and sends those events to the judge next;maxWaitMson the gate's clock: not judged before the cap, and judged with every event at it;59c0ced3's own wiring of the framework's wakes into the judge, whose removal no earlier test inevent-gate-judgecaught;maxWaitMstest now covers.claude-haiku-5-5) at effortlow, with a seeded draw of 100 batches, three verdicts per batch per side, and every batch where a verdict woke the agent read by hand.Not verified
ReplyJudgedirectly, with a context built from the room's transcript as plain messages. A resident's realpreviewActivationcompile, with its tool calls and results, notices, gate lines and images asrenderMessageshows them, wasn't in them.recordUsagecounts only a call that returns a response.changelog.d/reply-judge.added.mdWritten by Crispin-2713 (Claude Opus 5.5), to design calls by Ada-1017 (Claude Opus 5.5). Imogen-2490 (Claude Opus 5.5) ran the judge comparisons and read this round.
Description checked and posted by Fiona-2757 (Claude Opus 5.5).
🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared
nissa-seruaccount.