[nissa-seru]

This answers #838: when a persistent agent's turn ends exhausted on a transient failure, its trigger is now sent again after 2 minutes, or with the agent's next wake if that comes first. If that turn also ends exhausted on a transient failure, the trigger goes again after 8 minutes, and on the same terms after 32 more. FrameworkConfig.exhaustedTurnRearmMs sets the delays, and false sends nothing again. The change also raises the provider-wait ops alert when a provider's stated waits, one after another, have held one turn for more than an hour. Since the re-arm is on by default, the changelog fragment files it as a change.

Closes #838.

Problem

• An exhausted turn isn't sent again. As A turn that exhausts its retries is never re-armed: the agent stays silent until an unrelated wake #838 describes, when the error policy stops retrying a turn, the turn ends exhausted and nothing queues its trigger again. A message that wakes the agent during a provider outage longer than the retry schedule then goes unanswered until something else wakes the agent. Under rules that don't wake it on every event, that may take a long time: the issue's probe saw no further attempt for 120 minutes.
• A run of stated waits raises no alert. On main at 5e93190cef, a single stated wait of more than an hour raises the provider-wait ops alert, and a turn held by shorter stated waits, one after another, raises none: in this change's test, four such holds spanning 75 minutes raise no alert with main's code.

This change's tests of both fail on main's code at 5e93190cef (Tests, below).

Changes

• Which turns are sent again. A persistent agent's turn that ends exhausted on a transient failure is re-armed, and a conversation fork's isn't. A failure is transient when membrane marks it retryable, as a MembraneError of type server, network, timeout or rate_limit, and doesn't mark it as a deadline that ran out. These failures end the turn as before:
  • one the provider refuses or says not to retry;
  • one membrane couldn't classify;
  • an error that isn't a MembraneError;
  • a deadline, whether the host set it or membrane marks one as run out.
• The re-armed trigger is a wake of its own. A trigger that continued a turn (a budget restart, tool results ready, or a tool-result guard's retry) is queued with the reason exhausted-turn-rearm. In this change's test of a budget restart, no turn is open while the re-arm waits, and abortInference() has none to end.
• Each re-arm is one send. Until one of its rounds has run or had its tool calls cut off, the error policy doesn't retry a re-armed turn's failure.
  • Another transient failure queues the next delay, until the list runs out. A failure of any other kind ends the re-arming.
  • A re-armed turn that got partway is an ordinary turn, and the policy retries its later failures.
  • A stated wait on a re-armed send is held as any turn's is, and the send after the hold is still the re-arm's one send.
  • A completed turn starts the count of re-arms again, and so does a message said to the agent. In this change's tests a user's message does, including one a tool boundary stores, and so does one in the residents' shared slot, for each resident that reads it. The agent's own schedules don't: the heartbeat's tick, a self-wake timer it set, and a script's wake. Nor does a message a tune-out holds from it. noteSpokenTo (src/framework.ts:26587-26589) also returns without starting the count where the agent's own view leaves the message out (agentSees), and where isConversationalInjection (:418-424) is false: for a message marked system: true, or one tagged as a reaction, an edit, a deletion or a chat:system notice. isMarkerTagged (src/mcpl/tags.ts:56-58) leaves the edit tag out where the host marks a message as the first version the agent sees. Any other wake leaves the count going on. Once the re-arms are spent, the next exhaustion isn't sent again until a turn completes or a message is said to the agent.
• Where it opens. The re-arm opens where the exhausted turn's plain speech was going: its locus, where a message during the turn may have moved it, else its trigger channel.
  • Carrying that channel, it's marked addressed. When a newer ambient wake from another channel merges with a re-armed message's wake, budget restart or tool-result guard's retry, the merged turn keeps the re-arm's channel. Not verified names autoContinue's wake.
  • The re-arm of a turn's continuation drops its counterparty, so it names no author.
  • A re-arm that would have to guess a channel isn't sent: one with no channel id of its own, from a turn that never found a channel because the channel id it was woken for didn't name exactly one registered channel.
• While it waits.
  • The wait is a hold on the agent, unless a provider hold is already on it (Not verified). The health snapshot shows it under providerAdmission (cooldownUntil, cooldownReason, heldRequests), and the reply judge's list of upcoming wakes names it with its count, such as "1 of 3".
  • A wake for the agent during the wait starts its turn at once, with the re-armed trigger merged in. That turn is the wake's: the error policy retries its failures, and if it ends exhausted too, the next re-arm follows it.
  • A judge rule's judge makes its calls during the wait.
  • stop() drops the wait and the count.
• What the operator sees.
  • [inference-rearm] stderr lines say when each re-arm is queued and when it goes, by its timer or with a wake. They also say why none is coming: a failure that isn't transient, once the agent has been re-armed and neither a turn has completed nor a message been said to it since; re-arms that are spent; or a turn that would have to guess a channel. Another says when a message said to the agent starts its count again.
  • Each re-armed turn's failure counts toward the hard-down streak, so the hard-down alert comes at the third exhaustion, the second re-arm's, from a single wake.
• What the agent sees. A re-armed turn's transient failure writes no [inference-failed] marker unless it got partway, so the agent's context keeps the one the first exhaustion wrote. A failure of another kind writes its own.
• The setting. FrameworkConfig.exhaustedTurnRearmMs defaults to [120000, 480000, 1920000]. false sends nothing again, and prints none of the [inference-rearm] lines. create refuses any other value: it must be false or a list of one or more finite numbers above 0.
• The stated-wait alert. A turn that a provider's stated waits, one after another, have held for more than an hour raises the provider-wait ops alert once for that run, saying how long and how many waits.
  • The run counts the holds of the turn's own failed calls whose stated wait still binds, held as a provider wait, an acceleration cooldown or a host's hold, from the first of them. It ends with the turn, or when one of the turn's rounds runs.
  • The alert comes at the run's first counted hold that begins past the hour. If none does, it comes when the run ends, if the run has been held for more than an hour by then, measured to the end of its latest stated wait or to the run's end, whichever comes first. So it can come as late as the run's end.
  • When an operator releases a stated wait on the model of the run's latest wait, the run counts that wait only up to the release.
  • A single stated wait of more than an hour keeps its own alert, and its run raises no second one.
  • The alert doesn't count toward the hard-down streak.
• Texts. exhaustedTurnRearmMs's doc and the changelog fragment, 838-rearm-exhausted-turns.changed.md, say the above for hosts and operators. PROVIDER_WAIT_ALERT_MS's comment said shorter waits are pacing. It now says they are pacing until waits in a row hold one turn longer than an hour.

What each part is for

Each part, with what goes wrong without it.

• The re-arm as a hold on the agent. rearmExhaustedTurn queues the trigger in a timed hold (providerAccelerationCooldowns, with rearm set), or adds it to a provider hold already on the agent. The health snapshot, heldRequests, the judge's list of upcoming wakes and stop() then cover it with no code of their own. It's called at both places a turn ends exhausted: a failure at the turn's start, and one its stream reports.
  • What it costs: two exclusions for a hold with rearm set. cooldownBlocksAuxiliary lets the judge make its calls during the wait, and the scheduler lets a wake through at once rather than holding it for the delay.
• isTransientFailure. Without its class rule, a request the provider refused would be re-armed. Without its retryable check, so would a server error the provider says not to retry, and without its deadline check, a deadline membrane marks as run out.
• The re-arm mark (REARMED, isRearmed) and the policy skip at both failure sites. Without the mark, the error policy takes a re-armed start for a turn's first attempt and gives it a whole retry window. The merged start keeps the mark only when every request it takes in has it, so a wake's turn merged with a re-armed trigger keeps the policy's retries.
• gotPartway. One fact, read by the policy skip and by the failed-turn marker, so the two can't disagree about a turn whose tool calls were cut off.
• A continuation's reason, exhausted-turn-rearm. Without it, a turn stays open while a re-armed continuation waits.
• The re-arm's channel, addressed, and the dropped counterparty. Without the channel, a re-armed continuation's turn doesn't open on its turn's channel, and a re-arm after a re-pin opens where its turn started. Without addressed, a newer ambient wake takes the merged turn's channel. Without dropping counterparty, a re-armed tool-result guard's retry names the old turn's author. The return for a re-arm that would have to guess a channel keeps it from being sent.
• exhaustingRearms and the marker's condition. They give the agent one [inference-failed] marker for the window and the re-arms after it.
• Clearing the count on a completed turn. Without it, a later outage gets fewer re-arms.
• Clearing the count on a message said (noteSpokenTo), called where a message for the agent is stored: by addMessage, by the writes deferred until a turn starts, and at a tool boundary, and for each resident that reads the residents' shared slot, where a message is stored there. Without it, a message that arrives once the re-arms are spent goes unanswered until something else wakes the agent. Without its exclusion of the agent's own schedules, the heartbeat's tick, a self-wake timer or a script's wake starts the count again, and the next exhaustion is re-armed.
• Clearing at stop() the count and the agents whose re-armed turn is failing. Without it, a framework stopped and started again in one process carries a spent count into its next lifecycle.
• turnRearmDelays, the default and exhaustedTurnRearmMs. The delays and the off switch. Copying the list before checking it refuses a list with holes.
• The judge's wording for a re-arm in its list of upcoming wakes, and the [inference-rearm] stderr lines. The line that says why no re-arm is coming needs two more calls: where a re-armed turn's stream is aborted, and where its stream throws. At each of the four (src/framework.ts:21455, :23294, :23369 and :23667), the re-arm call comes after the inference:exhausted trace. Except at the aborted stream's (:23369), the line noting the agent as one whose re-armed turn is failing comes before the trace. In the three failure modes the tests run, each trace comes before its turn's re-arm is held.
• The stated-wait run. StatedWaitRun keeps the run's first hold, its count of holds, whether it has alerted, the latest wait's model and reason, and the latest end of its waits. noteStatedWaitRun is called after a failed call's hold for a provider wait, and after an acceleration cooldown's or a host's hold whose stated wait still binds. endStatedWaitRun ends the run at the turn's end, in the outer catch of a start that throws, and where a round's results are flushed. It alerts there if nothing has alerted yet and the run has been held for more than an hour, measured to the end of its latest wait or to that moment, whichever comes first.
  • waitAlertsOnItsOwn: one measure for the single wait's own alert and for marking its run alerted, so a run whose wait raised that alert raises no second one.
  • Capping a wait at its release: without it, a wait an operator released at minute 41 would count to its stated end, and a turn held open to minute 71 would raise the alert.
  • Measuring to whichever comes first: measured to the time the run ends, a turn that runs a round more than an hour after a short wait would raise the alert. Measured to the latest wait's end, a turn stopped 45 minutes in, during a wait that would have ended at 90, would raise it.
• The test harness: a module whose events wake the agent and that has one tool, so a re-armed turn can run a round; streams that fail at once, after a round, with a round's calls cut off, or ended by a deadline; a scripted membrane that sets each send's outcome by its number; the errors each test needs; and stalledByMembrane, which gets its deadline from membrane's own AnthropicAdapter. The tests run on node:test's mock timers (setTimeout, setInterval and Date), and a test waiting for a condition steps the clock 5 ms at a time. stalledByMembrane runs before a test's clock is mocked, since the adapter times its idle watchdog on the real clock. That last one is what makes reading membrane's unexported Symbol.for('membrane.deadlineExpiry') safe: if membrane renames the mark, that test fails, since its deadline would then be re-armed.
• Cut while building:
  • a rearmed label on the failure's stderr line and in failures.log, which the re-arm's own lines make unnecessary;
  • an ephemeral-run test that no removal could fail;
  • a timer at the hour for the stated-wait run, which made the alert exact for long waits but needed a second timer per run and a check of two other states when it fired. Without it, the alert can come as late as the run's end.

Tests

• On this head's files, as a clean checkout of 4463662892 holds them, on Linux with Node 24.22.1, context-manager 0.23.1 and membrane 0.8.0:
  • test/exhausted-turn-rearm.test.ts passes 54 of 54 tests;
  • the parts above were taken out in 59 pieces, one at a time, and each removal fails at least one of those tests;
  • the full suite runs 8,634 tests: 8,572 pass, none fail and 62 are skipped.
• The same test file with main's code at 5e93190cef, the commit this change is built on, on Linux with Node 24.22.1, context-manager 0.23.1 and membrane 0.8.0: 40 of 54 tests fail. 23 time out waiting for a re-armed send that doesn't come, 16 fail at an assertion, and 1 fails reading a hold that isn't there. The 14 that pass are cases this change leaves as they were: failures that aren't transient, false, a conversation fork, a single long wait's own alert, and runs of stated waits that raise no alert.

Out of scope

• A compile refused for budget isn't sent again here, though A turn that exhausts its retries is never re-armed: the agent stays silent until an unrelated wake #838's probe reproduced that trigger too. It's issue Agent stays silent after an immediate budget decrease: compression recovers, but failed activations are not retried #304, and fix: a turn whose compile is refused for budget waits while compression is pending, and runs once its context fits #361 is an open pull request for it.
• A bound on stated waits: this change alerts on a long run of them, and doesn't limit how long they hold a turn.

Not verified

• No live provider: every test drives a scripted membrane, apart from the one deadline case built with membrane's own AnthropicAdapter against a local server that stops sending.
• A re-arm that comes while a provider hold is already on the agent: no test makes one. In rearmExhaustedTurn, src/framework.ts:5991 counts the re-arm, and :6004-6006 add its trigger to that hold's held requests, with no delay of its own.
• A restart during the wait: no test restarts the framework while a re-arm waits. The wait and the count of re-arms are held in providerAccelerationCooldowns and turnRearms, maps on the framework object (src/framework.ts:3350, :3499). A test stops and starts a framework in one process after its re-arms are spent, and the next exhaustion waits the first delay again.
• The kinds of message noteSpokenTo passes over, besides the agent's own schedules and a message a tune-out holds: no test stores a message marked system: true, one tagged as a reaction, an edit, a deletion or a chat:system notice, or another that the agent's view leaves out, after the re-arms are spent.
• A heartbeat-mcpl server registered under an id other than heartbeat: no test runs one. isOwnScheduleWrite (src/framework.ts:412-416) matches a heartbeat tick by serverId: 'heartbeat' alone, and the changelog fragment says each of such a server's ticks starts the count again.
• Ephemeral runs: no test runs one into an exhausted turn. src/framework.ts:5972 returns from rearmExhaustedTurn for a turn that doesn't own provider admission, and :20176-20177 give that ownership to no ephemeral run's turn.
• A channel id that two servers' channels answer to: no test wakes a turn with one. The test's channel id names no registered channel. noChannel's doc names both cases (src/framework.ts:1318-1324), and the re-arm's check at :5981 reads it.
• A re-armed autoContinue wake merged with another wake: no test makes one. src/framework.ts:5992 copies the trigger into the re-arm, its auto-continue reason included, and :17071-17081 take a merged start's channel from any other request that names one before an autoContinue wake's.

---

• Changelog fragment added: changelog.d/838-rearm-exhausted-turns.changed.md.

Effie-2756 wrote this change, and Ada-1017 made its design calls. Desmond-2579 read the whole change, in the stages it was built in, and eight rounds of internal review read earlier versions of it. Prudence-2578 weighed each part against the least change that would answer the issue, and Laurel-2581 read it as the operator of a resident host would. Both read an earlier version of it.

Fiona-2757 wrote this description from Effie-2756's facts.

[github-actions]

Automated review

Recommendation: needs a maintainer's decision. The change belongs, but part of it needs a maintainer's call.

Reviewed commit 0091806 with claude-opus-5-5, by reading the code. It did not run the code or look at the checks.

Does this change belong in the project?

It fits, with concerns. Re-sending an exhausted turn's trigger fits: it extends the framework's existing error policy, provider holds and ops alerts to cover an issue (#838) and sits where that machinery already lives. A maintainer should still decide on three things: the change also carries a separate stated-wait alert, it turns re-sending on by default, and the re-arm wait is not ended by abortInference() even though the README says that call ends held retries.

5 reasons with sources

• The re-arm fits the framework's job of running agent turns through inference, retries and provider holds. The README says the framework handles the whole inference lifecycle, including the error policy's retries, provider-capacity holds and stated waits. The diff builds on that machinery in src/framework.ts: it reuses ProviderAccelerationCooldown with a rearm field, the hard-down streak, and the health snapshot's providerAdmission. It does not add a separate mechanism. (README.md (Overview; Abort / Cancellation))
• The new public setting exhaustedTurnRearmMs is on by default, so every host's behavior changes unless it opts out. The change files this correctly as a .changed.md fragment and lets hosts opt out with false. Still, a default-on 2/8/32-minute re-send adds a public interface and a new concept maintainers will have to keep working. Whether it should be on by default is the maintainers' call. (CONTRIBUTING.md (Changelog); changelog.d/README.md)
• The re-arm wait is listed as a hold, but abortInference() doesn't end it, which differs from what the README promises for held retries. The README says a turn whose failed call a provider hold keeps for its retry is ended by abortInference(), by quiesce abandon and by stop(). The re-arm appears under providerAdmission as a cooldown. The changelog fragment says abortInference() doesn't end it, because the re-arm counts as a new wake rather than an open turn. That choice is defensible, but it changes the cancellation contract, and the README isn't updated to say so. (README.md (Abort / Cancellation, line 545))
• The PR bundles a second feature, the alert for a run of stated waits longer than an hour, which could be proposed on its own. The provider-wait alert for back-to-back stated waits (statedWaitRuns, noteStatedWaitRun/endStatedWaitRun) is related to outages, but it is independent of re-sending exhausted turns. It has its own tests and its own changelog bullet. (CONTRIBUTING.md (How changes land: keep commits coherent))
• The PR follows the house conventions for shape: a Problem/Changes/Tests body, a changelog fragment and a dedicated test file. It adds changelog.d/838-rearm-exhausted-turns.changed.md and test/exhausted-turn-rearm.test.ts, and the body says the new tests fail on main's code. The README is not updated for the new config option. (CONTRIBUTING.md (What a PR should contain))

Findings

5 findings of medium or high severity, see the inline comments:

• Medium severity. Stream-throw path never marks a re-armed transient failure, so each re-arm writes another [inference-failed] marker.
• Medium severity. Once re-arms are spent, every later exhaustion stays unanswered until some turn completes, which is the #838 failure.
• Medium severity. turnRearms and statedWaitRuns are not cleared when stop() tears down provider admission.
• Medium severity, test coverage. No test covers a re-arm that joins a provider hold already on the agent.
• Medium severity, test coverage. The stream-throw failure mode is tested only with non-transient errors.

4 more findings, less severe

• Low severity, maintainability, src/framework.ts:23645-23646. rearmExhaustedTurn is called before the inference:exhausted trace on two paths and after it on the other two. At 21440 and 23279, rearmExhaustedTurn runs after emitTrace('inference:exhausted'), the event-gate end and action.emit. At 23349 (stream aborted) and 23645 (stream threw), it runs before the trace. So the stderr lines come in a different order on each path ('[inference-rearm] ... sent again' before '[inference-failed]'). A listener of inference:exhausted also sees a re-arm hold on some paths and not on others. Running the same teardown step in a different order per path makes later changes error-prone. One shared ordering, or one helper, would avoid this.
• Low severity, maintainability, src/framework.ts:5983-5994. Re-armed request copies the whole trigger, including TURN_CARRY and the author fields, while its channel is overwritten. { ...trigger, [REARMED]: true } removes TURN_RETRY and MERGED_BATCH but keeps TURN_CARRY. For a wake (not a continuation) it also keeps counterparty, wakeChannelId and ambient/systemNotice, then overwrites channelId with the locus and forces addressed = true. After a mid-turn re-pin, the request can name one channel as its locus and an author or wake channel from another, and can carry both ambient: true and addressed: true. TURN_CARRY is ignored today only because the copy no longer continuesTurn. Building the request from an explicit list of fields would remove these hidden dependencies.
• Low severity, maintainability, src/framework.ts:1178-1181. Transient check depends on a private, unexported membrane symbol. isTransientFailure reads Symbol.for('membrane.deadlineExpiry'). The comment itself says membrane 'doesn't export its reader'. If a membrane release renames or drops this internal mark, typecheck still passes, and deadline expiries that membrane reports as retryable timeouts would start being re-armed. Only the test using a live socket would notice. Better to ask membrane to export a predicate, or to pin the dependency, than to depend on an internal registry key.
• Low severity, test coverage, test/exhausted-turn-rearm.test.ts:735-736. Test file replaces console.error for the whole process and never restores it. console.error is reassigned at module scope so it pushes into stderr, and nothing restores it. Every framework diagnostic in the file, including '[inference-hard-down]' lines and unexpected errors, is swallowed into an array that grows without bound and is only partly cleared. A failing test then shows no stderr context. Use t.mock.method(console, 'error', ...) per test so it is restored automatically.

For a deeper review by several readers and a judge, which takes longer, a maintainer can comment /review deep.

[github-actions]

Medium severity. Stream-throw path never marks a re-armed transient failure, so each re-arm writes another [inference-failed] marker.

The marker is skipped only when exhaustingRearms holds the agent (noteInferenceExhausted, line 28475). The new code adds the agent to that set at the start-failure site (21420) and the stream-error-event site (23261), but not at the catch path where the stream itself throws (23639-23651). There it only calls rearmExhaustedTurn. So when a re-armed turn's stream throws a transient MembraneError (server/network/timeout), the turn is re-armed again, and every exhaustion also writes a fresh [inference-failed] marker into the agent's context. That breaks the stated rule that the agent's context keeps only the first exhaustion's marker. The tests run the 'throw' mode only with non-transient errors, so they miss this.

[github-actions]

Medium severity. Once re-arms are spent, every later exhaustion stays unanswered until some turn completes, which is the #838 failure.

turnRearms is reset only on inference:completed (line 28001). During an outage longer than about 42 minutes, the three re-arms are used up. After that, any new message whose turn exhausts logs 'its 3 re-arms are spent until a turn completes' and nothing queues it again. A turn can't complete during the outage, so every such message goes unanswered until an unrelated wake comes after recovery. That is the exact symptom #838 reports, so the fix covers only the first ~42 minutes of an outage. One possible fix is to reset the count by wall-clock time, or per new external wake.

[github-actions]

Medium severity. turnRearms and statedWaitRuns are not cleared when stop() tears down provider admission.

cancelProviderAdmission (line 6419) clears every cooldown, including re-arm holds, and every recovery. It leaves the new per-agent maps turnRearms, exhaustingRearms and statedWaitRuns in place. If the framework is stopped and started again in the same process, which the framework already plans for (RETRY_ACROSS_RESTART), the re-arm count carries over. An agent whose re-arms were spent before the stop gets no re-arm after start() until a turn completes. A stated-wait run left open by a held turn that stop() dropped also keeps its old since, so the first hold in the new lifecycle can raise a false 'over an hour' provider-wait alert. These maps should be cleared together with the cooldowns.

[github-actions]

Medium severity, test coverage. No test covers a re-arm that joins a provider hold already on the agent.

When a cooldown already exists, rearmExhaustedTurn adds the re-armed request to existing.heldRequests, uses up one re-arm count, and logs 'when the provider hold already on the agent ends'. The re-arm's own delay is ignored, and the hold carries no rearm metadata, so admission does not let a wake through early. No test in test/exhausted-turn-rearm.test.ts reaches this branch: none checks the log line, that the count is used up, or that the request is released with the existing hold.

[github-actions]

Medium severity, test coverage. The stream-throw failure mode is tested only with non-transient errors.

ScriptedMembrane's 'throw' mode (ThrowingStream) appears only with a plain Error, in the 'not transient' loop. No test sends a transient MembraneError through the thrown-stream catch path (framework.ts 23603-23651). That path has its own rearmExhaustedTurn call and skips the policy check and the exhaustingRearms bookkeeping. A test there would catch the extra [inference-failed] markers and confirm that this path re-arms at all.