Repository navigation
fix(rearm): take a re-armed turn's marked deadline cut as transient unless it's the second in a row - #871
Open
nissa-seru wants to merge 20 commits into
Open
fix(rearm): take a re-armed turn's marked deadline cut as transient unless it's the second in a row#871nissa-seru wants to merge 20 commits into
nissa-seru wants to merge 20 commits into
Conversation
added 20 commits
October 27, 2026 22:37
…d ends the re-arming A re-armed turn makes one send, with no retries from the error policy until a round of it has run or had its tool calls cut off. A deadline membrane marks as run out, such as a stalled stream cut by the runtime's fetch timeout or the Anthropic adapter's watchdog, isn't a transient failure, so one stall on that one send ended the re-arming. A turn that ends on deadline expiries isn't re-armed, since repeated stalls of one request point at the request rather than the provider; a re-armed send that stalls has stalled once. Now, in a turn whose trigger is a re-arm, the first such cut is taken as a transient failure: the next delay follows, and the cut is marked as a re-armed turn's transient failure is: no [inference-failed] marker unless a round of the turn ran or had its tool calls cut off. A second such cut ends the re-arming with the existing "its failure isn't transient" line. A completed turn, a message said to the agent, and stop() clear it, where they start the re-send count again. A deadline the host sets on a turn still ends the re-arming, and a turn that isn't itself a re-arm is still never re-armed on a deadline. The re-sends stay one per delay in the list. The deadline mark's reader is split out of isTransientFailure, so the transient rule and the exception read the mark one way. The type doc and the unreleased changelog fragment state the exception. Tests: a re-armed send cut once is sent again at the next delay and, cut again, ends the re-arming with that line and a marker of its own; a message said, a completed turn, and a stop() and start() each let the next re-armed send's first cut re-arm again.
…m its stream The test of a re-armed send cut once and then again now runs with the cut at the send's start, reported by its stream, and thrown by its stream, as the default delays' test does. The start's failure path is the one that settles a turn whose stream never began, so the exception is pinned there too. A retryable failure a stream throws reaches the turn as its error event, so both stream cases take that path.
…ens, and decide it once A re-armed turn that got partway gets the error policy's retries, and the policy retries a deadline membrane marks, since it is a retryable timeout. Those cuts went unrecorded, so a turn whose retries all stalled was re-armed when it ended exhausted, as if its last cut were the chain's first. Each such cut is now recorded when it happens: when the policy retries it, and where the turn ends exhausted, before the exhausted trace. Recording before the trace also keeps a message said by a trace listener from being undone. It now starts the chain afresh after the cut; before, the cut was added after the trace and landed in the new chain. The cut's verdict is decided once, where the turn ends, and passed to rearmExhaustedTurn, so the marker the turn writes and whether it is re-armed can't disagree when a listener changes the chain in between. A cut whose re-arm is then refused, its re-sends spent or no channel to speak on, is recorded all the same, since it happened. Tests: a re-armed turn that got partway and whose policy retries end in cuts ends the re-arming; a message said while a cut's exhaustion is reported lets the next cut be re-armed; the once-then-again test now measures the next delay after the first cut, with distinct delays, and counts the markers after each cut.
…n on its second cut, and keep none past stop() Three holes in "every such cut counts": - A host's provider hold that took a re-armed turn's marked cut returned before the cut was recorded, so the held send's next cut counted as the chain's first. Each failure is now recorded once, at the top of its failure site, before any hold, and the hold, the error policy and the exhausted path all read that one result. - A re-armed turn that got partway gets the error policy's retries, so its second cut was retried, and a later server error re-armed the turn again although its request had stalled twice. A re-armed turn's second cut now gets no retry from the policy: the turn ends on it, with the existing "its failure isn't transient" line. - stop() clears the cuts, but a re-armed send still being set up holds no stream for stop() to cancel, so its cut could be recorded after the clear and outlive a start() on the same framework, ending the next lifecycle's first re-armed cut. A cut is no longer recorded once admission is closed; whether it counts as the chain's first is decided as before. The type doc and the changelog fragment now say that the first cut since a turn last completed or a message was last said is taken as a transient failure, so a turn that ends on it is re-armed at the next delay; that a second ends the re-arming, with no retry from the error policy; and that a cut the policy retries or a host's hold takes counts. Tests: a partway turn's second cut gets no retry and ends the re-arming; one cut the policy retried, then server errors, then the next re-armed cut ends it; a partway turn ending on its first cut is re-armed and writes the partway marker; a cut a host's hold takes counts; a send cut at its start once stop() has begun leaves no cut for the next lifecycle.
A deadline membrane marks is kept out of re-arming because two stalls of one request point at the request rather than the provider. Between two cuts, though, the provider can answer: a re-armed turn's send can fail with a server error, or a round of it can answer. Then the two cuts weren't one request stalling twice, and ending the re-arming gave up on a message the provider would have answered once it recovered. Now a transient failure of a re-armed turn, and a round of one that answered, clears its cut, so two cuts end the re-arming only when no other answer from the provider came between them. The round is noticed at the stream's usage event, which membrane emits once a round's call has resolved in full, so a stream that starts and then stalls doesn't clear its own cut. The type doc and the changelog fragment say the rule as two such cuts in a row, and that a turn ending on a first cut is re-armed as for a transient failure. Tests: a cut, then server errors, then a cut is re-armed again; a round answered between two cuts lets the second get the error policy's retry; a partway turn's second cut, at its start too, still ends the re-arming and writes its marker. The stall stream replays the events membrane's own stalled stream emitted before its cut, and the scripted tool round reports its usage before its tool calls, as membrane does.
"No other answer from the provider between them" read as if the provider had to respond, though a network failure counts too. The type doc and the changelog fragment now say "no transient failure or answered round between them", and that a re-armed turn's transient failure or answered round clears an earlier cut, as a completed turn, a message said to the agent and stop() do.
Two marked cuts end the re-arming only when the provider gave no other answer between them. A turn that isn't itself a re-arm can carry such an answer: a wake that merges into a held re-arm starts its own turn, and its rounds can answer and its sends fail with a server error. Clearing the cut only in a re-armed turn took the next re-armed cut as the chain's second though the provider had answered in between, and with a longer delay list that ended a chain the rule would re-arm. Now a transient failure or an answered round in any of the agent's turns clears an earlier cut. Only a re-armed turn's cut is ever recorded, so a cut in another turn still never counts. That makes three other pieces redundant, and they are dropped. Every chain of re-arms begins with a turn that isn't a re-arm ending on a transient failure, which now clears the cut, so no cut recorded before it can reach the chain: - the clear at a completed turn: a turn completes only after a round answered, and that round's usage event has already cleared the cut; - the clear in stop(): stop() drops every held re-arm, so a chain after start() begins afresh. A re-armed send whose start outlives both stop() and start() can meet a cut kept from before stop(), which can only end its chain sooner; - not recording a cut once stop() has begun: such a cut is cleared before any later chain reaches it. The type doc and the changelog fragment say a message said to the agent clears a cut beside the provider's answers, and no longer list stop(). Tests: a wake merged into a held re-arm answers a round and fails with server errors, and the next re-armed cut is re-armed again. The stall test asserts that membrane's stalled stream emits nothing before its cut, so a membrane that began emitting anything there, usage included, would fail it. The tests that a completed turn, a stop() and start(), and a cut once stop() has begun each leave the next chain's first cut a first are unchanged and pass.
Only a re-armed turn's marked deadline cut is recorded and taken as transient. Until any turn's server error cleared an earlier cut, the check that the turn is a re-arm was pinned by a test whose window, a turn that isn't a re-arm, was cut twice and then failed with a server error: without the check those cuts were recorded, and the re-armed send's cut after them was taken as a second. That server error now clears them, so nothing failed without the check. This test has the provider answer a turn's first two tries with server errors and its last with a marked cut. The turn isn't a re-arm, so it ends exhausted with nothing kept to be sent again; without the check its cut is taken as a first and the turn is re-armed.
0.29.0 folded changelog.d/838-rearm-exhausted-turns.changed.md into CHANGELOG.md, and this branch had edited that fragment to say a marked deadline in a re-armed turn's send counts as a transient failure once. The fragment is gone, and the addition moves to its own fragment, changelog.d/871-rearm-deadline-cut.changed.md, written against 0.29.0's released entry, which listed every deadline membrane marks as run out among what the re-arm leaves unchanged: the new fragment says that still holds for a turn that isn't itself a re-arm. src/framework.ts merged without conflict: main's heartbeat drafts (anima-research#861) and this branch's deadline rule change different hunks. The exhaustedTurnRearmMs doc in src/types/framework.ts and test/exhausted-turn-rearm.test.ts are this branch's alone; main changed neither.
…tween two cuts A message said to the agent clears an earlier cut, so it comes between two cuts as a transient failure or an answered round does; the fragment now says so where it defines two cuts in a row. A deadline the host sets on a turn ends the re-arming, which the fragment now says in those words.
…and the progress bounds' deadline tries Main brings the stop-ends-its-instance change, which makes start() refuse once a stop() is admitted, and the progress bounds change, whose error policy counts a deadline membrane marks by its own tries (deadlineCuts). Git conflicts in src/framework.ts, resolved to keep both rules: - main now reads membrane's deadline mark in inference-bounds.ts (isDeadlineExpiry), so this branch's own reader and symbol go and its callers use main's; - at each error-policy decision, the policy is asked with main's count of deadline tries, and this branch's rule still refuses a retry to a re-armed turn that hasn't got partway, or whose cut isn't its chain's first; main's count is kept or cleared by the decision as main has it, so a re-armed turn whose first cut the policy retried, and whose retry was cut again, ends with the marker saying the call was made again; - an answered round clears both this branch's cut and main's count. With a stop ending its instance, a re-armed chain can no longer begin after a stop() on the same instance: a re-arm queued across the stop never runs, and a new framework starts with no cut recorded. Main's own restart test in test/exhausted-turn-rearm.test.ts comes in as main rewrote it, on a new framework over the stopped one's store. Two of this branch's tests restart the same instance after its stop() and would now throw at start(), so they go with this merge: - the "a stop() and start()" case of "... starts the chain afresh, so its next re-armed send's deadline cut is re-armed again", which pinned that a cut didn't carry into the instance's next lifecycle; an instance has none now, and a new framework starts with no cut, since the cut is kept in memory; the loop's two other cases no longer share a store, which only the restart case needed; - "a re-armed send cut at its start once stop() has begun leaves no cut for the next lifecycle", which has no next lifecycle on the same instance.
…een them exhaustedTurnRearmMs's doc defined two cuts in a row as having "no transient failure or answered round between them", while the next sentence, noteSpokenTo's clear, and the changelog fragment all count a message said to the agent too. The definition now names all three, in the fragment's words.
A transient failure in any of the agent's turns clears an earlier deadline cut, not only in a re-armed turn. The merged-wake test pinned that with a turn that also answered a round, and an answered round clears the cut on its own, so the test passed with the transient clear limited to re-armed turns. It now runs twice: with the round and server errors, and with server errors alone, the case that only the transient clear covers.
…ade again In a re-armed turn that got partway, the error policy retries the chain's first deadline cut, counting it among its deadline tries, and the retry's cut ends the re-arming. The marker that turn writes says the call was made again and timed out again, as any turn's does when a retried deadline is cut again.
…ollows the deadline-cut rule Main now reports a call cut by a bound the host sets (inferenceBounds) as a deadline marked as run out, as membrane marks its own. Such a cut is the same kind as membrane's watchdog, a provider that stopped making progress, so in a re-armed turn it follows the same rule: the chain's first cut is re-armed as a transient failure is, and a second ends the re-arming. The doc and the changelog fragment no longer except a deadline the host sets, and the fragment's note on 0.29.0 quotes that line's words on which deadlines it meant; a host that wants no such bound still has inferenceBounds: false. Tests follow: the host's deadline leaves anima-research#846's list of failures that end the re-arming for a test of its own under the rule; under the default error policy a re-armed turn that got partway no longer ends on its chain's first cut, since the policy's deadline schedule calls again after it, so that test goes, the two-cut test pinning what happens instead; and a turn that isn't a re-arm makes the calls the deadline schedule makes again before it ends on the cut, still not re-armed.
… deadline cut with it releaseAgentRuntime clears a name's per-name turn state so a name bound again, a channel's next conversation or a resume, starts afresh: the re-arm count, the exhausting re-arm and the exhausted turn. The set of agents holding a re-armed turn's deadline cut wasn't among them, so a name that ended holding one would have its next binding's first re-armed cut refused as a second. It is cleared there now, beside the count, as main's release clears its own count of deadline tries.
…he exception; a test comes back under a host's policy - exhaustedTurnRearmMs's doc said a deadline "ends the turn as before", which, read beside the exception two sentences on, contradicted it now that the host's bound is a marked deadline too. It now says a turn that ends on a deadline, the host's or membrane's, isn't re-armed, save the exception; and the agent's end joins what clears an earlier cut. - A re-armed turn that got partway and ends on its chain's first deadline cut is re-armed, and writes the marker a partway turn does. Under the default error policy that case doesn't arise, since its deadline schedule makes a fresh call after the first cut; under a host's policy that makes none, it does, so the test comes back under such a policy. - A stream that ends aborted for a reason other than a timeout or the caller's cancel still ends a re-armed turn's re-arming. anima-research#846's list of failures that do had pinned that with the host's deadline, which main now reports as a marked error instead, so the list gets a stream aborted with another reason. - Two test titles say what they pin: the release guard's ends at what it asserts, and the not-transient test's deadline cases are in a turn that isn't itself a re-arm.
The spec's stall comes from a real AnthropicAdapter against a local server that sends message_start and stops, so each cut is membrane 0.8.0's own marked deadline; each call waited out the adapter's 200 ms idle timeout, so the file spent most of its wall time waiting. The stall's error is now made once per file and shared: the framework writes to no error it is handed (the only marked error it makes is deadlineCut's own), so a test can't see another's use of it.
…s exception inferenceBounds' doc said DefaultErrorPolicy gives each call a bound cuts one fresh call on an agent's turn. A turn that re-arms an exhausted one gets no policy retry until it has got partway, and once partway a second marked cut in a row gets none either; what a cut does there is exhaustedTurnRearmMs' rule, which this change makes the host's bound follow, so the doc points to it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A re-armed turn's send cut by a deadline marked as run out, membrane's own or the host's bound on the call (
inferenceBounds), as when a provider's stream stalls, is now taken as a transient failure, so a turn that ends on it is re-armed as for a transient failure, rather than ending the re-arming that #846 added. Two such cuts in a row, with no transient failure, answered round from the provider or message said to the agent between them, end the re-arming as before, and the second gets no retry from the error policy. A transient failure or an answered round in any of the agent's turns clears an earlier cut, and so does a message said to the agent. Every such cut in a re-armed turn counts, including one the error policy retries or a host's provider hold takes. A first cut writes no[inference-failed]marker unless a round of the turn ran or had its tool calls cut off. A turn that isn't itself a re-arm is still never re-armed on a deadline.Problem
exhaustedTurnRearmMs, as one send with no retries from the error policy until a round of it has run or had its tool calls cut off. A transient failure queues the next delay, and any other failure ends the re-arming. At0f11975e48,mainat 06:11 UTC on October 29, which this head merges in,isTransientFailurereturns false for an error that carries the deadline mark (src/framework.ts:1236). So a re-armed send that such a deadline cuts ends the re-arming after one cut. With this head's tests and0f11975e48'ssrc/framework.tsandsrc/types/framework.ts, 17 of the spec file's 74 tests fail: every test this change adds except the test at:349, the host-hold test at:501and the aborted stream it adds to the loop at:368, which pass there.dist: the Anthropic adapter's idle and first-event watchdogs (providers/anthropic.js:757, each 600 s by default at:386and:392), the Anthropic SDK's own timeout (:924), socket deadline codes, and aTimeoutErrorin an error's cause chain (types/errors.js:742). At0f11975e48,isDeadlineExpiryreads that mark (src/inference-bounds.ts:82), andturnStreamEventsreports a stream'sabortedevent with reasontimeoutas an error fromdeadlineCut(src/framework.ts:2554), which carries the same mark (src/inference-bounds.ts:109).Changes
All line numbers are at this head, and in
src/framework.tsunless another file is named.rearmDeadlineCuts(:3780) is a set of agent names. An agent is added at a marked cut in a re-armed turn of its own, and removed by the clears below.takeAsTransient(agentName, error, trigger)(:6470) decides each failure once, at the top of each failure site and before any hold: a send's start (:24232), the stream's error event (:26079) and the outer catch (:26512).:6472).:6477), and is taken as transient only if the agent wasn't in the set already (:6476).:24250,:26096), and, passed torearmExhaustedTurn(:6481), whether the turn is re-armed. At those two policy sites, a marked cut in a re-armed turn that isn't taken as transient gets no retry, even in a turn that got partway.deadlineCuts) is kept and cleared asmainhas it, and the policy is asked with it as onmain. So in a re-armed turn that got partway, a policy that retries a deadline cut, as the spec file's does, retries the chain's first cut. When the retry is cut too, that second cut ends the re-arming, and its marker says the call was made again (test/exhausted-turn-rearm.test.ts:424). Under a policy that makes no fresh call for a deadline, such a turn ends on the chain's first cut and is re-armed (:485there).usageevent (:26268), in any turn, besidemain's clear ofdeadlineCuts. membrane 0.8.0 emitsusageafter a round's call has resolved (membrane.js:2372and:2770, each afterawait this.streamOnce);noteSpokenTo(:29604), beside the re-send count;releaseAgentRuntime, which ends a name's turn state, beside the re-send count (:11892).falsetorearmExhaustedTurn(:26250).exhaustedTurnRearmMs's doc insrc/types/framework.ts, and a changelog fragment,changelog.d/871-rearm-deadline-cut.changed.md, which states the rule and the line of 0.29.0's entry it corrects. No log line or setting is added.git merge-treeof each of the other 42 open pull requests' heads with this head gave the same conflicting files as withmain, then4a45766f01, except feat(agent): declare later tool and system-text changes in the context #690 at740f4f979d. That one conflicts with this head insrc/framework.tsandsrc/types/framework.ts, and with4a45766f01insrc/agent.tsandsrc/framework.ts.2c6c343543, the head an earlier version of this description described, nine commits:91cd7ce5c7merges0f11975e48. That brings fix(framework)!: a stop ends its AgentFramework instance, and start() and addModule() are refused from then on (#315) #881, after whichstart()is refused once an instance has stopped, and feat(inference)!: cut an AgentFramework model call that stops making progress, and give a deadline expiry one fresh call #378's count of deadline tries, which the two policy sites now pass the policy. It also bringsturnStreamEvents' report of a stream aborted with reasontimeoutas an error fromdeadlineCut(src/framework.ts:2554), which carries the marktakeAsTransientreads. Two of this change's tests restarted a stopped instance, and they go.a814087deaadds a message said to the agent to the doc's definition of two cuts in a row.609c804c36runs the merged-wake test with server errors alone, as well as with a round (test/exhausted-turn-rearm.test.ts:456).ea7dcceb50checks the marker of a partway turn's second cut (:419there).dfcc0502e1names the host's bound in the doc and the fragment, and moves the host's deadline from the list of failures that end the re-arming (:363there) to a test of its own under the rule (:404there). It drops the test of a partway turn ending on its chain's first cut, a cut the spec file's policy now retries, and the test at:344there now counts the calls that policy makes again.7fa8f64930adds the release's clear and its test (:999there).56444dd442rewords the doc on a deadline, and adds two tests: the partway turn ending on its chain's first cut, back under a policy that makes no fresh call for a deadline (:480there), and a stream aborted for another reason, in the loop at:363there.ead844dc69makes the spec file's stall from membrane's real Anthropic adapter once per file, shared by the tests that use it, where each such test made its own (test/exhausted-turn-rearm.test.ts:116–121).01f17ddb61adds toinferenceBounds' doc that the one fresh callDefaultErrorPolicygives a call a bound cuts doesn't hold "in a turn that re-arms an exhausted one, whereexhaustedTurnRearmMssays what a cut does" (src/types/framework.ts:318–320).Tests
Runs on Linux 6.12.113 (x86_64) with Node 24.22.1, each after
npm installwith nonode_modulesor lockfile. agent-framework commits no lockfile.ead844dc69, whose only difference from this head's is that sentence ofsrc/types/framework.ts's doc: the build succeeds, andtest/exhausted-turn-rearm.test.tspasses 74 of 74 andtest/prose-drafts.test.ts253 of 253, with membrane 0.8.0 and context-manager 0.24.1. With0f11975e48'ssrc/framework.tsandsrc/types/framework.ts, 17 of the spec file's 74 tests fail, as the Problem says.ead844dc69's files, 74 tests, 2.782 to 3.150 s of wall time and 2.079 to 2.420 s of user CPU time over six runs, at one-minute loads of 17.94 to 19.03; with0f11975e48'ssrc/framework.ts,src/types/framework.tsand spec file in their place, 55 tests, 2.654 to 2.892 s of wall time and 1.900 to 2.032 s of user CPU time over three runs, at loads of 19.48 to 19.61. Three of the six runs onead844dc69's files came before those three, and three after.test/exhausted-turn-rearm.test.tsadds these tests. Where a test's cut is membrane's, it's the error membrane 0.8.0's Anthropic adapter gives when its idle watchdog runs out (idleTimeoutMs) on a local server that stalls aftermessage_start.:349).:385), at a send's start, reported by its stream, and thrown by its stream. The first is re-armed at the next delay ((2 of 3)), at least 290 ms and under 890 ms later with delays of 100, 300 and 900 ms, and writes no marker of its own. The second ends the re-arming with "turn not re-armed: its failure isn't transient (timeout)" and its marker. That's five sends. The test also checks that membrane 0.8.0's stalled stream emits no event before its cut (:389).:409): a re-armed send whose stream ends in anabortedevent with reasontimeoutis re-armed ((2 of 3)), and a second such cut ends the re-arming. That's five sends.:424): the spec file's policy retries its chain's first cut, and the retry's cut, at its start or from its stream, gets no retry and ends the re-arming. Its marker says "The call was made again, and it timed out again." That's five sends.(3 of 3),:443). A wake merged into a held re-arm, whose turn answers a round and fails with server errors, or fails with server errors alone, so the next re-armed cut is re-armed ((4 of 4),:456). A round answered between two cuts lets the second have the policy's retry, which answers (:474).:485), a turn that got partway and ends on its chain's first cut is re-armed, and writes the marker a partway turn writes. That's six sends.:501).:515).:534).:544).:1004).fastPolicy(:180,:240) unless a test gives its own.{ toolRoundThen }runs a tool round and then fails with a given error, for the partway tests. The scripted tool round reports itsusagebefore its tool calls, so a scripted round that answers clears a cut.openingOfrecords what membrane's real stalled stream emits before its cut, and the stall stream replays it before the error.:330). The host's deadline leaves the list of failures that end the re-arming (:368), for its own test at:409, and the list gains a stream aborted for another reason.src/framework.tsrun against the spec file onead844dc69's files. 16 remove this change's parts, and 6 re-check feat(turns): send a turn's trigger again when its retries run out on a transient failure, with delays of 2, 8 and 32 minutes, and alert on an hour of stated waits #846's parts where this change edits their lines. Every one fails at least one test except the outer catch's (Not verified). What each part is for, below, gives what failed without it.The full suite ran in CI's run 40835344638 for this head, on GitHub's merge of
01f17ddb61into0f11975e48, whose tree is this head's. On Ubuntu, with Node 22.12.0, 24.22.1 and 26.13.0, it ran 9,230 tests each, none failing, with 63, 63 and 65 skipped. On macOS with Node 22.12.0, it ran 9,232, none failing, with 48 skipped.mainhas since moved to4a45766f01(#869), and this head merges with it without conflicts.How many sends, before and after
In a simulation that isn't committed, the spec file's resident ran on a mocked clock stepped 1 s at a time for 4 hours, with the default re-arm delays of 2, 8 and 32 minutes and the spec file's error policy,
fastPolicy(test/exhausted-turn-rearm.test.ts:180). A 2-hour outage began with the first message: each turn's own sends got a server error, and each re-armed turn's sends followed the row's shape. Each stalled send was cut at once, so the rows count sends, not time. A message was said at the start only, or every 10 or every 30 minutes until the outage ended. The0f11975e48column ran with that commit'ssrc/framework.tsandsrc/types/framework.ts, and this change's column with7fa8f64930's, whosesrc/framework.tsis this head's and whosesrc/types/framework.tsdiffers from this head's only in a doc comment. Both ran with7fa8f64930's spec file with the simulation's rows added, on Linux with Node 24.22.1, membrane 0.8.0, context-manager 0.24.1 and chronicle 0.5.0.Sends during the outage,
0f11975e48→ this change. "+N" counts the sends after the outage, among which the last message said is answered.0f11975e48, and in some cells it also answers the last message after the outage where0f11975e48doesn't. None of its rows sends more than the only-server-errors row.0f11975e48or fewer. Except where every send runs a round and is then cut, none of its rows sends more than the row with server errors throughout.usageevent (src/framework.ts:26269), sofastPolicyretries each such cut.What each part is for
Each removal below ran against the spec file's 74 tests on
ead844dc69's files.takeAsTransienttaking a re-armed turn's first marked cut as transient is the change itself. Without it, 16 fail.:6477): without it, 7 fail.:6475): without the first, so that any failure of a re-armed turn that isn't transient is recorded and can be a first, 5 fail. Without the second, the test at:330fails.rearmExhaustedTurn, so that a hold, the error policy and the exhausted path read one answer for the failure: withrearmExhaustedTurndeciding again from the set instead, after the failure site has recorded the cut, 14 fail.rearmExhaustedTurn, fails 4, and deciding after the provider-hold check fails 1.:24250,:26096): without it, 2 fail.:6472), 2 fail; without theusageevent's (:26268), 1; withoutnoteSpokenTo's (:29604), 3; without the release's (:11892), 1. Limiting the first two to a re-armed turn's answers fails 2.Not verified
fastPolicyor a test's own policy. No run here usedDefaultErrorPolicy, so what it does with a re-armed turn's deadline cuts isn't shown.takeAsTransientat the outer catch (src/framework.ts:26512) with a marked cut, though a test that throws a plain error reaches it.turnStreamEventspasses a stream'serrorevent on (:2558), and takes a retryable failure its stream throws as its error event (:2566-2569). membrane 0.8.0 marks two kinds of cut. A retryable timeout is made atproviders/anthropic.js:757-764andtypes/errors.js:156-164,:313,:319and:760. In the spec file's stalled-stream helper, the cut of membrane's idle watchdog arrives as the stream'serrorevent (test/exhausted-turn-rearm.test.ts:137). An abort with aTimeoutErrorin its cause chain, whicherrorFromThrowntypes as an abort and marks (types/errors.js:742,:750), is emitted by membrane's stream as anabortedevent with reasonerror(types/errors.js:637,membrane.js:2119,:2631). In a probe that isn't committed, on Linux with Node 24.22.1 and with no membrane version recorded, aMembranewhose adapter threw that error was run, and the probe printed the one lineZZPROBE aborted reason=error message=zz aborted. The framework's branch for an aborted turn passesfalsetorearmExhaustedTurn(:26250). We found no path that gives the framework such an abort.rearmExhaustedTurnadds the re-armed request to it withholdRequest(:6514-6516), which keeps one of two equal requests (:6370-6376). No run had the hold already holding a request equal to the re-armed trigger.92bf4b182a, itsmainat 05:43 UTC on October 29, sets its oneproviderHoldatsrc/index.ts:517, fromquotaProviderHold, which returns no hold unless the error's type israte_limit(src/quota-meter.ts:198). No run here had a real host's hold take a marked cut.:409scripts theabortedevent with reasontimeoutthatturnStreamEventstakes as a marked cut (src/framework.ts:2554). No run here had membrane 0.8.0 itself cut a re-armed send at a bound frominferenceBounds.providers/anthropic.js:386,:392).changelog.d/871-rearm-deadline-cut.changed.mdEffie-2756 made this change, and Ada-1017 set the rule it follows. Desmond-2579 read the changes each of its commits makes but
6f97b2af38, which changed onlysrc/types/framework.ts's doc and #846's changelog fragment, and2c6c343543, which changed only this change's fragment. He read the merges of 0.29.0 and of0f11975e48too. The house's internal review ran five times on earlier versions, and the findings judged worth fixing are fixed at this head. Fiona-2757 wrote this description.