Repository navigation
test(silent-heartbeat-rows): the retried reply's failure states no wait - #807
Merged
nissa-seru merged 1 commit intoOct 26, 2026
Conversation
"a reply retried after a failure past its first round keeps following the
prompt" failed now and then (about 1 run in 10 on main in one reviewer's
runs). Forced, it fails every time with "timed out waiting for the retry's
call".
The test's failure, transientFailure(), stated retryAfterMs: 1. A stated
wait is recorded as a provider wait (until = the recording's clock + 1 ms),
and holdProviderAcceleration hands it to holdForProviderWait, which holds
the turn's retry unless until <= Date.now(). So which path the retry took
depended on whether the clock moved on between the two reads:
- moved on: the error policy retries after its backoff (1 s), still inside
the first runUntilIdle(), since the failed frame's activeStreams entry
keeps the loop going. That's what the test says it checks ("the error
policy retries").
- same millisecond: the retry is held. The first runUntilIdle() then
returns at 3 calls with nothing queued, pending or streaming, the hold's
1 ms timer puts the retry in pendingRequests, and nothing runs it in a
framework the test drives itself, so waitFor(calls >= 4) times out.
Forcing the hold check to read the millisecond the wait was recorded in
failed 30 runs of 30 that way, and moving the wait's clock 500 ms ahead
failed 6 of 6. On the host those runs used, the two reads are 7 to 325 ms
apart (median 38, 40 runs), so the held path never came up there in over 300
unforced runs.
Without a stated wait nothing is recorded or held, and the error policy
retries, the path every unforced run took, so the test's assertions are
unchanged. A retry held by a provider cooldown has its own test, "a reply
resumed after a provider cooldown follows the message that arrived during
it, not [Continue]", which holds on a 429 and drives the release itself; a
stated wait's hold is released the same way.
The helper's comment went too: it said the default error policy retries
"at once", and its base schedule waits 1 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Automated review
Recommendation: merge. The review found nothing of medium or high severity.
Reviewed commit 1a5596b with claude-opus-5-5, by reading the code. It did not run the code or look at the checks.
Does this change belong in the project?
It fits the project. This change fits. It is a one-line, test-only fix to a flaky existing test so that it always takes the error-policy retry path the test is about, and it follows the project's PR conventions.
4 reasons with sources
- It only fixes a test of behavior the framework already has, and makes no product change. The diff drops
retryAfterMs: 1from the test helpertransientFailure()and removes that helper's comment. Nothing under src/ changes. (test/silent-heartbeat-rows.test.ts) - The PR body has the Problem / Changes / Tests / Not verified shape and gives numeric suite results, as the guide asks. It reports 7,528 pass, 0 fail and 44 skipped, and lists what it did not verify, including the reported failures themselves and Node 22 at the fix. (CONTRIBUTING.md, What a PR should contain)
- Skipping a changelog fragment is correct for a test-only change. The changelog CI check applies only to changes that touch src/. This PR touches only test/ and declares
no-changelog. (CONTRIBUTING.md, Changelog) - It is one coherent change and carries no unrelated work. (test/silent-heartbeat-rows.test.ts)
Findings
No findings of medium or high severity.
1 more finding, less severe
- Low severity, documentation,
test/silent-heartbeat-rows.test.ts:116-119. transientFailure() lost its comment, so nothing says why it must not state a retry-after. The PR deletes the inaccurate doc comment ontransientFailure()and adds nothing in its place. The neighbouring helper,accelerationLimit()at line 109, keeps its comment saying what it is for. Leaving outretryAfterMsis now what keeps this test away from the provider-wait hold, whichrunUntilIdle()never drains, and no comment says so. Someone who later adds a realisticretryAfterMsback to the error would bring the intermittent timeout back without warning. A one-line comment would cover it, for example: no stated wait, so the default error policy alone schedules the retry after its base delay and no provider wait holds it.
For a deeper review by several readers and a judge, which takes longer, a maintainer can comment /review deep.
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
test/silent-heartbeat-rows.test.ts's "a reply retried after a failure past its first round keeps following the prompt" fails now and then. The provider failure it injects stated a 1 ms retry-after. When the framework's two reads of the clock fall in the same millisecond, it holds the retry for that wait instead of leaving it to the error policy, and the test then times out waiting for the retry's call, since nothing in it runs a retry once a hold releases it. The failure now states no wait, so the test always takes the error-policy path it describes. Test-only.Problem
@Anarchid's review of #762 reported this test failing in its full-suite run of that PR merged into
main, on Node 22.22.1, and in 1 of 10 runs of the test alone onmain.By reading
mainat0a4a2c69:transientFailure()was a retryable 500 withretryAfterMs: 1(test/silent-heartbeat-rows.test.ts:117-120).providerWaits.setreads the clock once and recordsuntilas that reading plus 1 ms (src/provider-waits.ts:296,:70).holdProviderAccelerationpasses the wait toholdForProviderWait(src/framework.ts:5093,:5107), which holds the turn's retry unlessuntil <= Date.now()(:5327). So the retry is held when the second read is no later than the first: when both fall in the same millisecond, or when the clock is set back between them.pendingRequests(:5250). A started framework's loop runs it. This test drives the framework withrunUntilIdle()instead, and nothing runs it between those calls.By run at
0a4a2c69, on Linux with Node 24.22.1 and 12 CPUs. Some runs used a copy of the test that logs each call, each trace event and the framework's state.runUntilIdle(). On this host the two clock reads were 7 to 325 ms apart (median 38 ms, 40 runs), so the 1 ms wait never held the retry.runUntilIdle()returned after 3 calls, with nothing queued, pending or streaming and the agent idle. Then the hold's 1 ms timer released the retry intopendingRequests, andwaitFortimed out after 15 s with "timed out waiting for the retry's call".Changes
transientFailure()states noretryAfterMs. With no stated wait,recordProviderWaitrecords nothing (src/framework.ts:5292) and nothing holds the retry. The default error policy retries after its base delay of 1 s. Every logged unforced run took that path, so the test's assertions are unchanged.transientFailure()has no other user.Tests
At
1a5596b8's tree, on Linux with Node 24.22.1 and 12 CPUs:main: 30 of 30 and 6 of 6 pass. No wait is recorded, and the logs show no hold check at all.Not verified
mainused Node 22.22.1. The forced runs and the runs at the fix used Node 24.22.1.What each part is for, and what we cut
retryAfterMs: the test is about the error policy's retry ("the error policy retries"), and the stated wait is what let a run hold the retry instead.test/held-turn-retry.test.ts, "a resident whose turn a stated wait holds for its retry is mid-turn: … the hold's end runs the retry" holds one on a stated wait under a controlled clock.no-changelog).Written by Kestrel-2409 (Claude Opus 5.5). Ada-1017 (Claude Opus 5.5) chose its shape, and Effie-2289 (Claude Opus 5.5) read it.
Description checked and posted by Fiona-2757 (Claude Opus 5.5).
🤖 Written using Anthropic's Claude Opus 5.5 through the Rookery's tooling, and posted from the shared
nissa-seruaccount.