fix: recover interrupted model response streams - #39
Merged
Merged
Conversation
Retry response-body read errors, read timeouts, and remote protocol errors at most twice (1s, 2s) before committing a reply to history or running any of its tools. Handle both httpx and httpx2 transport families; the CLI previously only caught httpx.HTTPError, so SDK 1.5.0 failures on httpx2 ended the run. Retries reuse the last complete conversation, never execute tools from an interrupted attempt, and do not consume the turn budget. A 300-second window starting at the first interruption bounds new retry scheduling, while in-flight requests keep the SDK timeout and external deadline. Record each attempt as a journal v3 `model.failed` event and project it as an incomplete ATIF step. Reconcile generation costs for interrupted attempts and flag token and cost totals as partial when usage is missing.
Add the full 20-task live retest to the 0.8.x dev notes, with its changelog entry and the regenerated English dev notes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Recover interrupted model response streams instead of ending the run.
The SDK retries failures that happen before a response stream opens, but an
interruption during response-body iteration (
ReadError,ReadTimeout, orRemoteProtocolError) previously propagated out of the agent loop. SDK 1.5.0uses
httpx2, while the CLI only caughthttpx.HTTPError; the two exceptionfamilies share no base class, so a mid-stream failure ended the session with no
retry.
Changes
agent.py,transport.py): retry response-bodyread errors, read timeouts, and remote protocol errors at most twice, waiting
1s and 2s. Retries reuse the last committed conversation, never execute tools
from an interrupted attempt, and do not consume the turn budget. A 300-second
window starting at the first interruption bounds new retry scheduling; an
in-flight request keeps the SDK timeout and external run deadline.
httpxandhttpx2exceptionfamilies through a small
transportmodule, fixing the CLI's SDK 1.5.0 gap.event_journal.py,event-journal-protocol-v3.md): recordeach attempt as a
model.failedevent with the original error, retry intent,duration, and the generation ID captured before reading the body. Readers
still accept v1/v2; v1/v2 records containing
model.failedare rejected.atif.py,agent.py): projectfailed attempts as chronological incomplete steps, reconcile billing for
interrupted generations, and mark token and cost totals partial when usage is
missing.
protocol source plus English version, and the full 20-task live retest in the
0.8.x dev notes (zh + en).
locked dependencies.
Verification
tests/test_stream_recovery.py(run during the live experiment)Offline replay consumes the original captured response bytes through SDK 1.5.0,
raises the recorded transport error, and supplies a synthetic successful retry.
These replays make no external requests and add no task score.
Live experiment (20 tasks)
A full 20-task Terminal-Bench 2.1 run on this branch finished 20/20 trials:
11 reward 1, 6 reward 0, 3 unscored. All 576 model calls returned completely;
model.failedand interrupted HTTP response bodies were 0, so the recoverybranch was not exercised by real traffic. The run can only say "no stream
interruption occurred this run", not "live recovery succeeded"; the mechanism is
covered by the automated tests. Details are in
docs/dev_notes/{zh-CN,en}/0.8.x.mdand the Git-ignored job record
jobs/tb21-stream-recovery-pilot20-20260921-record/.Notes
docs/research/changes.Fixedentry added under[Unreleased].