Skip to content

fix(x): make transport failure injection race-free, honor the RoundTripper contract and bound window sizes - #251

Merged
ffumero2003 merged 15 commits into
mainfrom
fix-x-issue-176-transport-race-window
Oct 30, 2026
Merged

ffumero2003 merged 15 commits into
mainfrom
fix-x-issue-176-transport-race-window

Conversation

@ffumero2003

@ffumero2003 ffumero2003 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem

The exported helpers under x/ are used for fault testing, including by blazer's own integration tests. #176 lists five problems:

  1. transport.Trigger writes the failure rate from a goroutine while RoundTrip reads it, which is a data race under -race. With context.Background() the goroutine never exits.
  2. Synthetic responses (Response(status)) neither read nor close req.Body, and return a nil Header, which breaks the http.RoundTripper contract.
  3. Stall still sends the request after the request context ends.
  4. window.New divides by zero when size < resolution (on first use) or resolution == 0 (inside New).
  5. internal/retry (used by every API call) and x/transport had no tests.

Fix

  • x/transport:
    • The failure rate is stored atomically (atomic.Uint64 holding the float64 bits).
    • Trigger uses context.AfterFunc instead of a goroutine, so nothing outlives the context.
    • Synthetic responses close req.Body without draining it (draining could block forever on a pipe that is never written), and set Header, Proto and ContentLength.
    • Stall returns ctx.Err() when the context ends first, without calling the wrapped transport. It also closes the request body in that case, which Fix the x/transport data race and RoundTripper contract, the x/window divide-by-zero, and add tests #176 didn't ask for but the RoundTripper contract requires once the request isn't forwarded. Its godoc says so.
  • x/window:
    • New panics with a documented message when a finite window has resolution <= 0, like time.NewTicker. No signature change.
    • It uses one bucket when size < resolution.
    • sweep compares absolute time-slot indices instead of array indices. The SDK panel found that with one bucket, the modulo comparison was always equal, so a window polled more often than its resolution never expired events.
  • Tests: every x/transport option is covered (Trigger, Response/Body, Stall with a cancelled and a live context, MatchPathSubstring, FailureRate at 0 and 1, an unwritten io.Pipe body), in x/transport/failures_test.go, x/window/new_bounds_test.go and internal/retry/retry_test.go, in new files so they don't collide with test: add credential-free tests for calls only covered by live tests #231 and test: tighten minor sync, simulator and window test assertions #233.
  • Changelog: two Fixed entries.

Verification

Each break was applied alone, failed its test, then passed after revert:

Break Failing output
sweep back to the modulo comparison Reduce after resolution boundary = 1, want nil
One-bucket rounding removed integer divide by zero
resolution <= 0 check removed New panic = runtime error: integer divide by zero, want documented resolution error
Trigger back to a goroutine Trigger(context.Background()) added goroutines: before=2 after=34
Synthetic response leaves the body open request body closed=false reads=0, want drained and closed
Synthetic response with nil Header the synthetic-response assertion fails
Stall sends after the context ends stalled RoundTrip error = <nil>, want context.Canceled
Backoff cap removed Backoff(31s) = 1m3.3s, want capped below 32s
Backoff stops doubling Backoff(1s) = 1.03s, want [2.02s, 2.06s)
Synthetic response drains the body again (io.Copy) synthetic RoundTrip did not return while pipe body is unwritten
MatchPathSubstring matches every path non-matching response = … StatusCode:500 …, want wrapped response
FailureRate(0) treated as 1 response = … StatusCode:500 …, want wrapped response
Stall with a live context skips the wrapped transport wrapped RoundTripper was not called after the stall

With no B2 credentials set: go test -race -count=20 ./internal/retry/... ./x/..., go test -race -count=5 -shuffle=on ./..., go vet ./... and make ci pass. The live tests that use Stall (base/integration_test.go) still compile; their intent is unchanged, because the request was cancelled anyway.

Housekeeping in this PR

Two small changes outside #176's list, kept here because they touch the same files:

  • 18558a7 ("chore: satisfy static analysis"): lint fixes the new tests needed.
  • ioutil.NopCloser → io.NopCloser in x/transport/transport.go (io/ioutil is deprecated); no behavior change.

Deferred (pre-existing, not changed here): with an already-cancelled context, Trigger followed by a later FailureRate option leaves the final rate depending on option order.

Closes #176

@ffumero2003 ffumero2003 added this to the 0.10.0 milestone Oct 30, 2026
@ffumero2003 ffumero2003 added bug Something isn't working testing Tests, test coverage and test infrastructure labels Oct 30, 2026
@ffumero2003 ffumero2003 self-assigned this Oct 30, 2026

@david-santacruz-backblaze david-santacruz-backblaze left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mate, please review following suggestions for this pr

  • x/transport/transport.go, RoundTrip, o.status > 0 branch: io.Copy(io.Discard, req.Body) is unbounded and ignores req.Context(). Response(500) with an io.Pipe body that is never written or closed blocks past 2s. #176 item 2 covers pipe bodies. Close req.Body without draining, or drain in a goroutine that stops on ctx.Done().
  • x/transport/failures_test.go: add a case with an io.Pipe body and a timeout. It should fail on the current code and pass after the fix.
  • Split out the chore: satisfy static analysis commit and the ioutil.NopCloser to io.NopCloser swap, or note them in the PR description. Neither is in #176.
  • Note in the PR description that Stall now closes req.Body on cancel. #176 only asks for ctx.Err().
  • Change Window.New to window.New in the changelog fragment and state that it panics when resolution <= 0.
  • Add tests for MatchPathSubstring, FailureRate at 0 and 1, and Stall with a live context. #176 asks for each option to be covered.

@ffumero2003

Copy link
Copy Markdown
Contributor Author

@david-santacruz-backblaze Thanks, all six are addressed (c548485, c3e1103, 3071ff3, 4d054b8), and the branch is synced with main:

  1. Unbounded drain: fixed. Synthetic responses now close req.Body without draining it.
  2. Pipe test: added. TestSyntheticResponseClosesUnwrittenPipeBody uses an unwritten io.Pipe with a 2s bound; restoring the io.Copy fails it with synthetic RoundTrip did not return while pipe body is unwritten.
  3. Out-of-scope changes: noted. The description has a "Housekeeping" section for 18558a7 and the ioutil → io swap.
  4. Stall closing the body on cancel: noted in the description, with why (the request is not forwarded, so the contract requires closing it).
  5. Changelog: fixed. The fragment says window.New and that it panics when a finite window's resolution is <= 0.
  6. Option coverage: added. MatchPathSubstring (matching vs non-matching), FailureRate(0) and FailureRate(1), and Stall with a live context. Each has a break in the table that fails it.

go test -race -count=20 ./x/... ./internal/retry/..., go test -race -count=5 -shuffle=on ./..., go vet ./... and make ci (0 lint issues) pass.

@ffumero2003
ffumero2003 merged commit be2885c into main Oct 30, 2026
23 of 24 checks passed
@ffumero2003
ffumero2003 deleted the fix-x-issue-176-transport-race-window branch October 30, 2026 20:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working testing Tests, test coverage and test infrastructure

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fix the x/transport data race and RoundTripper contract, the x/window divide-by-zero, and add tests

2 participants