Skip to content

test(b2): pin writer retry caps, backoff delays and fake-only bucket tests - #247

Merged
ffumero2003 merged 7 commits into
mainfrom
test-issue-204-b2-unit-assertions
Oct 30, 2026
Merged

ffumero2003 merged 7 commits into
mainfrom
test-issue-204-b2-unit-assertions

Conversation

@ffumero2003

@ffumero2003 ffumero2003 commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Context

Part of the test-integrity audit (#97). #204 is a checklist of tests that check less than their names say. This PR covers the b2/b2_test.go items of its ### b2 section (slice C1). Sibling PRs: #232 (base, merged), #233 (b2/sync, x, internal, merged), #240 (b2 API, merged), #248 (live tests and twins), #249 (production edges), and C4 (writer, reader and progress, in progress).

What each test now proves

Every claim below is backed by the assertion named, and by a break in the Verification table that fails it.

Item Test What it now asserts
Upload attempt cap TestWriterLimitNewUploadAttempts With one worker, exactly six uploadPart attempts (the first plus five reuploads). With two workers, two parts start concurrently and no part exceeds six attempts.
Read after double Close TestReaderDoubleClose Closes before EOF, then requires Read to return zero bytes and an error or EOF.
Fake-echo bucket tests TestNewBucketCORSAndFileLock Through the recording server: the locked bucket's b2_create_bucket body carries corsRules and fileLockEnabled: true; the plain bucket's body omits both (per the b2_create_bucket docs, CORS is optional and file lock defaults to false); Attrs sends b2_list_buckets filtered by bucketName ("locked", then "plain") and reads each bucket's own settings.
Bucket ID TestBucketID Literal IDs from the create and list responses; looking up the ID makes no request.
Invalid bucket encryption TestBucketDefaultEncryptionRejected The exact validation error. NewBucket sends one b2_list_buckets (it lists first, by design) and zero b2_create_bucket / b2_update_bucket for a rejected setting, for both a new and an existing bucket.
Backoff TestBackoff, TestBackoffWithoutRetryAfter, TestBackoffWithMaxRetriesReached The requested delays, captured through the existing wait hook: Retry-After exactly, the jitter bounds of the first retry, and the 30-second cap. Before, they counted waits only, so forcing every delay to 1ns passed.
Delete bypass TestObjectDeleteBypassGovernanceReachesTheWire (unchanged) Already asserts fileId, fileName and bypassGovernance on the wire, so the fake-only test is removed.

Moved out: item 1, TestWriterReturnsError. Its stronger assertion exposed a product bug (io.Copy into a Writer returned context canceled instead of the upload error), filed as #245 and fixed in #252, which tightens that test.

Tests removed, and what covers them now

Removed Covered by
TestObjectDeleteBypassGovernanceOption (asserted a bool recorded by the fake) TestObjectDeleteBypassGovernanceReachesTheWire (b2/delete_bypass_wire_test.go)
TestNewBucketReplicationConfig (the fake echoed its arguments) TestNewBucketReplicationConfigurationWireRoundTrip (b2/replication_create_wire_test.go)
TestBucketAttrsCORSAndFileLockPerBucket (fake-only) TestNewBucketCORSAndFileLock, which now asserts the request bodies through the recording server

No assertion was narrowed elsewhere: the single-worker upload-cap case stays, and the two-worker case was added alongside it.

Verification

Each break was applied alone; the named test failed, then passed after revert. ("b2/b2.go:446" in 4c and 4d is the if bucket.name() == name match in Client.Bucket.)

Item Break Failing output
2a Reupload limit +1 → +2 uploadPart attempts = 7, want 6 (initial attempt plus five reuploads)
2b Initial-attempt guard flipped uploadPart attempts = 1, want 6 (initial attempt plus five reuploads)
2c Initial-attempt guard deleted uploadPart attempts = 1, want 6 (initial attempt plus five reuploads)
2d b2/writer.go:952 reupload cap +1 → +2, two workers uploadPart:1 attempts = 7, want at most 6; uploadPart:2 attempts = 7, want at most 6
3a Reader.Read returns (0, nil) after Close Read after Close = 0, <nil>; want 0 and an error or EOF
3b Reader.Read returns (1, nil) after Close Read after Close = 1, <nil>; want 0 and an error or EOF
4a Omit CORS in create mapping b2_create_bucket body corsRules: missing, want [{"allowedHeaders":["authorization"],"allowedOperations":["b2_download_file_by_name"],"allowedOrigins":["https://example.com"],"corsRuleName":"browser","exposeHeaders":["x-bz-file-name"],"maxAgeSeconds":60}]
4b Omit file lock in create mapping b2_create_bucket body fileLockEnabled: missing, want true
4c b2/b2.go:446, Client.Bucket name-match condition if bucket.name() == name flipped Bucket.Attrs().FileLockEnabled = false, want true
4d b2/b2.go:446, same Client.Bucket name-match condition removed so the first listed bucket returns unconditionally plain bucket attrs = … FileLockEnabled:true …, want no CORS rules and no file lock
4e Omit replication in create mapping create omitted replicationConfiguration
4f base/base.go:829 sends a CORS rule for the plain create b2_create_bucket body corsRules: got [{"corsRuleName":"unexpected"}], want missing
4g base/base.go:830 forces fileLockEnabled: true for the plain create b2_create_bucket body fileLockEnabled: got true, want missing
4h b2/baseline.go:278 drops the name passed to ListBuckets b2_list_buckets body bucketName: missing, want "locked"; … want "plain"
5a Bypass option assignment flipped to false Delete(BypassGovernance) sent bypassGovernance = <nil>, want true
5b Bypass option assignment deleted Delete(BypassGovernance) sent bypassGovernance = <nil>, want true
7a NewBucket encryption guard flipped invalid default sent 1 create requests, want 0
7b NewBucket encryption guard deleted invalid default sent 1 create requests, want 0
7c Update encryption guard flipped invalid default sent 1 update requests, want 0
7d Update encryption guard deleted invalid default sent 1 update requests, want 0
7e Move NewBucket encryption validation before b2/b2.go:465 list NewBucket listed 0 times, want 1
7f Flip b2/b2.go:470 existing-bucket name match NewBucket(existing) sent 1 create requests, want 0
8a Force backend delay to 1ns; TestBackoff wait 1 = 1ns, want [1s, 1s]; wait 2 = 1ns, want [2s, 2s]; wait 1 = 1ns, want [1.01s, 1.03s]
8b Force backend delay to 1ns; TestBackoffWithoutRetryAfter wait 1 = 1ns, want [1.01s, 1.03s]; wait 7 = 1ns, want [30s, 32s]
8c Force backend delay to 1ns; TestBackoffWithMaxRetriesReached wait 1 = 1ns, want [1.01s, 1.03s]; wait 2 = 1ns, want [2.04s, 2.122s]
8d First-delay guard flipped wait 1 = 0s, want [1.01s, 1.03s]
8e First-delay guard deleted wait 1 = 0s, want [1.01s, 1.03s]
8f Retry-After guard flipped wait 1 = 1.011091425s, want [1s, 1s]
8g Retry-After guard deleted wait 1 = 1.02398841s, want [1s, 1s]
8h 30-second cap guard flipped wait 1 = 30.010863997s, want [1.01s, 1.03s]
8i 30-second cap guard deleted wait 7 = 1m11.547314088s, want [30s, 32s]; wait 8 = 2m26.982487425s, want [30s, 32s]

With no B2 credentials set: go test -race -count=5 -shuffle=on ./b2/... passes, and the changed timing and concurrency tests pass 30 runs in a row under -race.

make ci (tail)

go build ./...
go vet ./...
go mod tidy -diff
go mod verify
all modules verified
0 issues.
go test -race -shuffle=on ./...
ok  	github.com/Backblaze/blazer/b2	15.605s
ok  	github.com/Backblaze/blazer/b2/sync	5.048s
ok  	github.com/Backblaze/blazer/base	7.533s
ok  	github.com/Backblaze/blazer/internal/b2types	2.702s
ok  	github.com/Backblaze/blazer/internal/b2wiretest	2.268s
ok  	github.com/Backblaze/blazer/internal/blog	1.435s
ok  	github.com/Backblaze/blazer/internal/retry	2.861s
ok  	github.com/Backblaze/blazer/x/b2sim	3.617s
ok  	github.com/Backblaze/blazer/x/consistent	2.567s
ok  	github.com/Backblaze/blazer/x/transport	3.291s
ok  	github.com/Backblaze/blazer/x/window	2.313s
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build ./...
CGO_ENABLED=0 GOOS=linux GOARCH=386 go build ./...
CGO_ENABLED=0 GOOS=linux GOARCH=arm go build ./...
CGO_ENABLED=0 GOOS=darwin GOARCH=arm64 go build ./...
CGO_ENABLED=0 GOOS=windows GOARCH=amd64 go build ./...
... scripts/*_test.sh: all ok

Part of #204

Reuse the recording server for CORS, file lock, bucket IDs and default-encryption validation. Remove fake tests already covered by the replication and delete-bypass wire tests. Record isolated mutation proofs for the C1 checklist; stop the Writer error-identity item because current main returns context cancellation instead of the injected error.
@ffumero2003 ffumero2003 added this to the 0.10.0 milestone Oct 30, 2026
@ffumero2003 ffumero2003 added the skip-changelog No changelog entry needed; skips the changelog check label Oct 30, 2026
@ffumero2003
ffumero2003 requested a review from goanpeca as a code owner October 30, 2026 18:51
@ffumero2003 ffumero2003 self-assigned this Oct 30, 2026

@david-santacruz-backblaze david-santacruz-backblaze left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's review these issues pls.

  • The PR body mentions the bypass twin asserting fileId and fileName, but it looks like it only checks bypassGovernance. Maybe worth a look at the wording, or at adding those assertions?
  • plan.md doesn't seem to have mutation rows for TestReaderDoubleClose where Read after Close returns 0, nil or 1, nil. Row 3 only shows a hang. Could be nice to add them.
  • TestBucketDefaultEncryptionRejected: NewBucket still sends b2_list_buckets, so "no request is sent" might be a bit strong. The existing-bucket subcase also doesn't assert request counts.
  • TestNewBucketCORSAndFileLock doesn't seem to assert the plain bucket's create body.
  • The PR body says per-bucket attribute reads assert the request body, but it looks like only create bodies are asserted.
  • plan.md rows 4c and 4d mention a "Bucket-name guard" that I couldn't match to the diff. Which guard is it?
  • TestWriterLimitNewUploadAttempts now runs with ConcurrentUploads=1, so concurrent cap behaviour may no longer be covered. Maybe worth a quick look?
  • Minor: I haven't run make ci or the 25-row mutation table, so those are unverified on my side.

@ffumero2003

Copy link
Copy Markdown
Contributor Author

@david-santacruz-backblaze Thanks, all addressed (5a572f0, 0c3e739), and the branch is synced with main. I rewrote the description from the final diff: each claim now maps to the assertion that proves it, and there is a table of the removed tests and what covers them.

  1. Bypass twin fileId/fileName: after syncing with main (test(b2): tighten API-facing b2 test assertions #240 added them), TestObjectDeleteBypassGovernanceReachesTheWire asserts fileId, fileName and bypassGovernance. No duplicates added.
  2. TestReaderDoubleClose: added breaks where Read after Close returns (0, nil) and (1, nil). Both fail: Read after Close = 0, <nil>; want 0 and an error or EOF (and the same with 1).
  3. TestBucketDefaultEncryptionRejected: fixed. It now asserts exact counts: one b2_list_buckets (NewBucket lists first) and zero b2_create_bucket / b2_update_bucket for rejected settings, in both the new- and existing-bucket subcases. The wording now says "no create or update request". Moving validation before the list fails with NewBucket listed 0 times, want 1; re-creating an existing bucket fails with sent 1 create requests, want 0.
  4. Plain bucket create body: added. It omits corsRules and fileLockEnabled, per the b2_create_bucket docs. Breaks fail with corsRules: got [...], want missing and fileLockEnabled: got true, want missing.
  5. Per-bucket attribute reads: now assert the request. b2_list_buckets is filtered by bucketName ("locked", then "plain"); dropping the name fails with bucketName: missing, want "locked".
  6. "Bucket-name guard": that is if bucket.name() == name in Client.Bucket (b2/b2.go:446). Rows 4c and 4d now name it.
  7. Concurrent cap: restored. TestWriterLimitNewUploadAttempts keeps the single-worker exact-count case and adds a two-worker case: two parts start concurrently and no part exceeds six attempts. Raising the cap fails with uploadPart:1 attempts = 7, want at most 6.

Also, as on #248 and #249: the intent folder is removed (the table is in the description), and the make ci tail is in the description.

@ffumero2003
ffumero2003 merged commit 13e8574 into main Oct 30, 2026
23 of 24 checks passed
@ffumero2003
ffumero2003 deleted the test-issue-204-b2-unit-assertions branch October 30, 2026 21:25
ffumero2003 added a commit that referenced this pull request Oct 30, 2026
#247 removed objectForDeleteOptionTest from b2_test.go while #249's
TestObjectDeleteRejectsFolderStatus still used it, so the b2 tests no
longer compile on main. Use the same setup as TestObjectDeleteAllowsCopyStatus.
ffumero2003 added a commit that referenced this pull request Oct 30, 2026
…ext canceled (#252)

* test(b2): reproduce writes after copy cancellation

Pin delayed source reads that resume after copyContext returns and Writer.ReadFrom has been cancelled and closed.

Part of #229

* fix(b2): stop a cancelled copy from writing after copyContext returns

A cancelled copy could return while its source Read was still blocked. When that Read resumed, the copy goroutine could write to a Writer that the caller had already closed, causing the slice-bounds panic seen in CI run 40925001975.

Guard destination writes and mark the guard stopped before returning on cancellation. Waiting for the copy goroutine is not an option because the source Read may block forever. Preserve the closer path that closes the body and waits for its copy to finish.

Closes #229

* test(b2): match the abandoned-copy error with errors.Is and document ReadFrom cancellation

Address the panel's nits on #229: the regression test compared the error
text, so wrapping errCopyAbandoned would break it while the guard still
works; and ReadFrom's godoc did not say that a Read already in progress on a
non-seekable source can outlive a cancelled call.

* docs(b2): state exactly what ReadFrom guarantees after a cancelled copy

The comment said nothing a pending Read returns is written, but the guard
only stops writes once copyContext returns: a Write already in progress, with
data read before the cancellation, can still complete. Say what the code
guarantees: after ReadFrom returns, nothing more from r reaches the Writer.

* test(b2): pin ReadFrom cancellation during blocked error cleanup

Hold cancel_large_file or WithCancelOnError callbacks behind channels after producer and worker upload failures. ReadFrom must return on cancellation before those channels are released, and the callback must receive the cancellation request result (#229).

* fix(b2): release the copy guard before upload error cleanup

Keep the existing Writer error path and release the copy guard only on terminal write errors. No buffer writes follow that hook, and the stopped check still gates every later write, so no source bytes can reach the destination after a cancelled copy returns.

Release the error-state mutex before cancel_large_file and errf as well, so a producer observing a worker failure does not wait behind cleanup. Close joins its existing worker wait group on error exits to preserve delivery of the cleanup result to WithCancelOnError callers.

Reject a separate deferred error-recording and cleanup mechanism: reusing setErr preserves first-error ownership and callback semantics without adding parallel state. The public API and ReadFrom godoc remain unchanged.

Closes #229

* test(b2): pin error mutex release and Close cleanup waiting

Hold the producer inside buffer allocation until a worker enters blocked cancellation cleanup, then release it into getErr while the copy guard is still held. This makes moving cleanup back under emux fail deterministically.

Gate the worker cancellation request separately and verify that its nonblocking error callback has completed when Close returns. Removing the deferred worker wait now fails without making the callback depend on Close.

Keep the existing cancellation cases and production code unchanged.

* docs(b2): document Writer.Close callback wait

* test(b2): make Writer.Close callback proof deterministic

* test(b2): pin Writer.ReadFrom upload error precedence

* fix(b2): preserve upload errors from Writer.ReadFrom

* test(b2): pin ReadFrom caller cancellation with recorded transport errors

* fix(b2): preserve caller cancellation cause in Writer.ReadFrom

* test(b2): pin caller custom causes in Writer.ReadFrom

* fix(b2): distinguish Writer cancellation from caller causes

* test(b2): start ReadFrom deadline after upload setup

* docs(b2): clarify Writer ReadFrom error identity

* test(b2): bound copy cancellation and Close cleanup waits

* docs(b2): clarify cancellation and breaking Close wait contract

* test(b2): acknowledge cancellation stub body close results

* test(b2): inline the folder-delete test setup removed by #247

#247 removed objectForDeleteOptionTest from b2_test.go while #249's
TestObjectDeleteRejectsFolderStatus still used it, so the b2 tests no
longer compile on main. Use the same setup as TestObjectDeleteAllowsCopyStatus.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog No changelog entry needed; skips the changelog check

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants