Repository navigation
test(b2): pin writer retry caps, backoff delays and fake-only bucket tests - #247
Merged
Merged
Conversation
Reuse the recording server for CORS, file lock, bucket IDs and default-encryption validation. Remove fake tests already covered by the replication and delete-bypass wire tests. Record isolated mutation proofs for the C1 checklist; stop the Writer error-identity item because current main returns context cancellation instead of the injected error.
david-santacruz-backblaze
suggested changes
Oct 30, 2026
david-santacruz-backblaze
left a comment
There was a problem hiding this comment.
Let's review these issues pls.
- The PR body mentions the bypass twin asserting
fileIdandfileName, but it looks like it only checksbypassGovernance. Maybe worth a look at the wording, or at adding those assertions? plan.mddoesn't seem to have mutation rows forTestReaderDoubleClosewhereReadafterClosereturns0, nilor1, nil. Row 3 only shows a hang. Could be nice to add them.TestBucketDefaultEncryptionRejected:NewBucketstill sendsb2_list_buckets, so "no request is sent" might be a bit strong. The existing-bucket subcase also doesn't assert request counts.TestNewBucketCORSAndFileLockdoesn't seem to assert the plain bucket's create body.- The PR body says per-bucket attribute reads assert the request body, but it looks like only create bodies are asserted.
plan.mdrows 4c and 4d mention a "Bucket-name guard" that I couldn't match to the diff. Which guard is it?TestWriterLimitNewUploadAttemptsnow runs withConcurrentUploads=1, so concurrent cap behaviour may no longer be covered. Maybe worth a quick look?- Minor: I haven't run
make cior the 25-row mutation table, so those are unverified on my side.
Contributor
Author
|
@david-santacruz-backblaze Thanks, all addressed (5a572f0, 0c3e739), and the branch is synced with main. I rewrote the description from the final diff: each claim now maps to the assertion that proves it, and there is a table of the removed tests and what covers them.
Also, as on #248 and #249: the intent folder is removed (the table is in the description), and the |
david-santacruz-backblaze
approved these changes
Oct 30, 2026
This was referenced Oct 30, 2026
ffumero2003
added a commit
that referenced
this pull request
Oct 30, 2026
…ext canceled (#252) * test(b2): reproduce writes after copy cancellation Pin delayed source reads that resume after copyContext returns and Writer.ReadFrom has been cancelled and closed. Part of #229 * fix(b2): stop a cancelled copy from writing after copyContext returns A cancelled copy could return while its source Read was still blocked. When that Read resumed, the copy goroutine could write to a Writer that the caller had already closed, causing the slice-bounds panic seen in CI run 40925001975. Guard destination writes and mark the guard stopped before returning on cancellation. Waiting for the copy goroutine is not an option because the source Read may block forever. Preserve the closer path that closes the body and waits for its copy to finish. Closes #229 * test(b2): match the abandoned-copy error with errors.Is and document ReadFrom cancellation Address the panel's nits on #229: the regression test compared the error text, so wrapping errCopyAbandoned would break it while the guard still works; and ReadFrom's godoc did not say that a Read already in progress on a non-seekable source can outlive a cancelled call. * docs(b2): state exactly what ReadFrom guarantees after a cancelled copy The comment said nothing a pending Read returns is written, but the guard only stops writes once copyContext returns: a Write already in progress, with data read before the cancellation, can still complete. Say what the code guarantees: after ReadFrom returns, nothing more from r reaches the Writer. * test(b2): pin ReadFrom cancellation during blocked error cleanup Hold cancel_large_file or WithCancelOnError callbacks behind channels after producer and worker upload failures. ReadFrom must return on cancellation before those channels are released, and the callback must receive the cancellation request result (#229). * fix(b2): release the copy guard before upload error cleanup Keep the existing Writer error path and release the copy guard only on terminal write errors. No buffer writes follow that hook, and the stopped check still gates every later write, so no source bytes can reach the destination after a cancelled copy returns. Release the error-state mutex before cancel_large_file and errf as well, so a producer observing a worker failure does not wait behind cleanup. Close joins its existing worker wait group on error exits to preserve delivery of the cleanup result to WithCancelOnError callers. Reject a separate deferred error-recording and cleanup mechanism: reusing setErr preserves first-error ownership and callback semantics without adding parallel state. The public API and ReadFrom godoc remain unchanged. Closes #229 * test(b2): pin error mutex release and Close cleanup waiting Hold the producer inside buffer allocation until a worker enters blocked cancellation cleanup, then release it into getErr while the copy guard is still held. This makes moving cleanup back under emux fail deterministically. Gate the worker cancellation request separately and verify that its nonblocking error callback has completed when Close returns. Removing the deferred worker wait now fails without making the callback depend on Close. Keep the existing cancellation cases and production code unchanged. * docs(b2): document Writer.Close callback wait * test(b2): make Writer.Close callback proof deterministic * test(b2): pin Writer.ReadFrom upload error precedence * fix(b2): preserve upload errors from Writer.ReadFrom * test(b2): pin ReadFrom caller cancellation with recorded transport errors * fix(b2): preserve caller cancellation cause in Writer.ReadFrom * test(b2): pin caller custom causes in Writer.ReadFrom * fix(b2): distinguish Writer cancellation from caller causes * test(b2): start ReadFrom deadline after upload setup * docs(b2): clarify Writer ReadFrom error identity * test(b2): bound copy cancellation and Close cleanup waits * docs(b2): clarify cancellation and breaking Close wait contract * test(b2): acknowledge cancellation stub body close results * test(b2): inline the folder-delete test setup removed by #247 #247 removed objectForDeleteOptionTest from b2_test.go while #249's TestObjectDeleteRejectsFolderStatus still used it, so the b2 tests no longer compile on main. Use the same setup as TestObjectDeleteAllowsCopyStatus.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Part of the test-integrity audit (#97). #204 is a checklist of tests that check less than their names say. This PR covers the
b2/b2_test.goitems of its### b2section (slice C1). Sibling PRs: #232 (base, merged), #233 (b2/sync, x, internal, merged), #240 (b2 API, merged), #248 (live tests and twins), #249 (production edges), and C4 (writer, reader and progress, in progress).What each test now proves
Every claim below is backed by the assertion named, and by a break in the Verification table that fails it.
TestWriterLimitNewUploadAttemptsuploadPartattempts (the first plus five reuploads). With two workers, two parts start concurrently and no part exceeds six attempts.TestReaderDoubleCloseReadto return zero bytes and an error or EOF.TestNewBucketCORSAndFileLockb2_create_bucketbody carriescorsRulesandfileLockEnabled: true; the plain bucket's body omits both (per theb2_create_bucketdocs, CORS is optional and file lock defaults to false);Attrssendsb2_list_bucketsfiltered bybucketName("locked", then"plain") and reads each bucket's own settings.TestBucketIDTestBucketDefaultEncryptionRejectedNewBucketsends oneb2_list_buckets(it lists first, by design) and zerob2_create_bucket/b2_update_bucketfor a rejected setting, for both a new and an existing bucket.TestBackoff,TestBackoffWithoutRetryAfter,TestBackoffWithMaxRetriesReachedRetry-Afterexactly, the jitter bounds of the first retry, and the 30-second cap. Before, they counted waits only, so forcing every delay to 1ns passed.TestObjectDeleteBypassGovernanceReachesTheWire(unchanged)fileId,fileNameandbypassGovernanceon the wire, so the fake-only test is removed.Moved out: item 1,
TestWriterReturnsError. Its stronger assertion exposed a product bug (io.Copyinto aWriterreturnedcontext canceledinstead of the upload error), filed as #245 and fixed in #252, which tightens that test.Tests removed, and what covers them now
TestObjectDeleteBypassGovernanceOption(asserted a bool recorded by the fake)TestObjectDeleteBypassGovernanceReachesTheWire(b2/delete_bypass_wire_test.go)TestNewBucketReplicationConfig(the fake echoed its arguments)TestNewBucketReplicationConfigurationWireRoundTrip(b2/replication_create_wire_test.go)TestBucketAttrsCORSAndFileLockPerBucket(fake-only)TestNewBucketCORSAndFileLock, which now asserts the request bodies through the recording serverNo assertion was narrowed elsewhere: the single-worker upload-cap case stays, and the two-worker case was added alongside it.
Verification
Each break was applied alone; the named test failed, then passed after revert. ("
b2/b2.go:446" in 4c and 4d is theif bucket.name() == namematch inClient.Bucket.)+1→+2uploadPart attempts = 7, want 6 (initial attempt plus five reuploads)uploadPart attempts = 1, want 6 (initial attempt plus five reuploads)uploadPart attempts = 1, want 6 (initial attempt plus five reuploads)b2/writer.go:952reupload cap+1→+2, two workersuploadPart:1 attempts = 7, want at most 6;uploadPart:2 attempts = 7, want at most 6Reader.Readreturns(0, nil)after CloseRead after Close = 0, <nil>; want 0 and an error or EOFReader.Readreturns(1, nil)after CloseRead after Close = 1, <nil>; want 0 and an error or EOFb2_create_bucket body corsRules: missing, want [{"allowedHeaders":["authorization"],"allowedOperations":["b2_download_file_by_name"],"allowedOrigins":["https://example.com"],"corsRuleName":"browser","exposeHeaders":["x-bz-file-name"],"maxAgeSeconds":60}]b2_create_bucket body fileLockEnabled: missing, want trueb2/b2.go:446,Client.Bucketname-match conditionif bucket.name() == nameflippedBucket.Attrs().FileLockEnabled = false, want trueb2/b2.go:446, sameClient.Bucketname-match condition removed so the first listed bucket returns unconditionallyplain bucket attrs = … FileLockEnabled:true …, want no CORS rules and no file lockcreate omitted replicationConfigurationbase/base.go:829sends a CORS rule for the plain createb2_create_bucket body corsRules: got [{"corsRuleName":"unexpected"}], want missingbase/base.go:830forcesfileLockEnabled: truefor the plain createb2_create_bucket body fileLockEnabled: got true, want missingb2/baseline.go:278drops the name passed toListBucketsb2_list_buckets body bucketName: missing, want "locked";… want "plain"Delete(BypassGovernance) sent bypassGovernance = <nil>, want trueDelete(BypassGovernance) sent bypassGovernance = <nil>, want trueinvalid default sent 1 create requests, want 0invalid default sent 1 create requests, want 0invalid default sent 1 update requests, want 0invalid default sent 1 update requests, want 0b2/b2.go:465listNewBucket listed 0 times, want 1b2/b2.go:470existing-bucket name matchNewBucket(existing) sent 1 create requests, want 0TestBackoffwait 1 = 1ns, want [1s, 1s];wait 2 = 1ns, want [2s, 2s];wait 1 = 1ns, want [1.01s, 1.03s]TestBackoffWithoutRetryAfterwait 1 = 1ns, want [1.01s, 1.03s];wait 7 = 1ns, want [30s, 32s]TestBackoffWithMaxRetriesReachedwait 1 = 1ns, want [1.01s, 1.03s];wait 2 = 1ns, want [2.04s, 2.122s]wait 1 = 0s, want [1.01s, 1.03s]wait 1 = 0s, want [1.01s, 1.03s]wait 1 = 1.011091425s, want [1s, 1s]wait 1 = 1.02398841s, want [1s, 1s]wait 1 = 30.010863997s, want [1.01s, 1.03s]wait 7 = 1m11.547314088s, want [30s, 32s];wait 8 = 2m26.982487425s, want [30s, 32s]With no B2 credentials set:
go test -race -count=5 -shuffle=on ./b2/...passes, and the changed timing and concurrency tests pass 30 runs in a row under-race.make ci(tail)Part of #204