Repository navigation
sdkharness: observe real concurrency in large-transfer checks; harden resilience executable - #1182
sophiecarreras wants to merge 3 commits into
Conversation
…ilience executable - large.concurrent_parts and large.parallel_download now observe overlapping b2_upload_part requests and download streams (ranged GETs) through a loopback observing proxy with a client-side hold, and verify the uploaded bytes. - resilience-upload.py follows LATEST_STABLE_VERSION and asserts it runs from this checkout, accepts contentSha1 'none' only for large files, names the scenario in the result line on configuration failure, and retries only its own authorize setup step. - Proxy variables are scrubbed from every CLI child environment and the printed-URL fetch bypasses ambient proxies. - Unit tests for the loopback and fixed-credential guards. - Changelog fragments, including the migration PRs #1171-#1177. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…serving proxy - Update test_sdkharness_resilience for the stricter contentSha1 rule and add a large-file positive case. - observing_proxy: forward only origin-form paths, validate Content-Length, stream non-JSON upstream bodies instead of buffering them. - Merge the two changelog fragments into one; rewrap the README paragraph. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
30a259d to
685dc49
Compare
|
@ffumero2003 this PR is ready for review. Every GitHub Actions job is green after the rebase onto master (the earlier red runs came from master's SSE-B2 integration tests before #1185). The only red check is |
|
Reviewed the diff locally (the observing proxy, the concurrency assertions, and the resilience/guard changes) — looks good, approving. The two code/snyk findings are the bare hashlib.sha1(payload) calls at .sdkharness/tests/resilience-upload.py:336 and :470, both false positives: SHA-1 only verifies B2’s required contentSha1/large_file_sha1 content checksum, not a security primitive. The credential constants are rejected test fixtures and the observing proxy is loopback-only (no SSRF/public bind). Tests pass 121/121, test/static and Ruff pass; no correctness blockers. Suggest adding usedforsecurity=False to those two calls (matching conformance-file-lifecycle.py:187) to make intent explicit and clear the informational Snyk check — code/snyk isn’t required, so not merge-blocking either way. |
469f200 to
9ea898a
Compare
What
Fixes review findings in the repository-owned
.sdkharness/checks.large.concurrent_partsandlarge.parallel_downloadnow observe real concurrency instead of only comparing sizes and bytes. The conformance run has no control/journal listener, so the check points the CLI at a small loopback proxy (.sdkharness/tests/lib/observing_proxy.py) in front of the simulator. The proxy counts requests in flight and holds each part upload / download stream for 0.5 s (a client-side throttle) so overlap is visible.concurrent_parts:--threads 1must peak at exactly 1b2_upload_partin flight,--threads 4at 2 or more; the uploaded bytes are now also downloaded and compared.parallel_download:--threads 1must use one stream and no Range request;--threads 4must make at least one ranged GET and have two or more download streams overlapping. b2sdk only splits a download per 100 MiB of content and the CLI cannot lower that, so the fixture is just over 200 MiB.--threads 1in the check makes both fail with a concurrency message.resilience-upload.py: followsLATEST_STABLE_VERSION(no hard-codedb2v5) and assertsb2is imported from this checkout;contentSha1: noneis accepted only for the large-file scenario (and thenfileInfo.large_file_sha1must match); the scenario field is filled in the result line when the configuration is refused (unknownif it is not a known scenario); the firstaccount authorizesetup step is retried up to 3 times with a NOTE (the earlier transientauthenticate: b2 exited 1on a cold first run could not be reproduced; authorize is idempotent and not under test, so a persistent failure still fails).HTTP(S)_PROXY,ALL_PROXY,NO_PROXY, either case) are scrubbed from the CLI child environment in all four executables viasimulator_guard.scrubbed_environment(loopback is exempted explicitly), and the printed-URL fetch inurls.native_downloadbypasses ambient proxies.test/unit/test_sdkharness_guards.py, for the loopback URL guard, the fixed-credential guard (no echo of the refused key), scrubbing, the resilience helpers and the observing proxy.+sdkharness-observe-concurrency.infrastructure.mdfor this change, and+sdkharness-migration.infrastructure.mdrecording the migration PRs test: add repository-owned bucket conformance contracts #1171-Complete repository-owned CLI resilience coverage #1177, which were merged without individual fragments (CONTRIBUTING requires one per PR).Not changed on purpose:
upload.retry_408andapi.retry_after_503are real known CLI findings and still fail.Testing
Against a loopback simulator only, with
HTTP_PROXY/HTTPS_PROXYset to a dead port in the environment:upload.stallSKIP (no-client-option),upload.retry_408andapi.retry_after_503FAIL (the known findings), as before.pytest test/unit/test_sdkharness_guards.py: 94 passed; the tests fail when the guards are removed.test/staticandruff checkpass.🤖 Generated with Claude Code