fix(client): raise on an already-failed Storage job instead of returning it - #603
Conversation
_wait_for_storage_job is reached by 20 call sites across client/ (storage
tables 15x, dev branches 2x, workspaces 1x) but had no tests of its own:
its success fast path was only covered incidentally (test_storage_truncate),
and the terminal-error fast path and the timeout path were not covered at
all. New TestWaitForStorageJob pins all four contracts.
test_already_failed_body_raises is xfail(strict=True): the poller returns an
already-terminal ERROR initial body as-is instead of raising, so a Storage
API fast fail reaches the caller as a normal return value -- and since every
call site either returns the job or job.get("results", {}), that surfaces as
an empty success. The fix lands in the follow-up commit, which removes the
marker; strict=True is what makes that removal mandatory (a non-strict xfail
would XPASS silently and never fail CI again in either direction).
xfail_strict is not set in pyproject.toml, so strict lives on the marker --
deliberately not a repo-wide default in a bugfix branch.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_wait_for_storage_job had two terminal-state checks -- an early return
before the poll loop and a second check inside it -- and they had drifted:
the in-loop check raised STORAGE_JOB_FAILED on status=error, the early
return handed the job back to the caller. A Storage API fast fail (terminal
straight away, never "waiting") therefore reached the caller as a normal
return value, and since all 20 call sites either return the job or
job.get("results", {}), that surfaced as a silent empty success.
Not just hygiene: storage_service.py's `create-table --if-not-exists`
idempotency keys off catching KeboolaApiError/STORAGE_JOB_FAILED, so on a
fast fail it got {} and never ran -- the flag silently did nothing.
Restructured to check-then-fetch, so the caller's initial body and every
polled body traverse identical code and the class of bug is no longer
expressible. This is the shape the sibling pollers wait_for_queue_job
(client/queue.py) and wait_for_query_job (client/query.py) already use.
Preserved verbatim: sleep-before-poll ordering, the deadline check before
the sleep, both messages, status codes and retryable flags.
Removes the xfail(strict=True) marker added in the previous commit --
mandatory, since a strict xfail that starts passing fails the suite.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `job:` arg docstring said "from POST/DELETE", which is wrong for change_sharing_type (storage_tables.py:310) -- it enqueues with PUT and then awaits. Pre-existing inaccuracy, carried over in the previous commit and caught in review. Verb breakdown across the poller's 19 call sites in client/: POST 11, DELETE 7, PUT 1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The class docstring said 20 call sites across client/. That total holds only on the #556 branch, which adds merge_requests.merge(); on main it is 19 -- storage_tables.py 16x, branches.py 2x, workspaces.py 1x. The earlier "15x" for storage_tables came from a grep that required the HTTP verb on the same line as _request(, which missed change_sharing_type's multi-line call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…docs Opens 0.84.3: v0.84.2 is tagged and published at main's HEAD, so there is no in-progress key to append to, and the repo's convention is that the substantive PR carries the bump (0.84.2 <- #594/#597, 0.84.1 <- #589, 0.84.0 <- auth login-password, ...). Neither `changelog-check` (audits that released versions have entries) nor `version-check` (plugin.json / marketplace.json / uv.lock vs pyproject) would have caught the omission -- the silent drift convention #17 warns about. The behaviour change is user-visible, so it also lands in gotchas.md tagged (since v0.84.3). Tests: the PR claimed poll counts are unchanged for every budget but nothing pinned it. test_timeout_raises_storage_job_timeout now records sleeps and asserts none happened -- verified that moving the deadline check after the sleep makes it fail (assert [1.0] == []) where before it merely ran a second slower, since the break still precedes the fetch. Adds test_budget_below_one_interval_still_polls_once for the other half of the claim (0.5s budget -> exactly one poll, overshoot preserved), and test_polled_success_returns_the_polled_body: the happy path was covered only incidentally, by a fixture that returns a terminal body and never enters the loop. Docstring: the "same shape as the sibling pollers" line read as a parity claim. Narrowed -- the check-then-fetch shape matches, the behaviour does not: this poller knows only success/error (so any other terminal status would exhaust the budget and surface as STORAGE_JOB_TIMEOUT, where the queue poller keys off isFinished), and its sleep is not capped to the remaining budget. Both predate this branch. _mk_client is now one module-level helper instead of two byte-identical methods 62 lines apart (the only two in the suite). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Thanks — I verified all six independently rather than taking them on trust, and all six hold. Addressed in Including the one I had argued against. On point 2 I claimed a regression would go red via
What changed:
Two notes on scope. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 7 out of 8 changed files in this pull request and generated no new comments.
Suppressed comments (1)
tests/test_client.py:3365
- This test currently relies on the implementation not making any HTTP requests; if it regresses and starts polling, the unmatched
httpx_mockrequest will raisehttpx.TimeoutExceptionandBaseHttpClient._do_request()will retry with realtime.sleepbackoff, making the failure slow and potentially confusing. Consider patchingtime.sleepto_noop_sleepin this test too so a regression fails fast.
def test_already_successful_body_returns_without_polling(self, httpx_mock) -> None:
"""A terminal-success initial body is returned as-is, with no HTTP call."""
with _mk_client() as client:
job = client._wait_for_storage_job({"id": 1, "status": "success", "results": {"x": 1}})
| error_code=ErrorCode.STORAGE_JOB_FAILED, | ||
| retryable=False, | ||
| ) |
There was a problem hiding this comment.
🔍 Error extraction still assumes error is a dict
job.get("error", {}).get("message", ...) raises AttributeError if the Storage API returns error as a plain string (the Query-Service poller in this package explicitly handles both shapes — see _extract_query_error tests in tests/test_client.py). Pre-existing, but the fix now makes this path reachable from the initial response body too, so a fast-fail whose error is a string would surface as an unhandled AttributeError rather than STORAGE_JOB_FAILED. Worth a one-line tolerance check while the code is being touched.
(Refers to lines 237-243)
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Confirmed and fixed in 992f995 — thanks, this one was real, not theoretical.
Reproduced the exact failure first:
>>> {'error': 'plain text boom'}.get('error', {}).get('message', 'Storage job failed')
AttributeError: 'str' object has no attribute 'get'
And your framing of the reachability is right: pre-existing, but the restructure means the terminal check now also sees the caller's initial response body, so the shape can arrive from one more direction than before.
Worth adding to the case you made — the field is demonstrably not reliably a dict in this repo's own history, not just in principle. Three precedents:
queue.py:250isinstance-guards its ownresultbefore.get("message")._extract_query_job_error(_transfer.py:152) is a documented helper for "strings, dicts and unknown shapes".- The Metastore once answered
{"error": 422}— an int — and the CLI renderedAPI error 422: 422. That cost a released bugfix, which is whyBaseHttpClient._raise_api_errornow acceptserroronly when it is a non-empty string.
_core.py was the last place reading it bare.
One deliberate deviation from the nearest precedent: the queue poller falls back to generic text when result is not a dict, which discards a string error's content. I followed _extract_query_job_error instead and use a string error as the message — it is the more useful of the two, and losing the operator's only diagnostic text to a type check would be its own small bug. Extracted as _storage_job_error_message with the reasoning in its docstring so nobody "simplifies" it back.
A dict with no usable message, an int, a list, or a missing field all fall back to "Storage job failed" rather than rendering None or a raw repr. Two tests cover it (the string shape, plus five unusable variants), and the 0.84.3 changelog bullet now mentions the hardening.
make check: 5755 passed, 12 skipped.
Devin review: `job.get("error", {}).get("message", ...)` raises AttributeError
when `error` is a plain string -- a traceback instead of a clean
STORAGE_JOB_FAILED exit. Confirmed: 'str' object has no attribute 'get'.
Pre-existing, but the restructure made the expression reachable from the
caller's initial response body too, so the shape can now arrive from one more
direction.
The field is demonstrably not reliably a dict in this codebase's experience:
the Metastore answered `{"error": 422}` (fixed in 0.62.x, which is why
BaseHttpClient._raise_api_error accepts `error` only when it is a non-empty
string), queue.py isinstance-guards its own `result`, and
_extract_query_job_error handles strings, dicts and unknown shapes. _core.py
was the last place reading it bare.
Extracted _storage_job_error_message: a string `error` is now used AS the
message rather than discarded (the queue poller's guard falls back to generic
text, _extract_query_job_error keeps the text -- followed the latter, it is
the more useful of the two precedents). A dict with no usable message, an int,
a list or a missing field all fall back to "Storage job failed" instead of
rendering None or a raw repr.
Tests: both shapes plus five unusable variants. Changelog bullet extended.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
padak
left a comment
There was a problem hiding this comment.
Review — verified locally, no blocking findings
Checked out 992f995 in a clean worktree and reproduced every claim in the description rather than reading them. Summary: the bug is real, the fix is behaviour-preserving apart from the one intended change, and the new tests genuinely catch it.
What I verified (not just read)
1. The new tests are red against the unfixed poller. Restored _core.py from origin/main on top of the PR's test file:
FAILED tests/test_client.py::TestWaitForStorageJob::test_already_failed_body_raises
FAILED tests/test_client.py::TestWaitForStorageJob::test_string_error_field_still_raises_with_its_text
FAILED tests/test_client.py::TestWaitForStorageJob::test_unusable_error_field_falls_back_to_generic_message
3 failed, 5 passed
So the fast-fail contract is pinned by a test that actually fails without the fix — the xfail(strict=True) → removal dance in commits 1/2 did its job.
2. "Poll counts are identical" holds exhaustively, not anecdotally. Extracted both loops (old + new) against a deterministic fake clock and swept max_wait ∈ {0, 0.001, 0.5, 1.0, 1.5, 2.0, 3.0, 60, 600} × six status sequences (terminal-success-first, terminal-error-first, waiting-forever, waiting→success, waiting→error, unknown-terminal-status):
| scenario | old (outcome, polls, sleeps) | new |
|---|---|---|
| terminal success first | return, 0, 0 |
return, 0, 0 |
| terminal error first | return, 0, 0 |
raise_failed, 0, 0 ← the intended change |
waiting forever, max_wait=0 |
timeout, 0, 0 |
timeout, 0, 0 |
waiting forever, max_wait=0.5 |
timeout, 1, 1 |
timeout, 1, 1 |
waiting forever, max_wait=60 |
timeout, 60, 60 |
timeout, 60, 60 |
waiting→error, max_wait=0.5 |
raise_failed, 1, 1 |
raise_failed, 1, 1 |
unknown terminal status, max_wait=60 |
timeout, 60, 60 |
timeout, 60, 60 |
9 differences across the whole sweep, all of them the status: error initial body. Nothing else moves — including the sub-interval overshoot and the max_wait=0 no-sleep case, which the two new timing tests pin.
3. make check green on this branch, macOS / py3.12: 5755 passed, 12 skipped, 154 deselected (the description's 5751 predates 992f995). CI is green on all four jobs too.
4. The bug class is confined to this poller. Read the two siblings: wait_for_queue_job (client/queue.py) and wait_for_query_job (client/query.py) both fetch before checking and neither accepts a caller-supplied initial body, so neither can grow the second check that drifted here. Nothing else in client/ needs the same treatment.
5. All 19 call sites re-counted on main — storage_tables.py 16×, branches.py 2×, workspaces.py 1×. Every one of them is return job.get("results", {}), return job, or a bare wait, so the "empty success" mechanism in the description is exactly right: an error job has no results, so {} came back as a success payload.
The strongest part of this PR is underadvertised
992f995 (_storage_job_error_message) is missing from the "Review follow-ups" list in the description and only shows up in changelog.py. It deserves top billing, because unlike the fast-fail it fixes something reachable today, on the already-working polled path:
# before
error_msg = job.get("error", {}).get("message", "Storage job failed"){"error": null} → None.get(...) → AttributeError; {"error": "some text"} → str.get → AttributeError. Either shape turns a cleanly-failed Storage job into a traceback, no fast fail required. That is a smaller blast radius than the headline bug but a much better-evidenced one — the description currently hedges the whole PR on the fast-fail hypothesis ("treat any specific 'this command was broken' claim as unproven", which is the right call) while sitting on a defect that needs no hypothesis at all. Worth a line in What.
Findings
🟢 nit — tests/test_client.py, test_unusable_error_field_falls_back_to_generic_message. None doubles as the "omit the key" sentinel:
for error_field in ({}, {"message": ""}, 422, None, ["boom"]):
job: dict[str, Any] = {"id": 1, "status": "error"}
if error_field is not None:
job["error"] = error_fieldso an explicit {"error": None} — one of exactly two shapes that made the old code raise AttributeError, and the one this loop looks like it is covering — is never constructed. The other ("error": "text") has its own test; this one has none. A distinct sentinel (_OMIT = object()) buys the missing case for one line.
🟢 nit — gotchas.md:3636. The new ## A Storage job that failed instantly… heading has no blank line before it; it sits flush against the previous section's last list item. 125 of the 127 ## headings in the file have one (the other exception is pre-existing at :1655). CommonMark still renders it as a heading, so this is consistency only.
🟢 nit — description drift. make check line says 5751; the follow-ups list stops at af9a790. Both fixed by one edit if you touch the description for the point above.
Optional, explicitly out of scope for a behaviour-preserving bugfix — follow-up candidates
- The failure message carries no job id.
wait_for_queue_jobraisesf"Queue job {job_id} failed: {msg}"; this one raises the bare API message, so a failed Storage job gives the user nothing to look up. The timeout branch right below it does includejob_id. Prefixing would not disturbstorage_service.py's"already has the same display name" in exc.messagesubstring match, but it is a message change and this PR deliberately preserved messages verbatim — better as its own change. - Unknown terminal statuses still burn the full budget. Documented honestly in both the docstring and the gotcha: anything that is not
success/errorpolls untilSTORAGE_JOB_TIMEOUTwithretryable: true, i.e. a permanent failure advertised as retryable. Pre-existing, correctly left alone here.
Verdict
Approve-quality from my side. The restructure makes the drift unexpressible rather than merely fixed, the docstring explains why for the next reader, the tests are the kind that fail for the right reason, and the version/plugin/changelog surfaces (pyproject → plugin.json → marketplace.json → uv.lock → changelog.py → gotchas.md tagged (since v0.84.3)) are all in step — 09876bf catching the missing 0.84.3 key is the sort of thing neither changelog-check nor version-check would have flagged, since v0.84.2 was already tagged at main's HEAD.
Merge-order note from the description confirmed as sensible: land this before #556 so its local guard in merge_requests.merge() drops out on rebase.
Automated review — three cosmetic nits, nothing blocking. Not an approval; a human still holds the merge gate.
|
To set expectations on the review above: all three findings are cosmetic (🟢), nothing blocks. The substance — the fix itself, the equivalence claim and the tests — I verified by reproduction, not by reading, and it all holds. I'll keep watching this branch. Push whenever you're ready: I'll re-review the delta, and once there's nothing left worth raising, I'll approve rather than leave another round of comments. No need to address the nits if you'd rather not — say so and I'll approve as it stands. |
…eading Review nits on #603, nothing behavioural. `test_unusable_error_field_falls_back_to_generic_message` used `None` as the "leave the key out" sentinel, which collapsed two different bodies into one and silently dropped the more interesting of them: `{"error": None}` is one of exactly two shapes that made the pre-fix extraction raise AttributeError (`{"error": "text"}` is the other, and it has its own test). Absence now has a `_OMIT` sentinel of its own, so the null case is a real case. Also gives the new gotcha section the blank line before its `##` heading that 125 of the file's other 127 headings have.
padak
left a comment
There was a problem hiding this comment.
No outstanding review findings — and a disclosure: I pushed the nits myself
Rather than leave you a second round of comments over three cosmetic things, I fixed two of them on your branch: 5e1d6bd. Revert it without discussion if you'd rather own them.
What's in it (no behaviour, no production code):
test_unusable_error_field_falls_back_to_generic_messageusedNoneas its "leave the key out" sentinel, so the loop that looks like it covers{"error": None}never actually built that body. It matters more than the others:{"error": None}is one of exactly two shapes that made the pre-992f995extraction raiseAttributeError— verified in isolation,job.get("error", {}).get("message", …)gives'NoneType' object has no attribute 'get'— and the other one ({"error": "text"}) already has its own test. Absence now has an_OMITsentinel, so null is a real case.- The new gotcha section got the blank line before its
##heading that 125 of the file's other 127 headings have. (The remaining exception atgotchas.md:1655is pre-existing; left alone.)
make check green on the pushed head: 5755 passed, 12 skipped.
Left for you — I tried to correct it and my tooling refused to rewrite another author's PR description, which is fair enough: the Testing section still says 5751 passed (5755 as of 5e1d6bd), and the follow-ups list still stops at af9a790, so 992f995 appears nowhere in the description. That last one is worth a line in What, not just the changelog — it is the one defect in this PR that needs no hypothesis at all. The fast-fail bug is correctly hedged as unproven, while {"error": null} / {"error": "text"} turn a cleanly-failed Storage job into a traceback on the polled path that has always worked. Smaller blast radius, much better evidence.
What I verified by reproduction, not by reading
- The tests are red against the unfixed poller.
_core.pyrestored fromorigin/mainunder your test file: 3 failed, 5 passed —test_already_failed_body_raisesamong them. Thexfail(strict=True)→ removal sequencing did its job. - "Poll counts are identical" holds exhaustively. Both loops against a deterministic fake clock,
max_wait ∈ {0, 0.001, 0.5, 1, 1.5, 2, 3, 60, 600}× 6 status sequences: 9 differences in the whole sweep, every one of them thestatus: errorinitial body. Sub-interval overshoot and themax_wait=0no-sleep case included. make checkgreen locally (macOS/py3.12) at both992f995and5e1d6bd; CI green on all four jobs.- The bug class is confined to this poller.
wait_for_queue_jobandwait_for_query_jobboth fetch before checking and neither takes a caller-supplied initial body, so neither can grow the second check that drifted here. - 19 call sites re-counted on
main(16/2/1), allreturn job/return job.get("results", {})/ bare wait — the "empty success" mechanism is exactly as described.
Nothing left on my side worth raising against 1–5, which cover your work. 5e1d6bd is mine and I am explicitly not reviewing my own commit — give it a glance before merging. The formal approval comes from a human hand, not from this pass.
Merge order per your note: land this before #556 so its local guard in merge_requests.merge() drops out on rebase.
|
@padak Ready for the approval pass — everything from both rounds is resolved:
Nothing else queued from my side. After the approval this squash-merges (subject from the PR title; body drafted), then #556 rebases and drops its local |
|
Your correction is right and mine was wrong — checked it rather than take it on trust:
Four shapes, not two — the rule is "anything that is neither a dict nor absent", and I generalised from the two the docstring happened to name instead of enumerating. Worth correcting in the record since I put a number in a review; it doesn't move the conclusion, it strengthens it — three of the four ride the fallback loop and the string has its own test, so Description reads right now: Nothing outstanding from me. The approval comes from @padak's own hand — my tooling declines to cast it, which is the correct call for a repo whose merge gate is one approving review. |
padak
left a comment
There was a problem hiding this comment.
Approved — with a disclosure: I pushed the nits myself
Rather than leave you a second round of comments over three cosmetic things, I fixed two of them on your branch: 5e1d6bd. Revert it without discussion if you'd rather own them.
What's in it (no behaviour, no production code):
test_unusable_error_field_falls_back_to_generic_messageusedNoneas its "leave the key out" sentinel, so the loop that looks like it covers{"error": None}never actually built that body. It matters more than the others:{"error": None}raisedAttributeErroron the pre-992f995extraction — verified in isolation,job.get("error", {}).get("message", …)gives'NoneType' object has no attribute 'get'. Absence now has an_OMITsentinel, so null is a real case. (Per your correction, which I re-checked and agree with: the old expression raised on four shapes —None,"text",422,["boom"], i.e. anything neither dict nor absent — so the fallback loop now exercises three of the four and the string has its own test.)- The new gotcha section got the blank line before its
##heading that 125 of the file's other 127 headings have. (The remaining exception atgotchas.md:1655is pre-existing; left alone.)
make check green on the pushed head: 5755 passed, 12 skipped.
Description: resolved on your side — Testing reads 5755 … as of 5e1d6bd, 992f995 has top billing in How, and both follow-ups are listed with the disclosure on mine. That was the third nit and it is closed.
What I verified by reproduction, not by reading
- The tests are red against the unfixed poller.
_core.pyrestored fromorigin/mainunder your test file: 3 failed, 5 passed —test_already_failed_body_raisesamong them. Thexfail(strict=True)→ removal sequencing did its job. - "Poll counts are identical" holds exhaustively. Both loops against a deterministic fake clock,
max_wait ∈ {0, 0.001, 0.5, 1, 1.5, 2, 3, 60, 600}× 6 status sequences: 9 differences in the whole sweep, every one of them thestatus: errorinitial body. Sub-interval overshoot and themax_wait=0no-sleep case included. make checkgreen locally (macOS/py3.12) at both992f995and5e1d6bd; CI green on all four jobs.- The bug class is confined to this poller.
wait_for_queue_jobandwait_for_query_jobboth fetch before checking and neither takes a caller-supplied initial body, so neither can grow the second check that drifted here. - 19 call sites re-counted on
main(16/2/1), allreturn job/return job.get("results", {})/ bare wait — the "empty success" mechanism is exactly as described.
Approving on the strength of 1–5, which cover your work. 5e1d6bd is mine and I am explicitly not reviewing my own commit — give it a glance before merging.
Merge order per your note: land this before #556 so its local guard in merge_requests.merge() drops out on rebase.
#603 Rebased onto main with #603, where _wait_for_storage_job itself raises on an already-terminal error body (converged on the wait_for_queue_job shape). Per the rebase checklist on #556: the local guard, its now-unused errors import, the docstring clause, and its dedicated test are removed; test_merge_failed_job_raises stays (polled-error path). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ract on the Protocol Follow-ups after #603 superseded merge()'s local terminal-error guard: - StorageRequester.wait_for_storage_job now DOCUMENTS the raise-on-failure contract (initial-body or polled) as a requirement on future transports -- merge() no longer re-checks the returned job, so a Protocol implementation that returns a failed job would silently reintroduce the pre-#603 blind spot. merge()'s docstring points there; the seam-test stub gets a do-not-copy note for the same reason. - New test: merge() passes max_wait=MERGE_JOB_MAX_WAIT to the poller (asserted through a stub requester -- httpx mocks never see the kwarg). Previously the 600 s budget could be dropped without any test going red, silently reverting merge to the 60 s default and a mid-merge STORAGE_JOB_TIMEOUT with retryable=True. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Implements docs/merge-requests-layer3-rfc.md (decisions D1-D10 there; this
message covers what shapes the code):
client/merge_requests.py -- the nine MR endpoints as a namespace,
client.merge_requests.{list,get,conflicts,create,update,request_review,
approve,request_changes,merge}. The namespace depends on a StorageRequester
Protocol, not on the client; a temporary _ClientRequester adapter satisfies
it until the client-split RFC (draft #595) builds a real transport under the
seam (D10). Two invariants deliberately break the surrounding idioms and are
called out in docstrings: paths are NEVER branch-prefixed (every MR endpoint
is project-level), and bodies are JSON with real types (the backend asserts
branchFromId as int; form-encoded values stay strings and fail validation).
_optional_mr_fields keeps create/update from drifting and is keyword-only:
four of its five parameters are str | None, so a positional transposition
would type-check cleanly and surface only as a backend 422.
merge() awaits the Storage job implicitly like every job-backed method in
client/, with a dedicated MERGE_JOB_MAX_WAIT (600 s) budget -- merging a
many-config branch can outlive the default 60 s. It does NOT re-check the
returned job: raising on a failed job (fast-fail included) is the poller's
contract since #603, stated as a requirement on the Protocol so a future
transport cannot reintroduce the blind spot. The await covers the merge
outcome only; the source-branch deletion runs as a second, unhandled job.
client/configs.py -- get_config_diff + rebase_config/rebase_config_delete.
branch_id is required with no production fallback (the endpoints 400 on the
default branch, D5). Keep and delete rebases are separate methods so no
illegal combination is expressible (D6). The keep rebase requires the FULL
replaced body (name, rows, configuration, is_disabled, description): /rebase
replaces rather than patches, so an omitted key takes the server-side
default -- a caller sending only name+rows would wipe the configuration and
re-enable a disabled config, then merge that into production.
constants.py -- MERGE_JOB_MAX_WAIT and FEATURE_BRANCHES_MERGE_REQUESTS.
Layer 3 does no feature check itself (a missing feature is a 403 identical
to a role denial); Part 2's service pre-flights with the constant (D9).
tests/test_merge_request_client.py pins the wire contract: bare vs
branch-prefixed paths, JSON types, presence detection, the diff envelope and
the {} delete resolution, merge-job waiting and its 600 s budget, the
replaced-body requirement, include=activityLog, and the stub-requester seam.
Part 2 (service + commands) follows separately; no CLI command is added
here, so no E2E / docs surfaces change yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
client/merge_requests.py -- the nine MR endpoints as a namespace,
client.merge_requests.{list,get,conflicts,create,update,request_review,
approve,request_changes,merge}. The namespace depends on a StorageRequester
Protocol, not on the client; a temporary _ClientRequester adapter satisfies
it until the client-split work (draft #595) builds a real transport under
the seam. Two invariants deliberately break the surrounding idioms and are
called out in docstrings: paths are NEVER branch-prefixed (every MR endpoint
is project-level), and bodies are JSON with real types (the backend asserts
branchFromId as int; form-encoded values stay strings and fail validation).
_optional_mr_fields keeps create/update from drifting and is keyword-only:
four of its five parameters are str | None, so a positional transposition
would type-check cleanly and surface only as a backend 422.
merge() awaits the Storage job implicitly like every job-backed method in
client/, with a dedicated MERGE_JOB_MAX_WAIT (600 s) budget -- merging a
many-config branch can outlive the default 60 s. It does NOT re-check the
returned job: raising on a failed job (fast-fail included) is the poller's
contract since #603, stated as a requirement on the Protocol so a future
transport cannot reintroduce the blind spot. The await covers the merge
outcome only; the source-branch deletion runs as a second, unhandled job.
client/configs.py -- get_config_diff + rebase_config/rebase_config_delete.
branch_id is required with no production fallback (the endpoints 400 on the
default branch). Keep and delete rebases are separate methods so no illegal
combination is expressible. The keep rebase requires the FULL replaced body
(name, rows, configuration, is_disabled, description): /rebase replaces
rather than patches, so an omitted key takes the server-side default -- a
caller sending only name+rows would wipe the configuration and re-enable a
disabled config, then merge that into production.
constants.py -- MERGE_JOB_MAX_WAIT and FEATURE_BRANCHES_MERGE_REQUESTS.
Layer 3 does no feature check itself (a missing feature is a 403 identical
to a role denial); Part 2's service pre-flights with the constant.
tests/test_merge_request_client.py pins the wire contract: bare vs
branch-prefixed paths, JSON types, presence detection, the diff envelope and
the {} delete resolution, merge-job waiting and its 600 s budget, the
replaced-body requirement, include=activityLog, and the stub-requester seam.
Part 2 (service + commands) follows separately; no CLI command is added
here, so no E2E / docs surfaces change yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
client/merge_requests.py -- the nine MR endpoints as a namespace,
client.merge_requests.{list,get,conflicts,create,update,request_review,
approve,request_changes,merge}. The namespace depends on a StorageRequester
Protocol, not on the client; a temporary _ClientRequester adapter satisfies
it until the client-split work (draft #595) builds a real transport under
the seam. Two invariants deliberately break the surrounding idioms and are
called out in docstrings: paths are NEVER branch-prefixed (every MR endpoint
is project-level), and bodies are JSON with real types (the backend asserts
branchFromId as int; form-encoded values stay strings and fail validation).
_optional_mr_fields keeps create/update from drifting and is keyword-only:
four of its five parameters are str | None, so a positional transposition
would type-check cleanly and surface only as a backend 422.
merge() awaits the Storage job implicitly like every job-backed method in
client/, with a dedicated MERGE_JOB_MAX_WAIT (600 s) budget -- merging a
many-config branch can outlive the default 60 s. It does NOT re-check the
returned job: raising on a failed job (fast-fail included) is the poller's
contract since #603, stated as a requirement on the Protocol so a future
transport cannot reintroduce the blind spot. The await covers the merge
outcome only; the source-branch deletion runs as a second, unhandled job.
client/configs.py -- get_config_diff + rebase_config/rebase_config_delete.
branch_id is required with no production fallback (the endpoints 400 on the
default branch). Keep and delete rebases are separate methods so no illegal
combination is expressible. The keep rebase requires the FULL replaced body
(name, rows, configuration, is_disabled, description): /rebase replaces
rather than patches, so an omitted key takes the server-side default -- a
caller sending only name+rows would wipe the configuration and re-enable a
disabled config, then merge that into production. rebase_config is
keyword-only after the ids: name/description/change_description are
same-typed neighbours, so a positional transposition would type-check
cleanly and silently land review text inside the replaced body.
constants.py -- MERGE_JOB_MAX_WAIT and FEATURE_BRANCHES_MERGE_REQUESTS.
Layer 3 does no feature check itself (a missing feature is a 403 identical
to a role denial); Part 2's service pre-flights with the constant.
tests/test_merge_request_client.py pins the wire contract: bare vs
branch-prefixed paths, JSON types, presence detection, the diff envelope and
the {} delete resolution, merge-job waiting and its 600 s budget, the
replaced-body requirement (incl. the keyword-only signature),
include=activityLog, and the stub-requester seam.
Part 2 (service + commands) follows separately; no CLI command is added
here, so no E2E / docs surfaces change yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
client/merge_requests.py -- the nine MR endpoints as a namespace,
client.merge_requests.{list,get,conflicts,create,update,request_review,
approve,request_changes,merge}. The namespace depends on a StorageRequester
Protocol, not on the client; a temporary _ClientRequester adapter satisfies
it until the client-split work (draft #595) builds a real transport under
the seam. Two invariants deliberately break the surrounding idioms and are
called out in docstrings: paths are NEVER branch-prefixed (every MR endpoint
is project-level), and bodies are JSON with real types (the backend asserts
branchFromId as int; form-encoded values stay strings and fail validation).
_optional_mr_fields keeps create/update from drifting and is keyword-only:
four of its five parameters are str | None, so a positional transposition
would type-check cleanly and surface only as a backend 422.
merge() awaits the Storage job implicitly like every job-backed method in
client/, with a dedicated MERGE_JOB_MAX_WAIT (600 s) budget -- merging a
many-config branch can outlive the default 60 s. It does NOT re-check the
returned job: raising on a failed job (fast-fail included) is the poller's
contract since #603, stated as a requirement on the Protocol so a future
transport cannot reintroduce the blind spot. The await covers the merge
outcome only; the source-branch deletion runs as a second, unhandled job.
client/configs.py -- get_config_diff + rebase_config/rebase_config_delete.
branch_id is required with no production fallback (the endpoints 400 on the
default branch). Keep and delete rebases are separate methods so no illegal
combination is expressible. The keep rebase requires the FULL replaced body
(name, rows, configuration, is_disabled, description): /rebase replaces
rather than patches, so an omitted key takes the server-side default -- a
caller sending only name+rows would wipe the configuration and re-enable a
disabled config, then merge that into production.
constants.py -- MERGE_JOB_MAX_WAIT and FEATURE_BRANCHES_MERGE_REQUESTS.
Layer 3 does no feature check itself (a missing feature is a 403 identical
to a role denial); Part 2's service pre-flights with the constant.
tests/test_merge_request_client.py pins the wire contract: bare vs
branch-prefixed paths, JSON types, presence detection, the diff envelope and
the {} delete resolution, merge-job waiting and its 600 s budget, the
replaced-body requirement, include=activityLog, and the stub-requester seam.
Part 2 (service + commands) follows separately; no CLI command is added
here, so no E2E / docs surfaces change yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
client/merge_requests.py -- the nine MR endpoints as a namespace,
client.merge_requests.{list,get,conflicts,create,update,request_review,
approve,request_changes,merge}. The namespace depends on a StorageRequester
Protocol, not on the client; a temporary _ClientRequester adapter satisfies
it until the client-split work (draft #595) builds a real transport under
the seam. Two invariants deliberately break the surrounding idioms and are
called out in docstrings: paths are NEVER branch-prefixed (every MR endpoint
is project-level), and bodies are JSON with real types (the backend asserts
branchFromId as int; form-encoded values stay strings and fail validation).
_optional_mr_fields keeps create/update from drifting and is keyword-only:
four of its five parameters are str | None, so a positional transposition
would type-check cleanly and surface only as a backend 422.
merge() awaits the Storage job implicitly like every job-backed method in
client/, with a dedicated MERGE_JOB_MAX_WAIT (600 s) budget -- merging a
many-config branch can outlive the default 60 s. It does NOT re-check the
returned job: raising on a failed job (fast-fail included) is the poller's
contract since #603, stated as a requirement on the Protocol so a future
transport cannot reintroduce the blind spot. The await covers the merge
outcome only; the source-branch deletion runs as a second, unhandled job.
client/configs.py -- get_config_diff + rebase_config/rebase_config_delete.
branch_id is required with no production fallback (the endpoints 400 on the
default branch). Keep and delete rebases are separate methods so no illegal
combination is expressible. The keep rebase requires the FULL replaced body
(name, rows, configuration, is_disabled, description): /rebase replaces
rather than patches, so an omitted key takes the server-side default -- a
caller sending only name+rows would wipe the configuration and re-enable a
disabled config, then merge that into production.
constants.py -- MERGE_JOB_MAX_WAIT and FEATURE_BRANCHES_MERGE_REQUESTS.
Layer 3 does no feature check itself (a missing feature is a 403 identical
to a role denial); Part 2's service pre-flights with the constant.
tests/test_merge_request_client.py pins the wire contract: bare vs
branch-prefixed paths, JSON types, presence detection, the diff envelope and
the {} delete resolution, merge-job waiting and its 600 s budget, the
replaced-body requirement, include=activityLog, and the stub-requester seam.
Part 2 (service + commands) follows separately; no CLI command is added
here, so no E2E / docs surfaces change yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
client/merge_requests.py -- the nine MR endpoints as a namespace,
client.merge_requests.{list,get,conflicts,create,update,request_review,
approve,request_changes,merge}. The namespace depends on a StorageRequester
Protocol, not on the client; a temporary _ClientRequester adapter satisfies
it until the client-split work (draft #595) builds a real transport under
the seam. Two invariants deliberately break the surrounding idioms and are
called out in docstrings: paths are NEVER branch-prefixed (every MR endpoint
is project-level), and bodies are JSON with real types (the backend asserts
branchFromId as int; form-encoded values stay strings and fail validation).
_optional_mr_fields keeps create/update from drifting and is keyword-only:
four of its five parameters are str | None, so a positional transposition
would type-check cleanly and surface only as a backend 422.
merge() awaits the Storage job implicitly like every job-backed method in
client/, with a dedicated MERGE_JOB_MAX_WAIT (600 s) budget -- merging a
many-config branch can outlive the default 60 s. It does NOT re-check the
returned job: raising on a failed job (fast-fail included) is the poller's
contract since #603, stated as a requirement on the Protocol so a future
transport cannot reintroduce the blind spot. The await covers the merge
outcome only; the source-branch deletion runs as a second, unhandled job.
client/configs.py -- get_config_diff + rebase_config/rebase_config_delete.
branch_id is required with no production fallback (the endpoints 400 on the
default branch). Keep and delete rebases are separate methods so no illegal
combination is expressible. The keep rebase requires the FULL replaced body
(name, rows, configuration, is_disabled, description): /rebase replaces
rather than patches, so an omitted key takes the server-side default -- a
caller sending only name+rows would wipe the configuration and re-enable a
disabled config, then merge that into production.
constants.py -- MERGE_JOB_MAX_WAIT and FEATURE_BRANCHES_MERGE_REQUESTS.
Layer 3 does no feature check itself (a missing feature is a 403 identical
to a role denial); Part 2's service pre-flights with the constant.
tests/test_merge_request_client.py pins the wire contract: bare vs
branch-prefixed paths, JSON types, presence detection, the diff envelope and
the {} delete resolution, merge-job waiting and its 600 s budget, the
replaced-body requirement, include=activityLog, and the stub-requester seam.
Part 2 (service + commands) follows separately; no CLI command is added
here, so no E2E / docs surfaces change yet.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Issue: DMD-1898
What
_wait_for_storage_job(client/_core.py) had two terminal-state checks — an earlyreturn before the poll loop and a second check inside it — and they had drifted:
status == "success"status == "error"STORAGE_JOB_FAILEDSo a Storage API fast fail (terminal straight away, never
waiting) reached the calleras a normal return value. All 19 call sites of the helper either return the job or
job.get("results", {})— and an error job has noresults— so the failure surfaced as asilent empty success. Call sites:
storage_tables.py16x (import/export/delete/snapshot/…),branches.py2x (create/delete dev branch),workspaces.py1x. Enqueued with POST 11x,DELETE 7x and PUT 1x (
change_sharing_type).Blast radius. Every one of those operations could report exit 0 with nothing done. The
size of the real-world exposure depends on when the API fails fast, which is not something
this PR establishes — e.g.
storage create-table --if-not-existskeys its idempotency offcatching
STORAGE_JOB_FAILED, and that path is live-validated in the 0.84.2 changelog("third create → original
STORAGE_JOB_FAILEDenvelope"), so the duplicate-name errorevidently arrives polled and that flag works today. The bug class is real and the fix is
unconditional; treat any specific "this command was broken" claim as unproven unless the
error is known to arrive terminal on the first response.
How
Restructured to check-then-fetch: one terminal-state check at the top of the loop, so
the caller's initial body and every polled body traverse identical code and the class of bug
(two checks that can drift) is no longer expressible.
This is the shape the two sibling pollers in the same package already use —
wait_for_queue_job(client/queue.py) andwait_for_query_job(client/query.py). Thestorage poller was the odd one out precisely because it receives the first job dict from the
caller instead of fetching it, and grew a second check for that case.
Preserved verbatim, so no existing behaviour moves: sleep-before-poll ordering, the deadline
check before the sleep (an exhausted budget never costs a poll interval), both error
messages, both status codes, both
retryableflags. Poll counts are identical to the oldloop for every budget, including
max_wait=0andmax_wait< one poll interval.Also fixed, and unlike the fast-fail this one needs no hypothesis: the failure-message
extraction assumed
erroris a dict.{"error": null}or{"error": "some text"}turned acleanly-failed Storage job into an
AttributeErrortraceback instead ofSTORAGE_JOB_FAILED— on the polled path that has always worked, no fast fail required (
992f995,_storage_job_error_message; a string error's text is now used as the message, unusableshapes fall back to the generic text). Smaller blast radius than the headline bug, but
directly evidenced — the repo has already paid for this shape once, when the Metastore
answered
{"error": 422}.Commits
test(client)— newTestWaitForStorageJob. The poller is reached by 19 call sitesbut had no tests of its own: its success fast path was covered only incidentally (in
test_storage_truncate), and the terminal-error fast path and the timeout path were notcovered at all. Four contracts pinned;
test_already_failed_body_raiseslands asxfail(strict=True)— red against the unfixed code, which is what proves it catches thebug.
fix(client)— the restructure, plus removal of that marker.strict=Truemakes theremoval mandatory: a strict xfail that starts passing fails the suite, so the marker
cannot be forgotten. (A non-strict xfail would XPASS silently and then never fail CI
again in either direction — effectively a disabled test.)
xfail_strictis deliberately not set inpyproject.toml;strictlives on the marker,since making it a repo-wide default is a convention change that should not ride along in a
bugfix branch.
Testing
make checkgreen: 5755 passed, 12 skipped as of5e1d6bd(5750 + 1 xfailed after commit 1)."status": "error"intests/were checked — for the storage poller the error always arrives in the polled response
(
test_client.py, import job), one is the Queue poller (a different code path), one isjob-listing fixture data. Nothing encoded the old broken behaviour.
--if-not-existspath already goes throughKeboolaApiError, which the fix guarantees.Merge order
Please merge before #556. That PR carries a local guard in
merge_requests.merge()forthis exact blind spot; once this lands, the guard is dead code and #556 drops it on rebase
(checklist posted there).
🤖 Generated with Claude Code
Review follow-ups
e719d5a— thejob:arg docstring said "from POST/DELETE";change_sharing_type(
storage_tables.py:310) enqueues with PUT. Pre-existing inaccuracy, carried over in8ecf99dand caught by review. Now names all three verbs.09876bf— second review round: opens 0.84.3 (pyproject +make version-sync+changelog bullet +
gotchas.mdtagged(since v0.84.3)).v0.84.2is tagged andpublished at main's HEAD, so there was no in-progress key to append to, and neither
changelog-checknorversion-checkwould have caught the omission. Also pins the"poll counts unchanged" claim with a sleep recorder (verified it now fails where it
previously only ran a second slower), adds the sub-interval-budget and polled-success
cases, narrows the sibling-poller wording from a parity claim to a shape claim, and
dedups
_mk_client.992f995— Devin review: tolerate a non-dicterroron a failed Storage job (see thebolded paragraph in How — the best-evidenced defect in this PR).
5e1d6bd— pushed by @padak (disclosed in his second review):_OMITsentinel so theexplicit
{"error": null}body is actually constructed by the fallback test, plus themissing blank line before the new
gotchas.mdheading. Reviewed and verified on pull:both claims check out (
Nonewas silently collapsing "absent" and "null" into one case).af9a790— corrected the call-site count in the test class docstring: 19 onmain(16/2/1), not 20. The 20 holds only on the [DMD-1833] Merge requests — Part 1, Layer 3 (client) #556 branch, which adds
merge(). Theearlier "15x" for
storage_tables.pycame from a grep that required the HTTP verb onthe same line as
_request(, so it missedchange_sharing_type's multi-line call.The commit message of
8ecf99dstill says "all 20 call sites"; leaving it, sincerewriting pushed history would orphan the review thread on it.