fix(http): never replay a credential-minting call on an ambiguous failure (#599) - #601
Draft
padak wants to merge 1 commit into
Draft
fix(http): never replay a credential-minting call on an ambiguous failure (#599)#601padak wants to merge 1 commit into
padak wants to merge 1 commit into
Conversation
…lure (#599) POST /v2/storage/tokens is a create, not a read, yet the shared retry loop replayed it up to 3 times on 500/502/503/504 and on read timeouts. A server that mints the token and then fails to answer hands back a second live credential the caller never sees a value for -- and therefore can never revoke. _do_request now takes `idempotent` (default True, so every existing call site is byte-identical); the two token-create paths and the data-app managed-repo git-credentials-create pass False. Failures that provably never reached the handler stay retryable even then -- 429 (rejected by the rate limiter before the handler runs), connect error, connect timeout -- so no resilience is lost where replay is safe. token refresh is deliberately left retryable: a replayed rotation overwrites the value the lost response carried instead of leaving a second credential behind. Second half of the issue: a 5xx now names itself as upstream and carries the support id. `API error 500 ... Application error.` gave the caller no next step, so a persistent upstream outage read like a broken request. Server errors now append the status-page/support hint plus `(exceptionId: ...)` when the body carries one; 4xx messages are untouched. A non-idempotent 5xx also reports retryable: false in the --json envelope.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Two CLI-side hardening asks from #599, whose reporter correctly concluded the underlying
500oneurope-west3.gcpis upstream and not a kbagent bug.1. Non-idempotent calls are no longer replayed on an ambiguous failure.
POST /v2/storage/tokensmints a credential, but the shared retry loop treated it like any other request and replayed it up to 3 times on500/502/503/504— and on read timeouts. If the server ever mints the token and then fails to answer, the replay leaves a second live credential behind that the caller never sees a value for, and therefore can never revoke.BaseHttpClient._do_requestnow takesidempotent(defaultTrue, so every existing call site is byte-identical). Four call sites passFalse:create_scoped_tokenandcreate_short_lived_token(POST /v2/storage/tokens)create_git_credential(data-app managed repo — mints a one-timehttp_tokensecret)The distinction is not "is it a POST" but "could the request have reached the handler":
4295xxSo the change costs no resilience anywhere replay is provably safe.
token refreshis deliberately left retryable: a replayed rotation overwrites the value the lost response carried rather than leaving a second credential behind, so the caller still ends up holding the only token that authenticates. Documented in the method docstring so it does not read as an oversight.2. A 5xx now says it is upstream, and carries the support id.
API error 500 ... Application error.gave the caller no next step — which is exactly why the reporter spent an investigation across two projects and two tokens before concluding it was not theirs to fix. Server errors now append the status-page/support hint, plus(exceptionId: ...)whenever the response body carries one (the first thing Keboola support asks for; previously parsed and discarded).4xx messages are untouched — a rejected request is the caller's to fix and must not blame upstream. A non-idempotent 5xx additionally reports
retryable: falsein the--jsonerror envelope, so an automated caller does not replay a mint the server may already have honoured. Nothing in this repo branches onretryable; it is advice carried to the caller.Not in this PR
500itself — a Keboola Connection backend issue, as the reporter says.kbagent token list, asked for in a follow-up comment on the same issue. Separate scope (new command + all doc surfaces), tracked separately.How it was tested
make checkgreen: lint, format, typecheck, skill, version, command-sync, changelog, error-codes, sentinel-guards, file-size, 5758 tests passed.New tests:
tests/test_http_base.py::TestNonIdempotentRetryPolicy— 7 cases pinning each row of the table above, plus a case proving the default (idempotent) path still makes its 3 attempts.tests/test_http_base.py::TestUpstreamErrorGuidance— hint on 5xx,exceptionIdsurfaced when present, no hint on 4xx.tests/test_client_device_enrollment.py::TestCreateScopedTokenIsNotReplayed— the reported scenario at the client layer: a persistent 500 oncreate_scoped_tokencosts exactly one attempt.No live-API testing: reproducing this needs the upstream outage.
Note for the maintainer
The changelog entry sits under
0.84.2, the current top key, sincemainis exactlyv0.84.2and there is no unreleased key. Renumber at release time. No command was added, renamed or removed, so no command doc surface needed syncing;docs/sdk.mdgained one line becausecreate_scoped_tokenis public SDK API and its retry behaviour changed.Fixes #599