Repository navigation
test_ui_dr_outlive fires the request deadline inside the drain, not on wall time (BACKLOG #3871) - #2438
Merged
Conversation
…he release (BACKLOG #3871) The console DR release test set a 0.2 s request deadline and asserted the release finished once outliving_operations.inflight was empty. The deadline is wall time from the start of the request, so when the auth dependencies alone outlasted 0.2 s the 503 came before the handler reached outliving.run: nothing was ever registered, the wait loop exited at once, and coord.active was still True. The failed CI log shows only the request-timeout warning, never outlive.run's "caller was cancelled" one. Reproduced 1 in 200 serial runs locally, and every time with the first step-up check slowed by 0.5 s. The drain now blocks on an event until the 503 is in, so it cannot finish first. An attempt that registered nothing (and never reached the drain) is retried with twice the deadline, up to 6.4 s, then fails loudly. The test asserts the release is still running when the deadline answers, then lets the drain end and waits for the operation itself. No product change: a deadline that fires before the release begins leaves the box active with nothing half done, which is correct. Proposed PR title: test_ui_dr_outlive waits on the coordinator, not wall time (BACKLOG #3871)
… failed-release row (BACKLOG #3871) Code-review round 1 on the previous commit: - each post is bounded (deadline + 10 s), so a deadline that never fires fails in about 10 s instead of hanging the session on the held drain; - every 503 must carry the middleware's "timed out" body, so a handler 503 is not mistaken for a deadline that beat the release; - no dr_release_failed row on a retried attempt, after the cut-off, or at the end, so a cancel that reached the release cannot pass as the flake; - the drain is released in a finally, so a failure does not park the release on the session-scoped loop; - the final wait polls the outcome (coord inactive) as well as the registry. Proposed PR title: test_ui_dr_outlive holds the drain and retries a deadline that beat the release (BACKLOG #3871) Proposed ledger banner for #3871: PARTIAL -- the console test (test_ui_dr_outlive) is fixed; the mechanism is "before the work is registered": the deadline fired in the auth dependencies, before the handler reached outliving.run. tests/test_dr_outlive_deadline.py, tests/test_reload_cancellation.py and tests/test_upload_api.py carry the same wall-clock race and are not changed here; the row stays open.
…ain (BACKLOG #3871) Code-review round 2: the retry ladder only lowered the odds of the race. The test now swaps asyncio for a capturing stand-in inside the deadline middleware's module only, opens the release request under a 30 s deadline, and the held drain reschedules that real deadline to now. The middleware's own TimeoutError arm answers 503 while the release sits in its drain, in one attempt, whatever the auth dependencies cost. The 503 body is compared exactly; outliving.drain(timeout=10) replaces the poll and runs again in the finally; audit reads filter by action in SQL. Verified: every step-up slowed to 1.0 s passes 50/50; 200 serial and 960 at -n 96 pass; four production mutations (run not outliving, route bypassing it, release never deactivating, deadline never armed) each fail. Proposed PR title: test_ui_dr_outlive fires the request deadline inside the drain, not on wall time (BACKLOG #3871) Proposed ledger banner for #3871: PARTIAL -- the console test (test_ui_dr_outlive) is fixed; the mechanism is "before the work is registered": the deadline fired in the auth dependencies, before the handler reached outliving.run. tests/test_dr_outlive_deadline.py, tests/test_reload_cancellation.py and tests/test_upload_api.py carry the same wall-clock race and are not changed here; the row stays open.
Collaborator
Author
|
QA for head 19ae348 (the Builder's line, posted by the Manager): QA -- korus roles/BUILDER.md step 11 / Level: xhigh, from the brief. Tag: none printed (JSON findings only, both rounds). / Rounds: 2. Findings: round 1 raised 9, round 2 raised 9, all fixed except 3 open, listed in the PR body (PARTIAL: three sibling tests remain; no third review of the final rewrite; mutation 3's assertion text). |
Collaborator
Author
|
Lander: enqueued at 19ae348, pinned, ahead of other work, because this flake evicted engine PRs 2436 and 2435 from the merge queue on 2026-10-19.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Vault BACKLOG #3871, PARTIAL: makes
packaging/messagefoundry-webconsole/tests/test_ui_dr_outlive.py::test_a_console_release_cut_off_by_the_deadline_still_completesdeterministic. It failed three times on 2026-10-19 on three unrelated PRs, and two of them (2436, 2435) were evicted from the merge queue. Own PR, so it can land ahead of the rest (the Lander asked for this). Test file only; no production code changed.The race
The request deadline is wall time from the start of the request. When the auth dependencies (session, MFA, step-up) took longer than 0.2 s, the 503 came before the handler reached
outliving.run. Nothing was registered, the wait onoutliving_operations.inflightreturned at once, andcoord.activewas still True.Evidence:
This is a test defect, not a product defect. A deadline that fires before the release starts leaves the box active with nothing half done, which is correct.
The fix
A stand-in for
asyncio, patched intomessagefoundry.api.request_timeoutonly, records the deadline the middleware opens. The release runs under a 30 s deadline, and the held drain reschedules that real deadline to "now", so the middleware's own TimeoutError path answers 503 mid-drain, whatever auth costs. The test then checks:inflight == ["DR release"];coord.activeis still True;dr_release_failedrow.It then releases the drain and waits with
outliving.drain(timeout=10) == [](repeated in afinally). Every post is bounded withwait_for. The final assertions are unchanged.Proof
-n 96Mutations, each temporary and reverted, all fail fast:
outliving.runawaits the task directly;dr_releaseskipsoutliving.run;_active;Checks
--stricton the changed file: all clean.-n autoand at-n 4; the 27 engine files that reference DrCoordinator give 1811 passed and 15 skipped.git merge-tree --write-tree origin/main 19ae348faexited 0 at main cc24a84, with a self-merge control at 0.Open findings (code review at xhigh, 2 rounds)
tests/test_dr_outlive_deadline.py,tests/test_reload_cancellation.py, andtests/test_upload_api.py(_settle)._DeadlineCapture, with a reschedule from the held operation, is the reusable shape for a follow-up. The row stays open.19ae348fa, because of the two-round cap.Proposed banner
PARTIAL -- test_ui_dr_outlive fires the middleware's real deadline from inside the release's drain, so it no longer races wall time (BACKLOG #3871). Three sibling tests remain.