Three deadline tests fire the real deadline inside the held operation, not on wall time (BACKLOG #3871)

[wshallwshall]

Vault BACKLOG #3871, the remainder: the last three tests on the row that raced a wall-clock request deadline, plus a fourth case found in review. Together with engine PR 2438 (test_ui_dr_outlive), this closes the row. Test-only; production code is unchanged.

The race (the same in every case)

The deadline was wall time from the start of the request. When the auth dependencies, the body read, or the file save outlasted it, the 503 fired before the route reached OutlivingOperations.run, so the test asserted on state the code never reached. The merge-queue failure of test_upload_api.py::test_a_deadline_during_the_create_row_still_writes_it ("ValueError: not enough values to unpack (expected 1, got 0)", run 39177122501) is this race: the deadline fired before or during us.save, which removes its pair on cancellation (#2262). That is correct product behaviour.

What changed

• New tests/_deadline_capture.py (registered for assert rewriting in tests/conftest.py):
  • capture_deadlines(monkeypatch, app) records the middleware's real asyncio.Timeout, set to a 15 s deadline that never fires on its own.
  • The held operation awaits capture.cut_off(), which reschedules that deadline to now, so the middleware's own 503 lands mid-operation.
  • async with capture.held(): releases the operation and waits with OutlivingOperations.drain, asserting nothing was cancelled.
  • expect_cut_off asserts the exact 503 body.
  • A bad fire (zero deadlines, more than one, already disarmed, or a refused reschedule) never deadlocks: it records the reason and releases.
• test_dr_outlive_deadline.py:
  • The release test cuts off after park_intake, with the box active and intake unbound. All original end-state assertions are kept, plus inflight == ["DR release"].
  • The activation test cuts off in _activate_profile, with the original assertions kept.
• test_reload_cancellation.py: cuts off in reload_detail. It asserts inflight == ["config reload"] and no row while held, then that the row is written after the caller left and there is no interrupted row. The dead deadline parameter and the _wait helper are removed.
• test_upload_api.py:
  • The create and delete tests cut off inside the parked audit write; mid-cut, the create test asserts one stored file.
  • The mid-sweep test, added in review, fires the deadline from the sweep thread via call_soon_threadsafe.
  • _settle is removed.
• PR 2438's web console copy is untouched; that PR owns its file.

Proof

Arm	Before	After
serial, 200 runs of each of the six	0 failed	0 failed
parallel: the three files x50, -n auto (3000 tests)	3000 passed	3000 passed
slowed arm (OutlivingOperations.run delayed 0.7 s before registering), five tests	25 of 25 failed	25 of 25 and 15 of 15 passed
slowed arm 2.5 s, mid-sweep	3 of 3 failed	3 of 3 passed

Plain loops do not reproduce the flake on this box; the slowed arm does, and it reproduces the exact CI failure. Mutations, each reverted:

• OutlivingOperations.run as a plain await: five tests red, also confirmed independently by the code-review skill.
• drained hard-coded True: the release test red.
• Sweep on_cancel=lambda: None: the mid-sweep test red.

Checks

• ruff and format: clean.
• mypy linux 361, win32 342, tests 1393: all clean, re-run on the final code.
• pytest: the three files give 60 passed; the related request-timeout, DR, reload, dual-control, upload and web console DR suites give 421 passed at -n auto.
• Not run: the full suite, and the Postgres and SQL Server legs.
• The Builder's git merge-tree --write-tree origin/main eaccaab9b exited 0 at main d61284d, with a self-merge control at 0.

Open findings (code review at xhigh, 2 rounds; no correctness defect in what the tests prove)

1. The mid-sweep test keeps 5 s wall-clock safety bounds, the sweep thread's abort wait and the route's ledger cancel wait. That is far looser than the old 2.0 s race, but not zero.
2. _admin_client in test_dr_outlive_deadline keeps a dead deadline parameter; two other test files import it.
3. The seam depends on how the middleware imports asyncio. A change there fails loudly, not silently; a first-class seam would be a production change.
4. held() cannot report the operation's own exception, because OutlivingOperations exposes no task results. The outcome-row assertions cover it instead.
5. Two copies of the stand-in exist: PR 2438's web console one, with fewer checks, and this helper. They could be consolidated after both land.
6. expect_cut_off pins the 503 body literal, as test_api_request_timeout.py does.

Proposed banner

SHIPPED -- all four tests on the row (test_ui_dr_outlive in PR 2438; test_dr_outlive_deadline, test_reload_cancellation and test_upload_api here) fire the middleware's real deadline inside the held operation instead of racing wall time (BACKLOG #3871).

[wshallwshall]

QA for head eaccaab (the Builder's line, posted by the Manager): QA -- korus roles/BUILDER.md step 11 / Level: xhigh, from the brief. Tag: none printed (findings list only, both rounds). / Rounds: 2. Findings: round 1 raised 13, round 2 raised 13, all diagnostic, robustness or cleanup, none a correctness defect in what the tests prove; most fixed in 55f7dae and eaccaab; 6 open, listed in the PR body.

[wshallwshall]

Lander: enqueued at eaccaab, pinned.

• Review: the QA line names this head (code-review at xhigh, 2 rounds, no correctness defect).
• This PR changes tests only.
• Its 6 open findings are on vault #3871, now on main through pass 10 (vault PR 2547). Closing step 5 carries findings 1 and 5 to a new row when #3871 closes. Graded non-blocking: no product behaviour changes and no control is weakened. That grade is a Lander decision under the owner's Driver delegation, not an owner ruling.
• All 9 live required contexts are SUCCESS on this head.