Batch 261 (api): the API lifespans' shutdown runs every step under one budget, and upload sweep, retention and quota follow-ups (BACKLOG #3191, #3184, #3240, #3256)

changelog.d/2264.fixed.md

@@ -1,6 +1,6 @@
 - **The upload retention stop logs an audit gap only when one exists, and names the files.** At
-  shutdown the retention runner gives a sweep in flight 5 seconds, then cancels it. It used to log
-  an ERROR saying files may have lost their `upload.prune` row, even when the sweep had removed
+  shutdown the retention runner gives a sweep in flight a bounded wait, then cancels it. It used to
+  log an ERROR saying files may have lost their `upload.prune` row, even when the sweep had removed
   nothing. It now tracks each file the sweep removed and each row written. The ERROR names every
   removed file with no row yet, and a sweep that lost no row logs a WARNING instead. A runner with
   no audit callback says it owed no rows. (`BACKLOG #2264`)

changelog.d/2322.fixed.md

@@ -2,8 +2,8 @@
   Sealing it would list an upload with no usable body, and would seal a planted sidecar too.
   `rotate-key` now counts such an upload as unsealable rather than skipped. The upload stays inside
   retention: the prune removes it at expiry, timed from the sidecar's mtime, and the quota bills
-  it. A holder of `files:access_any` can delete it with `DELETE /uploads/{file_id}`, and still
-  gets 423 on browse and resend. Everyone else, its owner included, still gets 404. The startup
+  it. A holder of both `files:access_any` and `files:delete` can delete it with
+  `DELETE /uploads/{file_id}`, after step-up, and still gets 423 on browse and resend. Everyone else, its owner included, still gets 404. The startup
   warning no longer tells the operator to run `rotate-key` for it, and the 423 text no longer
   promises that `rotate-key` will seal it. It stays out of the listing. At least an oversized
   non-text body, and one that is not a regular file, are not covered; `docs/PHI.md` section 3

changelog.d/3184.changed.md

@@ -0,0 +1,6 @@
+- **An upload no longer runs the uploaded-logs retention sweep on its own request.** Under `serve`,
+  the upload route wakes the retention runner, which sweeps in the background and audits each
+  pruned file as before. An app built without the runner still sweeps at save time. The sweep now
+  writes each `upload.prune` row as soon as it removes that file, so a shutdown that cuts a sweep
+  short loses at most the row of the file in progress. Each quota check now walks the uploads
+  directory once rather than twice. (`BACKLOG #3184`)

changelog.d/3240.fixed.md

@@ -0,0 +1,5 @@
+- **`rotate-key` counts a plaintext upload body it sealed even when that upload's sidecar could
+  not be read.** Such an upload is now in the sealed-plaintext count as well as in `skipped`, so a
+  planted body is not sealed uncounted. An upload whose two plaintext halves are sealed in two runs
+  counts in both. `DELETE /uploads/{file_id}` now answers 404, not 423, for an upload whose sidecar
+  is removed while the request runs. (`BACKLOG #3240`)

changelog.d/3256.fixed.md

@@ -0,0 +1,7 @@
+- **On Postgres and SQL Server, the engine-shard upload reservation now reads the database server's
+  clock.** The row's activity stamp and its staleness test used each engine host's own clock, so a
+  host whose clock ran ahead could hold a leaked slot for as long as the skew. Each heartbeat touch
+  of a live upload's reservation is now bounded, so a stalled store cannot hold the heartbeat
+  indefinitely. `serve` now bounds uvicorn's connection drain at a stop to 3 seconds, so an open
+  request can no longer use up the service's stop window before the engine shuts down.
+  (`BACKLOG #3256`)

messagefoundry/__main__.py

@@ -2479,6 +2479,22 @@ def _missing_service_config_refused(service_config: str | None, *, fleet: str =
     return True
 
 
+#: How long uvicorn waits, at a stop, for open connections and their requests to finish before it
+#: cancels them and runs the lifespan teardown (BACKLOG #3256). The whole stop shares NSSM's
+#: AppStopMethodConsole window (scripts/service/install-service.ps1) with that teardown and
+#: engine.stop(). Unset, this wait had no bound, so one long request could spend the window and
+#: the service would be killed mid-teardown. A request cut off here is cancelled like one past its
+#: deadline, and uvicorn starts the teardown without waiting for it, so its cancel paths run
+#: alongside the teardown. **Any route that acts in a worker thread and then writes its audit row
+#: can lose that row here,** such as ``DELETE /uploads/{id}``, whose unlink finishes in its thread
+#: while the request is cancelled before its ``upload.delete`` row. At least these upload cases
+#: lose too: a save whose write is still running waits for it, with no bound, and may then pay its
+#: quota slot back to a closed store, leaving the slot to go stale; and a save cancelled in its own
+#: ``upload.create`` audit write keeps its file with no creation row. Before this bound, a request
+#: still open at the window's end was killed with the process instead, at any point.
+_UVICORN_DRAIN_SECONDS = 3.0
+
+
 def _serve(args: argparse.Namespace) -> int:
     # Vault BACKLOG #2700: first in this function, ahead of the imports below and of config,
     # because a script injected before the hook is in place runs. It is NOT first in the process:
@@ -4643,6 +4659,7 @@ def registry_filter(reg: Registry) -> Registry:  # noqa: F811 (local shard-bound
         "server_header": False,
         "http": floored_http,
         "ws": floored_ws,
+        "timeout_graceful_shutdown": _UVICORN_DRAIN_SECONDS,  # BACKLOG #3256; see the constant
     }
     from messagefoundry.api.tls import build_api_ssl_context