Non-streaming requests keep decoding after the client disconnects #644

Only stream_with_cancellation (server/api_server.py:361) checks for a client disconnect. The non-streaming paths of chat completions, completions, /v1/messages and /v1/responses await the result without watching the connection, so an abandoned request keeps its slot until max_tokens.

Adding a poll there would not be enough: _record_request_middleware (api_server.py:449) is a BaseHTTPMiddleware, and under it request.is_disconnected() stays False while a non-streaming handler runs. A probe route on main's app that polls is_disconnected() never sees a client that closed its socket. With only that middleware removed from app.user_middleware, it does.

Expected: a disconnect aborts a non-streaming request, as it does a streaming one; for example, make the recorder a pure ASGI middleware and watch the connection while awaiting the result.

Environment: main 07813248 (CPU-only check), starlette 1.6.0, uvicorn 0.52.4, fastapi 0.141.1, Python 3.14.

Implementation (on my fork, on top of #222's watcher):

• 320b9e7: record requests with a pure ASGI middleware. A BaseHTTPMiddleware hides the client disconnect from the handler.
• 2be2931: test that every sample of an n>1 non-streaming request is aborted.
• 655d491: watch for disconnects on non-streaming /v1/messages and /v1/responses.