Move Python proxy telemetry writes to a bounded async writer #6

Problem

write_proxy_event currently opens, appends, and flushes the event log synchronously under a global lock on the request path. Slow storage, antivirus scanning, rotation, or write failure can therefore add request latency or couple best-effort telemetry to Gateway availability. The persistent debug recorder in #117 also needs a safe bounded writer primitive; it must not create a second ad-hoc queue.

Outcome

Introduce one reusable deep BoundedEventWriter module for sanitized append-only telemetry sinks. Callers learn a small non-blocking enqueue/status/flush/shutdown interface while the module owns queue bounds, batching, ordering, writer lifecycle, overflow accounting, rotation coordination, and failure isolation.

Move the existing proxy event log onto that module without changing its successful event schema or request behavior.

Scope

• Accept already-sanitized immutable records and enqueue them without file I/O on request threads.
• Bound the queue by both record count and serialized bytes. Enqueue never waits for disk; on exhaustion, fail open for the user request, drop the incoming telemetry record, and increment bounded aggregate drop counters.
• Preserve FIFO order for accepted records and use exactly one writer per sink.
• Batch append/flush work while preserving complete JSONL records; never expose a partially serialized record as valid telemetry.
• Surface sanitized writer status/counters: queued records/bytes, accepted/written/dropped counts, last failure category/time, and shutdown state. Do not expose record content or raw paths.
• Emit/reconcile a bounded aggregate overflow/failure event when writing recovers; avoid recursive logging through the same full/failed queue.
• Provide explicit bounded flush and shutdown semantics for Gateway stop/restart. A timeout or write failure must be reported but must not hang shutdown or a user request.
• Recover cleanly from writer-thread failure and repeated sink exceptions without spawning unbounded workers.
• Keep the interface reusable by Add a bounded persistent diagnostic recorder to the debug flavor #117 for its bounded diagnostics segments; Add a bounded persistent diagnostic recorder to the debug flavor #117 owns recorder schema, retention, and rotation policy.

Non-goals

• Changing route selection, request/response payloads, retry/SSE behavior, or Gateway lifecycle policy.
• Adding new diagnostic fields, persistent recorder retention, incident snapshots, or export UI.
• Guaranteeing delivery of best-effort telemetry during disk failure or process termination.
• Adding network telemetry upload.

Acceptance criteria

• Request-path calls perform bounded in-memory work only and never open, append, flush, rotate, or wait on a telemetry file.
• The queue has documented record and byte limits; overflow deterministically drops incoming telemetry, preserves accepted FIFO records, and exposes aggregate drop counts without recursion.
• One sink has exactly one writer lifecycle; concurrent producers cannot interleave or truncate JSONL records.
• Slow disk, permission errors, disk full, serialization rejection, writer crash, and repeated failure do not fail or materially delay user requests.
• Recovery resumes accepted future writes and reports one sanitized aggregate failure/recovery state.
• Bounded flush/shutdown either drains accepted records or returns a precise timeout/failure status without hanging Gateway shutdown.
• Existing proxy telemetry event bodies remain semantically equivalent on successful writes.
• The module can be reused by Add a bounded persistent diagnostic recorder to the debug flavor #117 without callers knowing thread, batching, file-handle, or recovery details.

Verification

• Deterministic tests with an injected sink/clock for FIFO ordering, batching, record+byte limits, overflow aggregation, recursive-failure avoidance, recovery, and bounded shutdown.
• Concurrency stress with multiple producers and one writer, verifying complete JSONL records and no request-thread file I/O.
• Fault injection for slow writes, permission denied, disk full, partial write, writer exception, and shutdown timeout.
• Existing focused telemetry/routing suites plus full Python tests.
• python scripts/report_quality_gates.py (report-only).

Expected hotset

• New focused Python bounded-writer module.
• Narrow integration in src-python/codex_proxy.py / existing telemetry helper.
• tests/test_proxy_event_logging.py, focused routing tests, and new deterministic writer tests.

Dependencies and relationships

• No hard blocker.
• Blocks Add a bounded persistent diagnostic recorder to the debug flavor #117's persistent debug recorder.
• Related: Diagnose generic Gateway upstream transport instability across providers #16 consumes telemetry; its analyzer semantics remain unchanged.