Repository navigation
Add a global Gateway concurrency limit for upstream requests #21
Description
Problem
The Python Gateway uses ThreadingHTTPServer, which creates a thread per connection and currently has no global upstream concurrency limit. During Codex App retry storms, repeated fast failures can amplify into local thread/resource pressure.
Scope
- Add a simple global concurrency guard around upstream-generating requests.
- Decide behavior when saturated: bounded wait, 429/503, or fast typed error.
- Ensure health/status endpoints remain responsive where practical.
Acceptance Criteria
- Upstream-generating requests are capped by a configurable limit.
- Saturation is visible in telemetry with a distinct error code/classification.
- Existing normal request behavior and tests remain stable.
- Reconnect diagnostics can tell resource saturation apart from upstream transport failure.
Required Verification
python -m pytest tests/test_routing.py tests/test_proxy_event_logging.pyRelated
- Add official transport recurrence analyzer for post-open failures #12
- Diagnose generic Gateway upstream transport instability across providers #16
docs/reviews/v0.2-external-audit-reconciliation.md
Activity
NOirBRight commentedon Jul 9, 2026
Agent Brief
Category: enhancement
Summary: Add bounded Gateway upstream concurrency to prevent retry-storm resource amplification.
Current behavior:
System.Collections.Hashtable.Current
Desired behavior:
System.Collections.Hashtable.Desired
Key interfaces:
Acceptance criteria:
Out of scope:
wzg0911 commentedon Jul 12, 2026
Thread-per-connection with no upstream concurrency limit is a classic failure amplifier. We've seen this pattern in the wild — during Codex Hub-level retry storms, the lack of a global ceiling turns a transient transport failure into a resource cascade.
Production observation: In one deployment, a single provider auth failure cascaded into rapid gateway restarts (every ~500ms) because each retry spawned a new thread, each thread hit the same dead auth path, and the OS thread scheduler compounded the latency. The fix was a global concurrency guard similar to what you're proposing.
Design thoughts on the saturated-state behavior:
The #12/#16 telemetry work is a good prerequisite. Without measuring the actual concurrency pressure during storms, it's hard to tune the limit correctly your first try. Ship the telemetry first, read the data for a week, then set the initial cap.
NOirBRight commentedon Jul 20, 2026
Moved to
needs-info. Do not select a queue limit or saturation status until the #18 → #19 → #20 transport chain and #114 Direct Official / Gateway Luna / Gateway Volc fault injection produce measured concurrent-request, queue-pressure, retry, and first-failure evidence. A configuration or HTTP preflight is not sufficient proof for this policy decision.