Repository navigation
Add pinned real-client E2E qualification foundation for Gateway candidates #190
Description
Outcome
Build the strict, repeatable twelve-round real-client qualification gate for every Gateway candidate SHA. HTTP/config preflight is setup evidence only; it cannot satisfy a real-client round.
Dependency and production seam
- Native blocker: Expose headless managed-client config materialization for isolated runners #194. The runner must invoke the candidate-built production Rust managed-client preview/apply/readback seam under fresh isolated roots.
- The runner must consume the actual managed selector, canonical Gateway model, endpoint root, and Responses/Chat format returned by that seam. It must not reimplement or hardcode Codex, OpenCode, ZCode, Pi, or OMP configuration, including Volc protocol choice.
- Preserve the parked Add pinned real-client E2E qualification foundation for Gateway candidates #190 Workspace/WIP. Resume it only after Expose headless managed-client config materialization for isolated runners #194 merges to
dev. - Preserve OpenCode tool-result history on third-party transparent Responses routes #193 follows Add pinned real-client E2E qualification foundation for Gateway candidates #190 and owns the transparent OpenCode/Volc tool-history product fix.
Authoritative environment and compatibility baselines
The dedicated machine-bound local Windows host environment codexhub-real-client-e2e is authoritative. Its host manifest must match the current Windows MachineGuid binding; a VM or named snapshot is not required.
Minimum compatible client baselines (newer stable versions are allowed and their actual versions are recorded):
- Codex Desktop
>=26.715.8383.0 - Codex CLI
>=0.144.5 - ZCode
>=3.3.6 - OpenCode
>=1.18.4;1.18.4is the first release containing upstream response-header-timeout fix67caf894e0843ee370e72839e8265e483233479b - Pi
>=0.80.6 - OMP
>=17.0.3
Versions below the floor, prerelease/suffix ambiguity, multiple conflicting version tokens, and unparseable output fail closed. Real-client protocol/output drift also fails the matrix even when the installed version is newer.
Every invocation uses a fresh output root, dedicated isolated Codex account/auth input, isolated Volc credential, isolated Gateway configuration, and isolated client/work roots. Host shared sessions, credentials, configuration, and work directories are never discovered, copied, mounted, or modified.
Twelve-round matrix
Run Codex Desktop, Codex CLI, OpenCode, ZCode, Pi, and OMP against both routes:
- Official canonical model:
gpt-5.6-luna - Third-party canonical model: Volc
glm-5.2
Use only production-managed selectors returned by #194. No fallback to Sol, Kimi, Ollama, or another provider.
Each round proves the real process/UI completes, the canonical model is correct, one random sentinel is read through a real read-only tool call, a terminal stream event occurs, exactly one final request_complete=200 is recorded, and there is no silent Official fallback, duplicate output, or unclassified reconnect. CLI clients are runner-launched. Desktop and ZCode use structured human replay at the authoritative machine-bound host console.
Retry once only for a clearly pre-output upstream 429, 503, or capacity failure; retain both attempts. Any output makes retry ineligible.
Bootstrap calibration
Run released CodexHub 0.1.6 commit cc9df197a709fb4c7548021819ecb8fa716ed664 across all twelve rounds to establish environment validity. The known OpenCode/Volc tool-loop defect is an expected stable product failure classified exactly excessive_tool_loop; it is not infrastructure success and does not make 0.1.6 an all-green baseline. The #193 fix candidate is the first required all-green twelve-round baseline.
Evidence and sanitization
Publish only redacted summaries, actual verified versions, candidate/build/evidence SHA-256 values, canonical models, bounded timings/counts/classifications, outcomes, and relative artifact names. Never publish prompts, outputs beyond fixed sentinels, credentials, authorization values, usernames, absolute paths, or request/session/runtime IDs.
Any code-fix SHA invalidates prior build, review, and twelve-round evidence.
Scope and verification
Repository changes stay in scripts/, tests/fixtures/real_client_e2e/, tests/test_real_client_e2e.py, and docs/agents/real-client-e2e.md. Production Gateway/Rust/frontend/installer/client adapters remain owned by other Issues.
Required gates: targeted tests, full python -m pytest -q, PowerShell parsing, git diff --check, report-only python scripts/report_quality_gates.py, exact-SHA GitHub Actions, strict heavy review, and authoritative machine-bound host evidence. Squash to dev only; never publish main automatically.
Activity
NOirBRight commentedon Jul 20, 2026
{ "contract": { "acceptance": [ "The dedicated machine-bound Windows host environment codexhub-real-client-e2e is authoritative; a VM or named snapshot is not required, and a mismatched or malformed machine-binding manifest fails before candidate/client launch.", "Each real client is verified from its authoritative Windows installation/version source against a minimum compatibility baseline: Codex Desktop >=26.715.8383.0, Codex CLI >=0.144.5, ZCode >=3.3.6, OpenCode >=1.18.4, Pi >=0.80.6, and OMP >=17.0.3. Newer stable versions are allowed and their actual normalized versions are recorded; versions below the floor, prerelease/suffix ambiguity, multiple conflicting versions, or unparseable output fail closed.", "Deterministic version-policy fixtures accept every exact baseline and representative newer stable releases (including Codex CLI 0.145.0), reject every below-floor release, and preserve authoritative Desktop AppX plus ZCode publisher/root binding; real-client protocol/output drift still fails the twelve-round evidence rather than being hidden by version acceptance.", "The runner invokes the candidate-built #194 seam and accepts only its production apply/readback selectors, canonical models, endpoints, and route formats; no hardcoded Volc Chat or duplicated client schema remains.", "All candidate and client processes consume fresh case-local configuration and secret-bearing isolated inputs; no host shared session, credential, client config, or work directory is read, copied, mounted, or modified.", "All twelve rounds use Official gpt-5.6-luna or third-party Volc glm-5.2 with no fallback to Sol, Kimi, Ollama, or another provider and prove the real process/UI, one successful read-only sentinel tool call, terminal stream event, one request_complete=200, no duplicate output, and no unclassified reconnect.", "Desktop and ZCode require structured human GUI finalization bound to the authoritative machine-bound host run and candidate SHA; missing login, credentials, GUI, or contradictory evidence fails closed.", "The 0.1.6 release calibration distinguishes environment validity from product success: the known OpenCode/Volc loop is exactly excessive_tool_loop, retains bounded evidence, and does not make the baseline all-green; the Issue #193 fix candidate is the first required twelve-round all-green baseline.", "Only one retry is allowed for a clearly pre-output upstream 429, 503, or capacity failure, both attempts are retained, and any output makes the attempt ineligible.", "Fake-executable tests cover exit, timeout/cleanup, truncation, redaction, malformed/error output, manual merge/staleness, version normalization, host-environment/machine-binding mismatch, production-seam readback, bounded excessive tool loops, and host-session rejection.", "Targeted tests, full pytest, PowerShell parsing, diff check, report-only quality gates, exact-SHA Actions, strict heavy review, and authoritative machine-bound host evidence pass for the same final SHA.", "A deterministic regression launches an expanding descendant tree with a long-lived ping process, removes an intermediate parent, and returns within the outer deadline with no surviving descendant.", "A subprocess timeout cannot hang while draining inherited stdout or stderr pipes, and an unattended Issue test or runner command cannot survive after its supervising agent exits.", "Manual Desktop/ZCode evidence timeout is explicit and bounded and is never converted into an infinite automated wait.", "Timeout diagnostics contain only the failed phase and bounded process counts and contain no credentials, prompts, absolute private paths, or process command secrets.", "Existing Issue #190 positive, timeout, GUI cleanup, candidate startup, isolation, and authoritative host evidence behavior remains fail-closed and green." ], "change_claims": { "paths": [ "scripts/Run-RealClientE2E.ps1", "tests/test_real_client_e2e.py", "tests/fixtures/real_client_e2e", "docs/agents/real-client-e2e.md" ], "resources": [ "host-environment:codexhub-real-client-e2e" ] }, "dependencies": { "dispatch_after": [ 194 ], "merge_after": [ 194 ] }, "design": [ "Use the dedicated machine-bound Windows host environment codexhub-real-client-e2e as the authoritative environment; a VM or named snapshot is not required. Require a fresh output root, isolated account/auth, Volc credential, Gateway config, and work directory for every invocation, and bind the run to the host manifest.", "Consume the candidate-built headless managed-client materialization seam from Issue #194 for Codex, OpenCode, ZCode, Pi, and OMP; never reimplement or hardcode client configuration, selector mapping, endpoint roots, protocol formats, ownership, or readback in the runner.", "Treat the documented client versions as minimum compatibility baselines, not exact pins: Codex Desktop 26.715.8383.0, Codex CLI 0.144.5, ZCode 3.3.6, OpenCode 1.18.4 (the first release containing the header-timeout fix), Pi 0.80.6, and OMP 17.0.3. Accept newer stable versions when their authoritative normalized version is at or above the baseline, record the actual version, and rely on the real-client matrix to fail closed on protocol incompatibility.", "Run all twelve Desktop, Codex CLI, OpenCode, ZCode, Pi, and OMP rounds against Official gpt-5.6-luna and third-party Volc glm-5.2 using the actual production-managed selectors, canonical models, endpoint roots, and protocol formats read back from the #194 seam.", "Treat the CodexHub 0.1.6 release run as environment calibration: all infrastructure and unaffected rounds must validate, while the already-known OpenCode/Volc tool-loop defect is an expected stable product failure classified excessive_tool_loop, never infrastructure success or an all-green baseline.", "Emit and merge structured human-finalized Desktop and ZCode evidence bound to the authoritative machine-bound host run and candidate SHA; login, credentials, security prompts, and observed GUI finalization remain human actions and fail closed.", "Emit exactly one sanitized summary.json with approved versions, candidate/build/evidence hashes, canonical models, bounded timings/counts/classifications, outcomes, and relative artifact names only; never expose credentials, absolute paths, prompts, outputs, or private runtime identifiers.", "Keep production Gateway, Rust, frontend, installer, and managed-client implementation out of this Issue; change only runner scripts, deterministic fixtures/tests, and operator documentation, then squash to dev only.", "Bound every unattended pytest and real-client runner invocation with an explicit outer wall-clock deadline; a per-subprocess timeout is not a suite deadline.", "Own and terminate the complete Windows process tree on timeout, including descendants that detach or outlive an intermediate parent, and ensure inherited stdout/stderr handles cannot keep Python subprocess communication blocked after child termination.", "On timeout, emit exactly one bounded sanitized failure record with phase and process counts, return non-zero promptly, and keep the explicit Desktop/ZCode human-evidence window finite and separate without weakening isolation or evidence binding." ], "difficulty": "heavy", "done_when": [ "python -m pytest -q tests/test_real_client_e2e.py", "Version compatibility tests accept baseline-or-newer stable clients, reject below-floor/ambiguous versions, and record the actual versions in sanitized evidence", "python -m pytest -q", "PowerShell parser succeeds for scripts/Run-RealClientE2E.ps1", "git diff --check", "python scripts/report_quality_gates.py (report-only)", "Authoritative machine-bound host calibration for cc9df197a709fb4c7548021819ecb8fa716ed664 records only the known OpenCode/Volc excessive_tool_loop product failure and validates the remaining environment", "GitHub Actions and strict heavy review pass; the #193 candidate later supplies the first all-green twelve-round matrix", "Issue #196 expanding-descendant, inherited-pipe, outer-watchdog, and manual-window regressions pass within their documented deadlines", "No timeout path leaves pytest, Gateway, GUI fixture, ping, or descendant processes running" ], "priority": "P1", "risk": "strict", "sha256": "ac13c0da8f179812aec60bd4090a4c22bfae0bc1b10cae8cf194e6ee44ddcc66", "unresolved_decisions": [] }, "dispatch": { "accepted_at": "2026-07-22T14:30:35Z", "attempt": 2, "base_sha": "0510367b3e34c5efe00ad487081892c51f762ae7", "branch": "work/issue-190", "candidate_sha": "3aae4a5df4f95e6c8ac6b3aa559ae3b7e7683fe3", "claimed_at": "2026-07-20T05:42:44.45368Z", "contract_sha256": "ac13c0da8f179812aec60bd4090a4c22bfae0bc1b10cae8cf194e6ee44ddcc66", "creator_agent_id": "fc91861e-6478-4d4e-8019-d7d0cff5c451", "id": "dispatch-issue-190-a2", "integrating_at": "2026-07-22T15:30:23.471002Z", "last_lifecycle_action_id": "resume-dispatch-issue-190-a2-g2", "lifecycle_generation": 2, "merged_at": "2026-07-22T15:30:50Z", "parked": false, "pr_number": 192, "recovery_prompt_sent": true, "resumed_at": "2026-07-21T12:29:40.303461Z", "status": "merged", "worker_agent_id": "f70e362a-0062-4612-aad9-768af7135ddb", "workspace_id": "wks_302da564197ebeb3" } }test(gateway): harden isolated managed-client config coverage (#194)
Expose isolated managed-client config CLI seam (#194) (#195)
NOirBRight commentedon Jul 22, 2026
Lifecycle finding: #196 now natively blocks completion of #190. On 2026-07-22 an earlier tests/test_real_client_e2e.py pytest survived over 24 hours after its parent exited, together with an orphan long-running ping fixture. The current candidate and green Actions remain useful, but #190 is not complete until #196 proves an outer watchdog, inherited-pipe-safe timeout handling, and full descendant cleanup. Preserve the current worktree and serialize this through the existing hotset.
NOirBRight commentedon Jul 22, 2026
Delivered by merged PR #192.