Add a bounded persistent diagnostic recorder to the debug flavor

Problem

Intermittent Codex Desktop/Gateway disconnects are difficult to reproduce on demand. The current #114 manual harness can capture an isolated bounded run, but it is disposable and previously recorded one event per SSE line; it is not safe or usable as an always-on recorder for normal daily work. Users need a debug flavor that continuously preserves enough privacy-safe pre-failure context to identify which side and phase failed.

Outcome

Add a deep DiagnosticRecorder module at the Gateway transport/lifecycle seam, enabled only by the compile-selected debug flavor from #116. It continuously keeps a bounded rolling two-hour evidence window, automatically or manually freezes an incident snapshot, and exposes status/mark/pause/delete controls without changing request behavior.

The recorder produces the durable evidence consumed by #114 and the versioned incident artifacts exported by #115.

Architecture and interface

Keep the caller interface small:

record one allow-listed phase transition or aggregate counter;
mark/freeze an incident with a sanitized reason category;
query recorder status;
pause/resume future capture;
delete frozen incidents.

Hide rotation, coalescing, persistence, recovery, privacy filtering, and retention inside the module. Tests and Gateway callers cross this same seam.

Scope

Continuous bounded capture

Activate automatically only in a debug flavor; normal has no supported runtime path that enables it.
Maintain a continuously rotating last-two-hours window rather than stopping after two hours.
Enforce a 32 MiB hard storage cap in addition to the time window; evict oldest complete records/segments first.
Never persist one event per SSE line. Retain first SSE, phase/terminal/close transitions, byte/line totals, and at most 16 periodic aggregate checkpoints per request.
Correlate requests/connections/retries with opaque process-local labels and monotonic ordering.
Write through the bounded asynchronous writer from Move Python proxy telemetry writes to a bounded async writer #6 so recorder I/O, rotation, and failures never block or fail a user request.

Evidence schema

Record only allow-listed sanitized fields needed to identify the first-close seam:

build version/flavor/source revision and schema version;
route/proxy mode and provider/model provenance already safe for telemetry;
downstream accept/body/response-open/headers/write/close phases;
upstream DNS/TCP/TLS/request-write/headers/first-SSE/EOF/read-failure phases where observable;
relay exposure, protocol terminal observation/forwarding, and local socket closure;
connection new/reused/unobserved state, retry attempt/budget, request-size bucket, elapsed timings, byte/line counters, and terminal outcome.

Do not retain prompt/response text, reasoning, tool names/arguments/results, request/response bodies, headers or cookies, credentials, environment/configuration values, absolute paths, raw URLs, task/session/thread IDs, or raw exception text that can contain sensitive values.

Incident lifecycle

Automatically freeze an incident for observable abnormal Gateway terminals, request errors, downstream write failures, and retry-budget exhaustion.
Provide a visible localized Mark incident action for renderer-only symptoms that the Gateway cannot detect.
Freeze the available pre-incident rolling window plus a bounded five-minute post-marker tail; normal traffic continues during and after freezing.
Retain at most three frozen incidents and delete incidents older than seven days. Eviction is oldest-first and reported in status.
Recover or discard incomplete segments atomically after a crash; never present a partial snapshot as complete evidence.
Expose recorder status: active/paused, flavor, rolling time/bytes, incident count, last marker category/time, truncation/eviction state, and schema version. Do not expose private identifiers.

Integration

Store recorder state below the authoritative runtime home in a dedicated diagnostics subtree with bounded permissions and atomic metadata updates.
Define a versioned, allow-listed incident manifest and snapshot layout for Add a privacy-safe diagnostic log export for user support #115 to export without reading internal implementation state.
Adapt the reusable, sanitized instrumentation/classifier seams proven on Find and fix Responses disconnects across official and external routes #114; do not copy the disposable runner or its unbounded per-line recorder into production.
Keep provider routing, request/response payloads, retry decisions, terminalization, and Gateway lifecycle unchanged.

Non-goals

Enabling the recorder in normal through a setting/environment toggle.
Capturing Codex App conversations, renderer contents, raw Codex logs, credentials, or configuration snapshots.
Automatic network upload or support transmission.
Fixing Find and fix Responses disconnects across official and external routes #114's initiating disconnect or changing transport/retry/SSE behavior.
Owning ZIP/save-dialog export UI, which remains Add a privacy-safe diagnostic log export for user support #115.
Replacing ordinary bounded operational telemetry.

Acceptance criteria

Verification

Deterministic unit/property tests for time/byte rotation, record-boundary eviction, 16-checkpoint coalescing, three-incident/seven-day retention, clock rollback, crash recovery, and atomic metadata.
Fault-injection tests for queue overflow, disk full, permission failure, malformed fields, interrupted writes, shutdown, and concurrent markers.
Privacy scans over rolling/frozen bytes with seeded prompts, tool data, credentials, headers, URLs, paths, IDs, and exception details.
Find and fix Responses disconnects across official and external routes #114 first-close replay plus a healthy long-stream replay using the versioned frozen artifact.
Packaged normal/debug Windows smoke proving activation boundary, UI status/mark/pause/delete, continuous Gateway traffic, and no duplicate Gateway.
Focused/full Python, Rust, frontend/UI-contract gates and python scripts/report_quality_gates.py (report-only).

Expected hotset

A new focused Python recorder/writer module and narrow hooks at src-python/codex_proxy.py transport/lifecycle seams.
Reusable sanitized instrumentation/classifier code currently proven by Find and fix Responses disconnects across official and external routes #114.
Runtime-path/config plumbing and focused Tauri commands for status/mark/pause/delete.
A small debug-only Settings/Diagnostics status surface and localization/UI-contract tests.
Privacy, rotation, fault, and packaged-smoke fixtures.

Dependencies and relationships

Blocked by Move Python proxy telemetry writes to a bounded async writer #6 for the reusable bounded async writer.
Blocked by Build normal and debug flavors from each release version #116 for compile-selected normal/debug flavor identity.
Blocks Add a privacy-safe diagnostic log export for user support #115's incident-artifact export integration.
Provides durable evidence to Find and fix Responses disconnects across official and external routes #114; it does not itself close Find and fix Responses disconnects across official and external routes #114.

Execution contract

Execution-Contract: v2
Verification-Class: strict
Verification-Commands:

python -m pytest -q
npm run build and npm run test:ui-contract in frontend/
cargo test --locked and cargo clippy --locked --all-targets -- -D warnings in src-tauri/
deterministic recorder privacy/rotation/fault fixtures named in this Issue
Manual-Evidence: packaged normal/debug Windows smoke proving the compile-selected activation boundary, visible recorder controls, continuous Gateway traffic, and no duplicate Gateway
Architecture-Decision: resolved
Review-Owner: orchestrator

First implementation slice after the natural reconnect capture

The natural Debug capture in #114 now validates the need for this recorder's per-attempt timing and connection-disposition fields. It observed two pre-response Official failures (about 61 seconds and 300 seconds), a third attempt that began streaming, and a later downstream close after 381 seconds total; the current generic tcp_connect label cannot distinguish the physical boundary. Evidence: #114 comment 4965232208.

To start immediately without colliding with active #106 / PR #122, the first Worker slice is constrained to:

a new focused Python DiagnosticRecorder deep module;
bounded rolling segments, allow-list/privacy filter, rotation/retention, incident freeze, and deterministic status model;
narrow instrumentation hooks in src-python/codex_proxy.py for per-attempt start/end elapsed time, response headers, first SSE, terminal/close, retry budget, and opaque connection new/reused/unobserved state;
deterministic replay/privacy/rotation/fault tests, including the sanitized 61s → 300s → streamed → downstream-close ordering;
no transport, retry, timeout, routing, payload, or terminalization behavior change.

Until #106 integrates, this slice must not modify src-tauri/src/proxy.rs, src-tauri/tauri.conf.json, src-tauri/src/models.rs, or the official catalog metadata/config hotset. The later Tauri/frontend activation and packaged-smoke slice remains part of #117, but integration is serialized after #106 to avoid duplicate edits and review churn.

The Issue remains one strict work item and one future PR. Worker formal Review-Runs must remain 0; the Orchestrator owns the single Standards/Spec review.

WORKER_DISPATCH

Profile/model: core/debug, gpt-5.6-terra / max.
Base: origin/dev@db1ea61de715f2f8857f27ef168b851f9bfc1e37.
Branch: codex/issue-117-diagnostic-recorder; PR target: dev.
Authorized first slice: focused Python DiagnosticRecorder, bounded rolling/privacy/freeze/status core, narrow src-python/codex_proxy.py observation hooks, and deterministic privacy/rotation/fault/first-close replay tests.
Prohibited until Restore reliable sidebar Task control and continuation while CodexHub is active #106 integrates: Tauri/frontend activation, official catalog/config hotset. Always prohibited: transport, retry, timeout, routing, payload, or terminalization behavior changes.
Verification-Class: strict. One candidate full relevant gate; Worker Review-Runs=0; Orchestrator owns the single Standards/Spec review. Report-only quality checks remain non-blocking.
Required signals: native callback for BLOCKED, DISCUSSION_REQUIRED, PR_OPENED, READY_FOR_REVIEW, and STOPPED before the local final response.

Claim and preflight are verified. Scoped edits begin only after the private CLAIM_CONFIRMED continuation.

#117 implementation plan — authorized first slice

Add src-python/diagnostic_recorder.py: a focused, fail-closed recorder over Move Python proxy telemetry writes to a bounded async writer #6's bounded writer with opaque monotonic labels, bounded rolling segments (two hours / 32 MiB), deterministic retention/freeze/status, and read-only artifact/replay helpers.
Add tests/test_diagnostic_recorder.py: deterministic privacy, coalescing (at most 16 checkpoints), rotation/retention, fault isolation, recovery, and sanitized 61s → 300s → streamed → downstream-close replay coverage.
Add narrow observation-only calls in src-python/codex_proxy.py for upstream attempts/headers, first-SSE/totals/terminal/close, retry budget, and opaque connection disposition. No routing, retry, timeout, transport, payload, or terminalization behavior changes.
Verification: focused recorder/proxy tests during development; one python -m pytest -q candidate gate for the changed Python surface; python scripts/report_quality_gates.py remains report-only; Worker Review-Runs stays 0.
Collision evidence: Restore reliable sidebar Task control and continuation while CodexHub is active #106 / PR Fix official model planner metadata catalog #122 changes catalog/Tauri paths and tests/test_catalog_sync.py; this slice is a new Python module/test plus codex_proxy.py, so the path intersection is empty. Tauri/frontend/catalog changes remain excluded.
Decision gate: none newly exposed; the resolved Issue contract governs the versioned read-only incident artifact. Debug activation/UI and packaged smoke stay serialized until Restore reliable sidebar Task control and continuation while CodexHub is active #106 integrates.

Second-slice implementation plan

Activate the recorder only from the compile-selected debug Tauri build by bootstrapping the existing Gateway child before its unchanged codex_proxy.main() entry point; the normal flavor keeps the direct child launch.
Add a diagnostics runtime/control subtree under the existing runtime home, plus narrow Tauri status, manual-mark, pause/resume, and delete commands. The child remains the sole Gateway process; no diagnostic listener or second Gateway is introduced.
Add a small debug-only Gateway UI panel and localized copy, and preserve the v1 read-only incident artifact contract for Add a privacy-safe diagnostic log export for user support #115.
Verify focused affected tests, Rust tests/clippy, frontend build/UI contract, flavor contract, and packaged normal/debug smoke. The accepted full Python run remains the only full Python gate.

Expected writes: src-tauri/src/proxy.rs, src-tauri/src/main.rs, new focused diagnostics Rust/Python runtime-control modules and tests, Gateway UI/types/locales, release/packaging checks, and the PR body. src-python/codex_proxy.py will remain exactly as in the accepted first slice.

Collision check: no additional proxy.py edits; no edits to src-python/model_limits.py, src-python/catalog_sync.py, src-python/config_overlay.py, or #124 context-limit tests. The normal merge brought #106 files forward without overlap.

Decision gate: resolved. The debug-only bootstrap will execute the already-owned Gateway entry point in the same child process and use the authoritative diagnostics subtree for bounded control exchange, so activation requires no protected proxy.py change and does not create a second Gateway or port.

#117 review-delta fix plan

Tighten DiagnosticRecorder freeze semantics so marker-time prehistory is retained immediately, retention is record-boundary/clock-rollback safe, and incident manifests are strict allow-list readbacks.
Add the missing sanitized evidence fields and deterministic privacy/clock/concurrent-marker/healthy-stream replay coverage in the focused recorder tests.
Make the authorized narrow codex_proxy.py diagnostic-only changes: observe terminal before downstream write/flush so terminal-not-forwarded is provable, add supported downstream/upstream phase observations and request-size bucketing, and populate connection disposition when safely observable.
Keep transport, retry, timeout, routing, payload, and terminalization behavior unchanged; do not touch Keep Codex auto-compaction within the current Official context budget #124 model/context paths. The only collision overlap is the expressly authorized diagnostic-only proxy seam.
Verify focused recorder/Gateway/privacy/replay tests, affected frontend/Rust checks only if a changed seam requires them, diff hygiene, and CI. The accepted full Python suite remains at one run; Worker Review-Runs remains 0.

Decision gate: inspect the existing supported HTTP pool seam for safe new/reused evidence. If it is not observable without a durable architecture expansion, I will send DISCUSSION_REQUIRED with the inspected seam before omitting it.

Packaged Windows smoke feedback — UI delta required

The normal/debug packaged candidate at c529f6ee passed the runtime portions of the manual gate:

normal: diagnostics disabled, no diagnostics runtime subtree, 20/20 continuous health requests, exactly one Gateway child;
debug: compile identity and source revision correct, recorder active, pause/resume/manual mark succeeded, 60/60 continuous health requests, exactly one Gateway child;
the debug controls are visibly present.

Visual acceptance is not complete. The diagnostics panel is too tall and keeps low-frequency details/actions permanently visible, reducing the existing Gateway page's information density.

Accepted narrow UI delta:

Default to a compact collapsed card matching the surrounding page's existing card height/density.
The collapsed surface shows only title/status and one concise summary line: rolling window, retained bytes, and incident count.
Clicking the card's disclosure/chevron expands inline details. Only the expanded state shows the explanatory copy, mark/pause-or-resume/refresh controls, and incident deletion list.
Preserve all recorder/Gateway behavior, polling, localization, keyboard focus, and an accurate aria-expanded state.
Add a focused UI contract/regression proving default-collapsed content hiding and explicit expansion. Run targeted frontend checks plus CI only; do not repeat full Python/Rust suites or the formal Standards/Spec review.

The previously passed runtime smoke remains valid. After this UI-only delta, rerun only the packaged debug visual check.

UI acceptance delta plan:

Expected writes: frontend/src/components/DebugDiagnosticsPanel.tsx, the English/Chinese locale resources, and the focused UI contract.
Make the existing debug card compact/collapsed by default, with an accessible inline disclosure; keep polling and recorder actions unchanged inside the expanded content.
Verify with the focused regression, npm run test:ui-contract, npm run build, diff hygiene, and CI only. The prior full-suite count remains 1.
Collision check: clean head at c529f6ee; no Python/Rust/Gateway/build-flavor or Keep Codex auto-compaction within the current Official context budget #124 model/context paths will be touched. The recorded UI contract leaves no material decision gate.

UI acceptance correction — compact height is part of the contract

The default-collapsed disclosure at f93f045b hides its detailed content correctly, but the Gateway page still assigns the diagnostics card the flexible minmax(320px,1fr) row. The result is a large empty card that breaks the page hierarchy.

Required final behavior:

collapsed Debug Diagnostics is content-height and approximately the same compact density as the existing Recovery card;
title/status/chevron plus the one-line rolling summary remain visible;
expanded details/actions/incidents grow naturally, without a forced page-filling minimum;
the Usage chart alone retains the flexible minmax(320px,1fr) row;
no recorder, Gateway, Rust, Python, or Keep Codex auto-compaction within the current Official context budget #124 behavior changes.

The intended narrow layout correction is a four-row left grid (auto / auto / auto / minmax(320px,1fr)) with focused UI-contract coverage. This supersedes visual acceptance of the prior collapsed layout; all earlier functional/package evidence remains accepted.

Implementation plan for the #117 UI allocation delta:

Expected writes: frontend/src/pages/GatewayPage.tsx (one four-row left-grid allocation) and frontend/scripts/ui-contract.test.mjs (focused regression).
Change the left Gateway column to auto auto auto minmax(320px,1fr) so status, Recovery, and Debug Diagnostics stay content-sized while only the usage chart receives the flexible row.
Collision check: fresh packaged preflight passed at f93f045b; working tree/index are clean; no Keep Codex auto-compaction within the current Official context budget #124 paths or backend/Gateway recorder surfaces are in scope.
Decision gate: none; this applies the documented visual diagnosis directly.
Verification: focused UI contract, npm run test:ui-contract, npm run build, git diff --check, then CI only.

Final Debug diagnostics information architecture

The inline diagnostics card is rejected even at compact height. Replace it with an on-demand overlay:

Remove the entire inline Debug Diagnostics card from the Gateway left-column flow.
Restore the normal three-row page hierarchy: Gateway status, Recovery, and the flexible Usage chart.
In the Recovery event row, add a diagnostics icon button immediately before the existing recovery-overview/list button.
Render that trigger only when build.flavor == debug and diagnostics_enabled == true; normal builds have neither the trigger nor overlay.
Clicking the trigger opens a localized, keyboard-accessible modal/overlay using the existing Recovery Overview overlay/card/control language.
Move recorder status, rolling summary, explanatory copy, mark, pause/resume, refresh, incidents, and deletion controls into the overlay. Preserve the existing five-second status refresh while the overlay is open and preserve all action behavior.
Provide an explicit close control, dialog semantics, focus treatment, unavailable/Gateway-stopped/loading/error states, and content scrolling without changing the page layout.
Reuse the existing icon library and design tokens; no new raster asset is required.

This supersedes the prior inline-card visual acceptance. All recorder/Gateway/package evidence remains accepted; only UI/localization/focused UI contracts and visual QA may change.

Implementation plan for the final #117 diagnostics UI delta:

Expected writes: frontend/src/pages/GatewayPage.tsx, frontend/src/components/DebugDiagnosticsPanel.tsx (recast as the on-demand overlay), both locale resources, and frontend/scripts/ui-contract.test.mjs.
Remove the inline diagnostics grid child; restore Gateway/Recovery/Usage three-row allocation. Add the debug-only 28px trigger immediately before Recovery Overview and render the overlay outside the page grid.
Reuse the Recovery Overview overlay/control language; poll diagnostics only while enabled, Gateway-running, and open. Keep actions/toasts/read-only recorder behavior unchanged.
Collision check: fresh preflight passed at 17d7a355; worktree/index are clean; no Keep Codex auto-compaction within the current Official context budget #124, Python, Rust, Gateway, recorder, flavor, transport, or design-QA artifact path is in scope.
Decision gate: none; this is the accepted final information architecture.
Verification: focused overlay contract, npm run test:ui-contract, npm run build, git diff --check, and CI only.

Completed by PR #125 at merge commit 7287d03. Final CI, Orchestrator review, packaged normal/debug smoke, and maintainer visual acceptance passed. Follow-up notification placement/policy work is tracked separately in #126.

Navigation Menu

Repository navigation

Add a bounded persistent diagnostic recorder to the debug flavor #117

Problem

Outcome

Architecture and interface

Scope

Continuous bounded capture

Evidence schema

Incident lifecycle

Integration

Non-goals

Acceptance criteria

Verification

Expected hotset

Dependencies and relationships

Execution contract

First implementation slice after the natural reconnect capture

#117 implementation plan — authorized first slice

#117 review-delta fix plan

Packaged Windows smoke feedback — UI delta required

UI acceptance correction — compact height is part of the contract

Final Debug diagnostics information architecture

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Add a bounded persistent diagnostic recorder to the debug flavor #117

Description

Problem

Outcome

Architecture and interface

Scope

Continuous bounded capture

Evidence schema

Incident lifecycle

Integration

Non-goals

Acceptance criteria

Verification

Expected hotset

Dependencies and relationships

Execution contract

Activity

NOirBRight commented on Jul 14, 2026

First implementation slice after the natural reconnect capture

NOirBRight commented on Jul 14, 2026

NOirBRight commented on Jul 14, 2026

#117 implementation plan — authorized first slice

NOirBRight commented on Jul 14, 2026

NOirBRight commented on Jul 14, 2026

#117 review-delta fix plan

NOirBRight commented on Jul 14, 2026

Packaged Windows smoke feedback — UI delta required

NOirBRight commented on Jul 14, 2026

NOirBRight commented on Jul 14, 2026

UI acceptance correction — compact height is part of the contract

NOirBRight commented on Jul 14, 2026

NOirBRight commented on Jul 14, 2026

Final Debug diagnostics information architecture

NOirBRight commented on Jul 14, 2026

NOirBRight commented on Jul 14, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions