Skip to content

Bound real-client E2E wall-clock and reap orphan process trees #196

Closed

Description

@NOirBRight

Outcome

Make the real-client E2E test and runner boundaries provably bounded: a failed or abandoned child process cannot leave pytest, GUI fixtures, Gateway fixtures, or descendants running indefinitely.

Observed evidence

On 2026-07-22, the Issue #190 worktree contained an orphaned command:

  • python -m pytest -q tests/test_real_client_e2e.py had been alive for more than 24 hours.
  • Its parent process no longer existed, CPU time was unchanged at 4.12 seconds, and all three threads were waiting.
  • An orphan ping.exe 127.0.0.1 -t from the expanding-process-tree fixture remained alive with a dead parent.
  • The relevant fixture path exercises tests/test_real_client_e2e.py::test_manual_timeout_cleanup_is_bounded_and_still_writes_one_summary.

This is a harness failure, not evidence that a real twelve-case E2E legitimately needs multiple days.

Scope

  • Put an outer wall-clock deadline around every unattended pytest/runner invocation.
  • Make timeout cleanup own and terminate the full Windows process tree, including descendants that outlive or detach from an intermediate parent.
  • Ensure inherited stdout/stderr handles cannot leave Python subprocess communication waiting forever after the direct child exits or is killed.
  • On timeout, write one bounded sanitized failure record and return non-zero promptly.
  • Document the distinction between automated deadlines and the explicit Desktop/ZCode human-evidence window.
  • Do not weaken real-client isolation, evidence binding, or fail-closed behavior.

Acceptance criteria

  • A deterministic regression launches an expanding descendant tree containing a long-lived ping process, removes an intermediate parent, and proves the test returns within its deadline with no surviving descendant.
  • A subprocess timeout cannot hang while draining inherited stdout/stderr pipes.
  • The Issue test module has an external watchdog; an unattended validation command cannot remain alive after its supervising agent exits.
  • Manual evidence timeout is explicit and bounded; it is never silently converted into an infinite automated wait.
  • Timeout diagnostics contain the failed phase and bounded process counts, but no credentials, prompts, absolute private paths, or process command secrets.
  • Existing Add pinned real-client E2E qualification foundation for Gateway candidates #190 positive, timeout, GUI-cleanup, candidate-startup, and isolation tests remain green.

Verification

python -m pytest -q tests/test_real_client_e2e.py
python -m pytest -q
git diff --check
python scripts/report_quality_gates.py

The report-quality command is report-only. Verification class is strict because this changes process lifecycle, cancellation, and release-gate evidence.

Relationships

Activity

  1. added
    bugSomething isn't working
    ready-for-agentFully specified, ready for an AFK agent
    wayfinder:taskWayfinder implementation task
    tier:heavyOrchestrator provider tier: heavy
    P1Priority 1: current iteration goal
    on Jul 22, 2026
  2. NOirBRight commented on Jul 22, 2026

    @NOirBRight
    OwnerAuthor

    Coordinator scheduling decision: keep this blocker in the existing #190 Workspace because the write set is identical to #190. Do not dispatch a competing Worker. After the current #192 exact-SHA Reviewer completes, the same #190 Worker will add the bounded outer watchdog, detached-descendant/process-tree cleanup, inherited-pipe regression, and explicit manual-window behavior. #196 closes only after the resulting new candidate passes its fresh strict review and authoritative host evidence.

  3. NOirBRight commented on Jul 22, 2026

    @NOirBRight
    OwnerAuthor

    2026-07-22 operational deadlock correction: this issue must not wait indefinitely for the exact failure mode it is intended to fix. Reviewer 3bff423c for PR #192 was bounded to an incremental diff/spec review using the already-green exact-SHA Actions evidence, with no further full/grouped pytest runs and a 20-minute hard stop. If that attempt fails or times out, #196 implementation may proceed in the existing #190 Workspace rather than waiting for another unbounded Reviewer. Reviewer orchestration/SLA is tracked separately in #201; this issue remains responsible only for E2E wall-clock, inherited-pipe handling, and full process-tree cleanup.

  4. NOirBRight commented on Jul 22, 2026

    @NOirBRight
    OwnerAuthor

    Completed in PR #192 and squash-merged to dev as e424f65. The merged runner includes bounded wall-clock supervision, Job Object process-tree ownership, bounded cleanup, and the reviewed argv-boundary fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Priority 1: current iteration goalbugSomething isn't workingtier:heavyOrchestrator provider tier: heavywayfinder:taskWayfinder implementation task

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions