Skip to content

Serialize DiagnosticControlBridge batches to prevent duplicate control execution #267

Closed

Description

@NOirBRight

Problem

DiagnosticControlBridge can execute one versioned control request more than once when its daemon thread and another caller enter process_once() concurrently.

start() launches _run(), whose first iteration invokes process_once() before waiting. Tests and supported in-process callers can also invoke process_once() explicitly. The batch path scans, executes, writes the response, and unlinks the request without a lock covering the whole operation; _io_lock protects only atomic response/status writes.

For a mark request, both callers can read the same request and call mark_incident() before either unlink completes. The later response then overwrites the first one: one logical command can freeze two incidents while reporting i000002.

Evidence

  • Exact dev-based full Python run for ci: stop caching multi-gigabyte Rust target directories #265 observed test_delete_and_status_are_deterministic_after_the_tail_freezes receive i000002 instead of i000001.
  • A focused single rerun passed, proving the window is timing-dependent rather than an assertion mismatch.
  • A barrier placed around the recorder call deterministically produced two concurrent process_once callers and response i000002 in 1/1 reproduction.
  • Ordinary focused repetition can miss the narrow window; absence in one CI run is not evidence of exactly-once behavior.

Outcome

A single DiagnosticControlBridge executes each accepted control file at most once even when daemon polling, deterministic/manual polling, status publication, and shutdown overlap.

Scope

  • Add one Bridge-owned serialization boundary covering the complete bounded batch: scan -> execute -> response -> unlink -> status publication/retention.
  • Preserve the existing single Gateway-child ownership boundary; do not introduce an unproved cross-process lock.
  • Keep shutdown bounded and avoid deadlock when recorder methods or status publication fail.
  • Cover mark, delete, pause/resume, malformed requests, and response retention under concurrent entry.

Non-goals

  • Do not change incident IDs, retry failed control operations, or weaken fail-closed request validation.
  • Do not paper over the race by changing the test expectation or adding a test retry.
  • Do not redesign the diagnostic recorder or its persisted artifact schema.

Acceptance criteria

  • A barrier-controlled two-caller regression proves one mark_incident call, one terminal response, and incident i000001 for one request.
  • Concurrent delete cannot execute twice or overwrite a successful response with a second false result.
  • Daemon start() plus explicit polling cannot duplicate pause/resume/status side effects.
  • Shutdown remains bounded while a batch owns the serialization boundary.
  • Malformed/unrecognized requests still fail closed and cannot block later valid work.
  • Focused concurrent replay passes 100/100 and the existing diagnostic-control suite passes 500 repeated iterations without retries.

Verification

Verification class: strict (concurrency and persisted diagnostic control).

  • focused barrier-controlled concurrency tests
  • repeated tests/test_diagnostic_control.py
  • Python core under Python 3.13
  • exact-SHA GitHub Actions
  • python scripts/report_quality_gates.py remains report-only

Relationship

Regression against the deterministic/concurrent-control contract delivered by closed #117. Keep this separate from #265, which owns only Rust dependency-cache stability.

Activity

  1. added
    ready-for-agentFully specified, ready for an AFK agent
    P1Priority 1: current iteration goal
    on Jul 30, 2026
  2. NOirBRight commented on Aug 1, 2026

    @NOirBRight
    OwnerAuthor

    This was generated by AI during triage.

    Agent Brief

    Category: bug
    Summary: Serialize each DiagnosticControlBridge batch so one control request has at-most-once effects inside one Gateway child.

    Current behavior:
    Daemon polling and explicit polling can enter a batch concurrently, execute the same request twice, and overwrite the first terminal response.

    Desired behavior:
    One bridge-owned boundary covers scan, execution, response publication, request removal, status publication, and response retention without weakening validation or bounded shutdown.

    Key interfaces:

    • DiagnosticControlBridge batch processing and its bridge-owned synchronization boundary.
    • Recorder mark/delete/pause/resume operations and terminal control responses.
    • Shutdown and malformed-request recovery.

    Acceptance criteria:

    • Barrier-controlled concurrent entry produces one side effect and one terminal response.
    • Mark, delete, pause, resume, malformed requests, retention, and shutdown are deterministic.
    • Focused repetitions and Python core pass without retries.
    • Production diagnostics remain content- and secret-safe.

    Out of scope:

    • Cross-process locking, recorder redesign, response-schema changes, or test retries.
  3. NOirBRight commented on Aug 2, 2026

    @NOirBRight
    OwnerAuthor

    Implemented and squash-merged via PR #321 at 9ed6355. Hosted CI / gate passed; local targeted, 100/100 concurrent replay, 500/500 diagnostic suite, and baseline-separated Python core evidence recorded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1Priority 1: current iteration goalbugSomething isn't workingcampaign:0.1.8-beta20.1.8 Beta2 generic tool compatibility campaignready-for-agentFully specified, ready for an AFK agent

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions