Skip to content

Recover interrupted workflows and reject stale writes - #354

Open
eshulman2 wants to merge 6 commits into
mainfrom
fix/architecture-reliability
Open

eshulman2 wants to merge 6 commits into
mainfrom
fix/architecture-reliability

Conversation

@eshulman2

@eshulman2 eshulman2 commented Oct 29, 2026 •

Copy link
Copy Markdown
Collaborator

Interrupted deliveries could remain stuck in Redis pending entries or be skipped because observation acceptance was mistaken for completed workflow execution. Expired ticket and effect owners could also keep writing, while incoming deliveries accumulated unbounded local tasks.

This change makes unfinished observations recoverable, rejects stale writes, and bounds delivery dispatch across new messages, pending recovery, and scheduled retries.

  • Resolve canonical ticket identity before locking, renew ownership, and fence Redis checkpoint writes atomically.
  • Persist observation processing and prepared command handoffs so redelivery resumes the original decision without reapplying its transition.
  • Reclaim abandoned stream entries, preserve stable retry identity/backoff, and make retry handoffs durable with bounded shutdown and capacity.
  • Renew effect claims, reject stale completions, cancel blocking Git subprocesses, and retain ambiguous provider outcomes for authenticated reconciliation.
  • Expose processing state and operational metrics, and document recovery and migration requirements.
  • Honor effect claim limits so the sweep never leases work it will not execute, and retain blocked observations (including legacy ambiguous deliveries) as ordering barriers until explicitly resolved.

The implementation separates ownership, effects, queue recovery, observation handoffs, and crash testing/documentation.

Validation:

  • Follow-up review fixes: 130 affected unit and Redis integration tests passed, including 10 new regression cases that failed before the fixes. All six subprocess crash-boundary cases also passed against a disposable Redis Stack instance using database 0; the initial database-3 run was rejected by Redis Search before execution.
  • Unit, workflow, contract, and flow suite: 3,530 passed. The final additional reconciliation API case passed in a targeted rerun (6 API tests).
  • Integration suite against local Redis Stack: 140 passed, 20 existing skips (process-crash matrix run separately).
  • Six real subprocess-termination tests passed across acceptance, preparation, checkpoint handoff, graph execution, graph completion, and finalization. Each command transition applied once; interrupted uncheckpointed station work may repeat.
  • End-to-end suite without external providers: 34 passed.
  • Ruff checks, source formatting, and git diff whitespace checks passed.

Deployment requires draining/stopping old workers before starting new workers; mixed protocol writers are unsafe. Preserve Redis data and classify legacy accepted observations without processing records as ambiguous rather than replaying them automatically. Local ownership cannot retract provider requests already in flight, and this does not promise exactly-once model execution. See docs/architecture/recovery.md and docs/operations.md.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant