Skip to content

fix(temporal): an activity failure reports the attempt the server ran (#547) - #548

Merged
gplanchat merged 3 commits into
mainfrom
fix/temporal-activity-attempt
Sep 25, 2026
Merged

gplanchat merged 3 commits into
mainfrom
fix/temporal-activity-attempt

Conversation

@gplanchat

@gplanchat gplanchat commented Sep 25, 2026 •

Copy link
Copy Markdown
Owner

Closes #547.

Done when

  • Test first, parity: the same activity failing on its last of N attempts gives the same $e->attempt() on the journal backend and on Temporal.
    • AnActivityFailureReportsItsAttemptTest::testAFailureOnTheThirdAttemptReportsTheAttemptTheJournalReports builds the journal's own exception (ActivityFailureEventFactory then toThrowable(), attempt 3), and asserts the same attempt() and the same message on Temporal.
    • Red first: "Failed asserting that 1 is identical to 3".
  • Both ACTIVITY_TASK_FAILED and ACTIVITY_TASK_TIMED_OUT read the attempt.
    • A start-to-close timeout after attempt 2 reports 2, and was red first.
    • A schedule-to-start timeout, with no started event, keeps 1, the journal's value there.
  • Replay, handled per alice's decision (a). My first claim here ("divergence is by slot, not payload") was wrong: ExecutionContext::activity() refuses a payload divergence (ExecutionContext.php:113-119), as dave found.
    • testAnAttemptCopiedIntoAPayloadDivergesOnARunRecordedUnderTheOldReading pins the one affected pattern. A run recorded under the old reading (attempt 1), whose workflow copies $e->attempt() into a later payload, now fails replay loudly. With main's reader it does not throw.
    • testAPlainReplayOfARetriedThenFailedActivityDoesNotDiverge: a workflow that does not copy the attempt replays the same history unchanged.
    • testTheVersionedWayOutReplaysARunRecordedUnderTheOldReading: UPGRADE.md's snippet, a version('real-activity-attempt', …) change point that keeps attempt 1 for runs that started before it, replays the old history without divergence.
    • UPGRADE.md has a "Replay:" entry: who is affected, the error they would see, and the two ways out (drain, or the change point).
    • The user can still prefer option (b), gating per history, before merge.

What changes

TemporalExecutionHistory now reads ACTIVITY_TASK_STARTED and records each started event's attempt. Temporal writes only the last attempt's started event, so that is the attempt that failed. The failure and timeout branches look it up through their started_event_id, falling back to 1 when there is none.

Checks

composer cs:check, PHPUnit (2,355 tests), PHPStan and Psalm all pass locally.

Reviewer: dave.

…#547)

A failure or a timeout read on Temporal always said attempt 1, where the
journal backends report the attempt that failed, so $e->attempt() and the
exception's message depended on the backend. The history now records each
ActivityTaskStarted's attempt, and ACTIVITY_TASK_FAILED and
ACTIVITY_TASK_TIMED_OUT read it through their started_event_id. With no
started event (a schedule-to-start timeout) it stays 1, as on the journal.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
gplanchat added a commit that referenced this pull request Sep 25, 2026
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@gplanchat gplanchat left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by dave, at 9d11935d. Verdict: changes needed, 1 finding. The parity fix is right. The claim that replay is safe is not.

F1: in-flight Temporal runs that put $e->attempt() into a payload diverge after upgrading

The PR body argues that divergence is checked by slot and kind, not payload. It is checked on payload too. ExecutionContext::activity() calls refusePayloadDivergence() with activityPayloadForSlot() (src/Durable/ExecutionContext.php:113-119), and child workflows and Nexus go through the same rule. Before this PR, attempt() was always 1 on Temporal. After it, the same code computes the real attempt, so a run whose history recorded 1 in a payload now fails its replay.

Proved with a throwaway WorkflowTaskRunner probe (deleted). The workflow catches the failure of double and schedules greet('attempt-' . $e->attempt()). The history holds double started as attempt 3 and then failed, followed by greet scheduled with {"name":"attempt-1"}, as the code before this PR recorded it.

  • On this branch: WorkflowTaskFailure: Replay divergence at activity slot 1 of execution "wf": "greet" is still the same activity, but its payload changed at byte 17. History recorded {"name":"attempt-1"}, code scheduled {"name":"attempt-3"}.
  • With main's TemporalExecutionHistory: no divergence; the run waits on greet.

This is exactly the guarantee the review rules name (replay determinism), so it needs a decision rather than a merge as is. Options:

  • (a) Keep the change and document the break. Add an UPGRADE entry: on Temporal, a run in flight whose workflow passes $e->attempt() into an activity, child or Nexus payload diverges after the upgrade; drain those runs first, or branch on a version (versionForChangeId). Add a test pinning that a plain replay, where the attempt reaches no payload, does not diverge.
  • (b) Report the real attempt only to histories recorded after the upgrade. No marker exists for that today, so this would need a new one, which is heavier.

For 0.1.0-beta1 I lean (a), since the journal backends always reported the real attempt, but it is the project lead's call.

Checked and OK

  • Parity: the journal reports the real attempt, and this PR aligns Temporal. The new tests fail on main and pass here: third-attempt failure, start-to-close timeout after attempt 2, schedule-to-start keeping 1.
  • started_event_id: Temporal records only the last attempt's ActivityTaskStarted, and FAILED/TIMED_OUT point at it. Earlier attempts leave no event to miss, so an old history cannot carry a retried attempt without its started event.

Non-blocking

A schedule-to-close timeout that elapses while attempt N is still waiting in the queue has started_event_id = 0, so it reports attempt 1. ActivityTaskTimedOut carries no attempt, so nothing better is readable there. Worth one line in the docblock of attemptOf().

Gates

259 Temporal unit tests pass; PHPStan and Psalm are clean.

gplanchat and others added 2 commits September 25, 2026 23:46
…, review)

Three runner tests over a history where double(2) ran three attempts, failed,
and the workflow scheduled greet(...) under the old reading:
- copying $e->attempt() into greet's payload diverges on replay (it passes
  with main's reader, which still says attempt 1);
- a workflow that does not copy it replays with no divergence;
- UPGRADE.md's way out, a change point that keeps attempt 1 for runs that
  started before it, replays with no divergence.
The attempt docblock also names the schedule-to-close case that reads 1.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…#547, review)

In-flight Temporal runs that put the attempt into a later payload diverge on
replay once the real attempt is read. The entry says who is affected, the
error they would see, and the two ways out: drain, or a change point that
keeps attempt 1 for runs that started before it, the snippet a test replays.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@gplanchat gplanchat left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-check by dave, at a899ec83. Verdict: OK. F1 is resolved by option (a), as the project lead ruled.

  • The three new runner tests pass. With main's TemporalExecutionHistory swapped in, exactly one of them fails, testAnAttemptCopiedIntoAPayloadDivergesOnARunRecordedUnderTheOldReading, so it pins the change. The plain-replay test and the versioned way out pass on both.
  • The UPGRADE "Replay:" entry names who is affected, the divergence message, and both ways out. Its snippet uses the real API (WorkflowEnvironment::version($changeId, ChangePoint::DEFAULT_VERSION, 1), the same pattern as the method's own docblock), and testTheVersionedWayOutReplaysARunRecordedUnderTheOldReading exercises it.
  • The docblock now names the schedule-to-close-while-queued case.

@gplanchat
gplanchat merged commit b32e9e3 into main Sep 25, 2026
38 checks passed
gplanchat added a commit that referenced this pull request Sep 25, 2026
…547)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task] On Temporal, an activity failure reports attempt 1 whatever attempt failed

1 participant