Skip to content

Dedupe batch completion across both code paths - #177

Open
lovitt wants to merge 1 commit into
keypup-io:masterfrom
castolonco:fix/batch-completion-double-fire
Open

Dedupe batch completion across both code paths#177
lovitt wants to merge 1 commit into
keypup-io:masterfrom
castolonco:fix/batch-completion-double-fire

Conversation

@lovitt

@lovitt lovitt commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

The bug

on_batch_complete can be triggered from two code paths that may observe a batch as complete at the same time:

  • on_child_complete — the last child to finish notifies the parent.
  • complete — the parent's own post-setup backstop, run at the end of execute.

Only on_child_complete claimed the batch_completion_gid SETNX token before firing on_complete; complete fired it directly. When both observe complete? == true concurrently, on_batch_complete could fire twice.

The fix

Extract the guarded fire into trigger_completion(status) and route both paths through it, so a concurrent backstop and child no longer both fire.

on_batch_node_complete stays outside the guard: it's a distinct per-node event that complete fires once on its own.

Scope and limitations

The SETNX guard dedupes the common case: two paths racing to complete a batch that's still in flight. It does not handle Cloud Tasks delivering the same job twice — an uncommon at-least-once redelivery. A job that already completed can arrive after cleanup has wiped the batch state and the token, re-create a phantom slot, and re-fire on_batch_complete. Cloudtasker runs a job every time Cloud Tasks delivers it and never checks whether that job already ran, so workers needing exactly-once completion should keep perform and on_batch_complete idempotent.

Related

Overlaps with #175 in on_child_complete. They fix independent bugs and compose cleanly, but they touch the same lines, so whichever lands first, the other needs a small rebase.

on_batch_complete can be triggered from two code paths that may observe a
batch as complete at the same time: the last child (on_child_complete) and
the parent's own post-setup backstop (complete). Only on_child_complete
claimed the batch_completion_gid SETNX token before firing on_complete;
complete fired it directly. When both observe complete? == true
concurrently, on_batch_complete could fire twice.

Extract the guarded fire into trigger_completion (SETNX -> on_complete,
clearing the key on error so completion can be reattempted) and route
both paths through it, so a concurrent backstop and child no longer both
fire. on_batch_node_complete stays outside the guard: it is a distinct
per-node event that complete fires once on its own.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant