Dedupe batch completion across both code paths - #177
Open
lovitt wants to merge 1 commit into
Open
Conversation
on_batch_complete can be triggered from two code paths that may observe a batch as complete at the same time: the last child (on_child_complete) and the parent's own post-setup backstop (complete). Only on_child_complete claimed the batch_completion_gid SETNX token before firing on_complete; complete fired it directly. When both observe complete? == true concurrently, on_batch_complete could fire twice. Extract the guarded fire into trigger_completion (SETNX -> on_complete, clearing the key on error so completion can be reattempted) and route both paths through it, so a concurrent backstop and child no longer both fire. on_batch_node_complete stays outside the guard: it is a distinct per-node event that complete fires once on its own.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
on_batch_completecan be triggered from two code paths that may observe a batch as complete at the same time:on_child_complete— the last child to finish notifies the parent.complete— the parent's own post-setupbackstop, run at the end ofexecute.Only
on_child_completeclaimed thebatch_completion_gidSETNX token before firingon_complete;completefired it directly. When both observecomplete? == trueconcurrently,on_batch_completecould fire twice.The fix
Extract the guarded fire into
trigger_completion(status)and route both paths through it, so a concurrent backstop and child no longer both fire.on_batch_node_completestays outside the guard: it's a distinct per-node event thatcompletefires once on its own.Scope and limitations
The SETNX guard dedupes the common case: two paths racing to complete a batch that's still in flight. It does not handle Cloud Tasks delivering the same job twice — an uncommon at-least-once redelivery. A job that already completed can arrive after
cleanuphas wiped the batch state and the token, re-create a phantom slot, and re-fireon_batch_complete. Cloudtasker runs a job every time Cloud Tasks delivers it and never checks whether that job already ran, so workers needing exactly-once completion should keepperformandon_batch_completeidempotent.Related
Overlaps with #175 in
on_child_complete. They fix independent bugs and compose cleanly, but they touch the same lines, so whichever lands first, the other needs a small rebase.