fix(sqs): confirm FIFO drain with queue attributes before exit_on_empty - #123
Merged
Divyanshu Tiwari (divyanshu-tiwari) merged 4 commits intoSep 11, 2026
Merged
Conversation
Empty ReceiveMessage is not a drained FIFO queue when a sibling consumer or a dead container still holds group heads. Ask GetQueueAttributes before treating the queue as empty.
Divyanshu Tiwari (divyanshu-tiwari)
requested a review
from a team
as a code owner
September 11, 2026 06:53
Keep the FIFO drain fix without adding a test file to the PR.
…docs The interface existed only for tests that are not in this PR. Restore *qs.Client and keep the drain rule in one place.
Outstanding only tracks this process; the rest of the why is in the attribute check.
Divyanshu Tiwari (divyanshu-tiwari)
deleted the
fix/sqs-fifo-confirm-empty
branch
September 11, 2026 09:49
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
exit_on_emptytreated an emptyReceiveMessageas a drained queue whenever this process had no outstanding receipts. That is wrong when a sibling consumer, or a dead container still inside the visibility timeout, still holds group heads. Prod hits:pdp_walmart_priority_queue.fifo(walmart_us_dag) andcatalog_listings.fifo(catalog_listings_hourly).#116already skipped drain while this process holds receipts. That in-memory check does not survive an ECS restart, and it does not cover messages in flight elsewhere.exit_on_emptynow also callsGetQueueAttributesand only exits when visible, not-visible, and delayed counts are all present and0. Attribute errors, incomplete maps, and this process's outstanding receipts also refuse to exit. Standard queues are unchanged.Types of changes
Checklist
Test plan
go test ./internal/pkg/pipeline/task/sqs/ ./internal/pkg/pipeline/task/exit_on_emptykeeps polling whenApproximateNumberOfMessages/NotVisibleis non-zero (catalog listings restart, walmart multi-task)Queue is empty, exitingonce attributes are all0GetQueueAttributes