Skip to content

fix: add NOT EXISTS guard to stop notification_id_seq burn on repeated job runs - #1005

Open
dylanjeffers wants to merge 2 commits into
mainfrom
fix/notification-seq-burn
Open

fix: add NOT EXISTS guard to stop notification_id_seq burn on repeated job runs#1005
dylanjeffers wants to merge 2 commits into
mainfrom
fix/notification-seq-burn

Conversation

@dylanjeffers

Copy link
Copy Markdown
Contributor

Problem

The notification_id_seq column hit INT max (2³¹−1) on July 29 because all four sub-steps in RemixContestNotificationsJob re-scan the same events on every 30-second tick and attempt duplicate INSERTs.

Postgres allocates a sequence ID before checking ON CONFLICT DO NOTHING, so every wasted attempt permanently burns an ID:

Step Window Re-scans/event/day Burn multiplier
fan_remix_contest_ending_soon 72 h 8,640 × N fan recipients
artist_remix_contest_ending_soon 48 h 5,760 × 1 (host only)
fan_remix_contest_ended 24 h 2,880 × N fan recipients
artist_remix_contest_ended 24 h 2,880 × 1 (host only)

July 29 spike: The Summer Cypher contest (1.97 M followers) entered the 72-hour ending-soon window. Each 30-second tick attempted 1.97 M INSERTs. After ~543 ticks (~4.5 h), ~1.07 B sequence IDs were burned — exhausting INT range and taking down notifications for all users.

Chronic baseline (~18 M IDs/day burned): Smaller active contests (combined fan audiences on the order of a few thousand) still burned millions of IDs daily through the same mechanism.

Fix

Add an event-level NOT EXISTS guard to each of the four SQL blocks:

AND NOT EXISTS (
    SELECT 1 FROM notification
    WHERE group_id = 'fan_remix_contest_ending_soon:' || e.event_id::text
)

Once any notification row exists for an event (i.e., after the first successful fan-out pass), subsequent job runs skip that event entirely. The per-recipient ON CONFLICT DO NOTHING is kept as a safety net for partial-failure retries.

Effect: Each contest triggers at most one fan-out pass instead of thousands. Sequence consumption drops from O(recipients × job_runs_in_window) to O(recipients × 1) per event.

What this does NOT change

  • No user-visible notification is dropped. The guard only prevents re-attempting INSERTs that would conflict anyway.
  • The bigint migration already deployed by Ray stays in place as the unblock. This PR stops the source of future exhaustion.

Testing

Existing TestRemixContest_Ended already asserts idempotency (runs the job twice and verifies the count stays the same). fanEndingSoon idempotency is implicitly covered: after the first run the NOT EXISTS check short-circuits, so no inserts are attempted on re-runs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant