Skip to content

fix(notifications): recover from a dead listener connection - #11

Merged
yordis merged 2 commits into
mainfrom
yordis/fix-notifications-resilience
Sep 15, 2026
Merged

yordis merged 2 commits into
mainfrom
yordis/fix-notifications-resilience

Conversation

@yordis

@yordis yordis commented Sep 14, 2026

Copy link
Copy Markdown
Member
  • An untrapped listener connection failure left the notifications pipeline silently dead, so appends stopped reaching subscribers with nothing in the logs pointing at the cause
  • The upstream version of this fix could never actually fire, since monitoring a registered name reports the name and node rather than the name itself, and it returned a shape the producer stage rejects

@cursor

cursor Bot commented Sep 14, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Changes core LISTEN/NOTIFY delivery and when events are read from storage; behavior is covered by new tests but affects all live event fan-out.

Overview
Fixes two notification pipeline issues: a dead PostgreSQL notifications connection could leave the listener running but never listening again, and every node was reading/deserializing notifications even when no local subscriber cared.

The Listener now monitors the listen_to process and stops with its exit reason when that connection dies, so supervision can restart a healthy listener; unrelated DOWN messages are ignored. The Publisher skips storage reads for notifications whose exact stream topic has no local subscribers via new PubSub.subscribed?/2, cutting wasted work in clustered setups while $all still gets its own notifications. Tests cover listener recovery after killing the notifications process, subscriber-gated reads, and persistent subscriptions catching up when a notification was skipped.

Reviewed by Cursor Bugbot for commit 8ea198e. Bugbot is set up for automated code reviews on this repo. Configure here.

@coderabbitai

coderabbitai Bot commented Sep 14, 2026

Copy link
Copy Markdown

Review Change StackReview Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: f0978248-1e68-4cf0-b585-e883c59d15b0

📥 Commits

Reviewing files that changed from the base of the PR and between f6a1fc6 and 8ea198e.

📒 Files selected for processing (5)
  • lib/event_store/notifications/listener.ex
  • lib/event_store/notifications/publisher.ex
  • lib/event_store/pubsub.ex
  • test/notifications/listener_test.exs
  • test/notifications/publisher_test.exs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


Walkthrough

The listener now monitors its notification connection and handles process termination messages. The publisher checks for active subscribers before reading events. New tests cover listener recovery, listener termination, subscriber filtering, and persistent subscription catch-up.

Changes

Notification changes

Layer / File(s) Summary
Subscriber-aware event publishing
lib/event_store/pubsub.ex, lib/event_store/notifications/publisher.ex, test/notifications/publisher_test.exs
PubSub.subscribed?/2 checks registry entries. Publisher skips notifications without subscribers before storage reads. Tests cover exact-stream and $all subscriptions, unregistering subscribers, and persistent catch-up.
Notification connection monitoring
lib/event_store/notifications/listener.ex, test/notifications/listener_test.exs
Listener stores a monitor reference for the notification connection. Matching :DOWN messages stop the producer, while unrelated messages are ignored. Tests cover connection replacement, listener termination, and unrelated process exits.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to d12cc

No actionable regression remains; the notification filtering retains $all delivery behavior.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 8 functions across 5 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the primary change: recovery from a dead notifications listener connection.
Description check ✅ Passed The description directly explains the listener failure and the corrections included in the changeset.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch yordis/fix-notifications-resilience

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

A rabbit watched the listener glow
And saw the connection restart and flow
Empty streams were left unread
While active paths sent events ahead
The registry kept the signals bright

Comment @coderabbitai help to get the list of available commands.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 622bf6b. Configure here.

Comment thread lib/event_store/notifications/listener.ex
satom99 and others added 2 commits September 15, 2026 10:23
(cherry picked from commit 97aa48402087e1c13189ba0ff635708643a6cdac)
Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
The monitor added for that purpose never matched, so recovery depended on an
unrelated crash path and surfaced a misleading exit reason.

Signed-off-by: Yordis Prieto <yordis.prieto@gmail.com>
@yordis yordis changed the title fix(notifications): recover from a dead listener and stop reading events nobody wants fix(notifications): recover from a dead listener connection Sep 15, 2026
@yordis
yordis force-pushed the yordis/fix-notifications-resilience branch from 8ea198e to d12cca0 Compare September 15, 2026 14:23
@yordis
yordis merged commit a1268b5 into main Sep 15, 2026
11 of 12 checks passed
@yordis
yordis deleted the yordis/fix-notifications-resilience branch September 15, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants