fix(opal-server): broadcaster reader liveness — silence watchdog for a half-open LISTEN after a DB failover

[Zivxx]

Scope change (round 3): the TCP keepalive half of this PR moved to the library that owns the socket — permitio/broadcaster#28 (permit-broadcaster 0.2.7) enables SO_KEEPALIVE / TCP_KEEPIDLE 30 / TCP_KEEPINTVL 10 / TCP_KEEPCNT 3 on every pooled asyncpg connection, on by default and tunable with the libpq-style keepalives* query parameters of the broadcast URI. The in-OPAL hook (broadcaster_keepalive.py, the BROADCAST_TCP_KEEPALIVE_* keys, the module-name patching) is gone. This PR is now the reader silence watchdog + its metrics + the hard terminate() of the dead listening connection before release — the backend-agnostic detector, and the only one that also catches a peer whose kernel still answers probes while its backbone delivers nothing. It adds two config keys (OPAL_BROADCAST_KEEPALIVE_INTERVAL 3600→60, OPAL_BROADCAST_READER_SILENCE_TIMEOUT 180) and two metrics (opal_server.broadcaster_reader_silent, opal_server.broadcaster_silence_trips). The dependency bump to 0.2.7 is a separate one-line PR once the library release is on PyPI.

fix(opal-server): broadcaster reader liveness — a half-open backbone connection must not leave a worker deaf

What happened (the finding)

During a staging campaign the broadcaster Postgres (AWS RDS, Multi-AZ) was failed over. The failover itself took ~40 s and the database name moved to the standby host. Every worker's broadcaster reader — the long-lived LISTEN connection through which it hears what other workers publish — kept its TCP socket to the old primary in state ESTABLISHED: the old host was taken down without a FIN/RST, and a listening socket never writes, so nothing on the client side ever noticed. Result, for hours, on every worker whose reader had been connected at that moment:

• the reader task waited forever on a dead socket — no reconnect, no resync;
• is_reader_healthy() stayed True (the task was alive and pending), /healthcheck 200;
• the worker's WebSocket clients stayed connected and silently stopped receiving every update published via other workers (one client received 4 updates in 9 hours instead of ~1 per minute; a fleet-wide integrity probe went from 60/60 to ~25/60 and stayed there until the pods were restarted).

The existing reconnect + resync path handles announced disconnects (the peer closes, the read errors) — a DB stop/start, a pod restart. It had no way to notice a silent death. Two independent detectors close that: TCP keepalive at the socket (library, 0.2.7 — catches an unreachable peer within ~60 s) and, in this PR, an application-level silence watchdog (catches any backbone that stops delivering, whatever the reason, on any backend).

How

1. Reader liveness by silence (ReconnectingBroadcaster). The reader records the monotonic time of the last backbone message of any kind — own keepalives included, they prove the pipe — stamped before and after the event is handled (fan-out time is not silence). Arming: the clock is unarmed until this process has heard the backbone at least once (a slow boot, a single pod whose leader has not published yet, a keepalive publisher still starting can never trip it); from then on every subscription is armed at subscribe time, with a first-message grace of max(timeout, 2×keepalive) — so a session that goes half-open before its first message (a post-failover reconnect landing on a stale endpoint, i.e. this PR's own scenario one step earlier) is caught too, not a blind spot. While reading, the subscriber is polled through a small _WatchedIterator that keeps one pending __anext__ across polls and wakes every second to compare against BROADCAST_READER_SILENCE_TIMEOUT (default 180 s) — it never cancels the in-flight __anext__ on a poll, because the broadcaster library's subscriber is an async generator and cancelling it closes it for good (that was the round-1 bug: wait_for per poll reconnected every second against the real library; the fake now has the same trap and a real-backend repro pins it). Silence beyond the timeout (with a 60 s floor between trips) raises BackboneSilent, which the reader loop treats like any failed session: gap generation bumped, the dead listening connection terminated (asyncpg terminate() — so the pool cannot hand the half-open socket back to the very next connect, and the reconnect path never runs UNLISTEN *; RESET ALL; on a socket that will never answer), backoff, reconnect, and on re-subscribe the normal gap recovery (replay buffer + client resync). The keepalive publish interval goes from 3600 s to 60 s so a healthy fleet is never silent for long, and the leader starts publishing it before load_scopes (which can outlast the timeout on a big fleet); effective_reader_silence_timeout() keeps the two coupled — watchdog off when the keepalive is off or the publisher is disabled (a quiet channel is not a dead one), and the timeout is raised to 2× the keepalive interval if set lower so one late keepalive can never trip it.
2. Health and observability: a silence trip does not flip /healthcheck. While the reader re-subscribes it is a pending task mid-reconnect — exactly the state a peer-announced disconnect leaves it in — and readiness must keep the worker so its clients can reconnect and fetch bundles/data (which need no backbone); the escalation for a backbone that never comes back is the existing give-up → graceful-shutdown path. Instead the state is visible: gauge opal_server.broadcaster_reader_silent{pid} (1 while tripped and not yet re-subscribed) and counter opal_server.broadcaster_silence_trips{pid}. The gauge is re-emitted from /healthcheck next to broadcaster_reader_healthy so it has a steady 0 baseline. A dashboard/alert on the counter is the follow-up in the deployments repo. Pre-existing gap property, unchanged: a message that arrives in the same tick as a trip is dropped with the session and reconciled by the resync that follows the reconnect.

Will this trip on the tiniest hiccup?

No — the watchdog fires only on a peer that has stopped delivering for minutes, and a trip costs exactly what today's clean-disconnect path costs (one reconnect, one client resync, with the existing settle + jittered staggering):

detector	fires when	cost of a false trip
TCP keepalive (library, 0.2.7)	the peer answers no keepalive probe for 30 + 10×3 = ~60 s (a working peer answers in microseconds; a blip that recovers within the window does not fire)	one reconnect + resync
silence watchdog (this PR)	no message of any kind for 180 s on a fleet that heartbeats every 60 s — i.e. ≥3 consecutive keepalives lost; floored at 2× the keepalive interval; at most one trip per 60 s	one reconnect + resync

A real backbone outage of any length already takes the reconnect + resync path today; this only adds the case where the outage is silent.

Config (conservative defaults)

key	default	meaning
OPAL_BROADCAST_KEEPALIVE_INTERVAL	60 (was 3600)	heartbeat every pod's leader publishes on the backbone; the silence watchdog listens for it; 0 disables both
OPAL_BROADCAST_READER_SILENCE_TIMEOUT	180.0	seconds of total backbone silence before the listener is treated as dead; raised to 2× keepalive if lower; 0 disables

TCP keepalive itself is configured on the broadcast URI (library, 0.2.7): postgres://…/db?keepalives=1&keepalives_idle=30&keepalives_interval=10&keepalives_count=3 — on by default with those values, keepalives=0 to disable.

Tests

opal_server/tests/broadcaster_reader_liveness_test.py (17 tests, in-memory backbone, no Postgres): a silent backbone trips the watchdog and reconnects exactly once (gap generation +1, task still pending, healthy throughout, broadcaster_reader_silent 1→0, trips counter +1, no re-trip while the new session is unarmed); no trip before the first message however long the backbone is quiet, across resubscribes too; a later session (after the process has heard the backbone) is armed at subscribe and a silent one trips exactly once after the first-message grace; the grace ends with the session's first message; the grace helper is ≥ 2× keepalive; a short silence / steady traffic / watchdog disabled / backbone-down-from-boot never trip; a handler slower than the timeout is not silence; the trip floor prevents a storm; the dead listening connection is terminated before release and NOT on a clean close; the leader starts the keepalive publisher before load_scopes; config defaults and the keepalive/timeout coupling (incl. publisher disabled). The shared _FakeSubscriber is now a real async generator (same cancellation trap as the library's); the 44 pre-existing reconnect tests are unchanged and pass.

Suite 324 → 342, all green (the 6 keepalive-hook tests left with the hook). Mutation-checked against the liveness tests — each of these edits is caught: watchdog never trips; stamp only before the handler; clock armed at subscribe instead of first message; health flips on silence (must not); no terminate before pool release; cancelling the pending __anext__ on a poll timeout (the round-1 bug); removing the heard-once latch; arming every subscribe regardless of the latch; ignoring the first-message grace; never ending the grace. Three repros against the real broadcaster memory backend (not the fake): 3 s of a healthy-but-quiet backbone with a 30 s timeout → connects=1, gap_generation=0; a real silent trip → trips=1, reconnect, delivery resumes, healthy=True; an event whose fan-out takes 1.5 s then 2.2 s of genuine silence against a 1 s timeout → exactly one trip. pre-commit (black/isort/codespell/docformatter at the pinned versions) clean.

Not in this PR / follow-ups

• permit-broadcaster pin bump 0.2.6 → 0.2.7 (one line) once the library release is on PyPI.
• A docker-bed gate for the half-open case (docker network disconnect the DB without stopping it, assert recovery within ~2 min and that a 10 s blip does not resync). The bed's existing postgres-bounce test covers the announced-disconnect case only.
• Operationally, until both ship: after any failover/maintenance of the broadcaster database, rolling-restart opal-server.

Release note

opal-server: broadcaster reader liveness — a silence watchdog on the backbone reader (keepalive heartbeat every 60 s, 180 s of silence → terminate the dead listener, reconnect + resync); new metrics broadcaster_reader_silent / broadcaster_silence_trips; /healthcheck is unchanged. Together with permit-broadcaster 0.2.7 (TCP keepalive on the Postgres connections) this fixes workers silently missing fleet updates after a broadcaster database failover.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Nxg7e1Jtk5qykAdYoYENdw

[netlify]

✅ Deploy Preview for opal-docs ready!

Name	Link
🔨 Latest commit	708257b
🔍 Latest deploy log	https://app.netlify.com/projects/opal-docs/deploys/6a85ef67f5a6da0008b0b110
😎 Deploy Preview	https://deploy-preview-946--opal-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.