Postgres backend: TCP keepalive on pooled connections (detect half-open LISTEN after failover) — 0.2.7

[Zivxx]

What

The Postgres backend now enables TCP keepalive on every pooled asyncpg connection, on by default (idle 30 s / interval 10 s / count 3 → a silently vanished server is detected within ~60 s), tunable or disableable with the libpq parameter names in the URL:

postgres://user:pw@host:5432/db?keepalives=1&keepalives_idle=30&keepalives_interval=10&keepalives_count=3
postgres://user:pw@host:5432/db?keepalives=0   # off

The parameters are stripped before the DSN reaches asyncpg (asyncpg forwards unknown query params to the server as session settings, which would fail the connection). Version bumped to 0.2.7.

Why

A LISTEN connection is idle by nature: it sends nothing and waits for NOTIFYs. When the server's address goes silent without closing the socket — an RDS Multi-AZ failover, a network partition, a dead node — the client never receives a FIN/RST, the kernel keeps the socket ESTABLISHED forever, and the listener is deaf with no error and no event to react to. Publishers in the same process notice on their next pg_notify, reconnect by DNS name and land on the new primary — so the process ends up notifying the new primary while still listening on the old one. This is exactly what happened to opal-server on an RDS failover (details in the internal incident report).

libpq-based clients get SO_KEEPALIVE for free (keepalives=1 is the libpq default); asyncpg exposes no keepalive option at all — MagicStack/asyncpg#606 (open since 2020; the maintainer's suggested workaround is exactly this: set it on the transport socket) and #519. AWS's own guidance for Postgres failover is to turn TCP keepalives on, aggressively (Fast failover with Aurora PostgreSQL).

Scope: this only makes the dead connection error; reconnecting is, as before, the caller's job (the subscriber gets Unsubscribed, the termination listener fires). It does not cover a peer whose kernel is alive but whose Postgres is mute — TCP keepalive cannot see that; an application-level heartbeat is the tool for that case.

Verification

Isolated lab: postgres:16 + this backend on one docker bridge network, the server cut with docker network disconnect (the address goes silent, like a failover — unlike docker stop/kill, which send FIN/RST and were always detected in ~1 s):

backend	subscriber learns the connection died after
0.2.6	never (socket still ESTABLISHED at +10 min)
this PR, defaults 30/10/3	61.8 s
this PR, ?keepalives_idle=5&keepalives_interval=2&keepalives_count=3	12.3 s

Tests (tests/test_postgres_keepalive.py, 11 tests): URL parsing/stripping/defaults/keepalives=0/non-integer → ValueError; apply_tcp_keepalive on a real TCP socket (asserts getsockopt), never raises on a bad socket; end to end against the test Postgres: pooled connections carry the options by default, honour URL params, and keepalives=0 leaves the socket alone. Ran locally on Python 3.11 against postgres:16, plus test_memory/test_postgres from the existing suite.

Notes for reviewers

• getsockopt(SO_KEEPALIVE) returns 8 on macOS and 1 on Linux — tests assert non-zero.
• The tests reset PostgresBackend._pools around each test: the class-level pool cache is bound to the event loop that created it and pytest-asyncio gives each test its own loop (pre-existing; surfaced by having more than one Postgres test).
• .venv/ added to .gitignore.

Release

After merge: tag 0.2.7 → publish.yml builds and uploads (scripts/publish, needs the PYPI_TOKEN secret; 0.2.6 appears to have been uploaded by hand in Dec 2024 — worth confirming the secret exists before relying on the workflow). Then bump permit-broadcaster[postgres,redis,kafka]==0.2.7 in OPAL.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Nxg7e1Jtk5qykAdYoYENdw