pg-walstream

A high-performance Rust library for PostgreSQL logical and physical replication protocol parsing and streaming. Provides a robust, type-safe interface for consuming PostgreSQL Write-Ahead Log (WAL) streams.

Features

Full Logical Replication Support: Implements PostgreSQL logical replication protocol versions 1-4
Physical Replication Support: Stream raw WAL data for standby servers and PITR
Base Backup Support: Full BASE_BACKUP command with progress, compression, and manifest options
Pure-Rust Backend (default): The default rustls-tls backend needs no libpq and no OpenSSL, using aws-lc-rs for hardware-accelerated TLS (AES-NI, AVX2, SHA-NI).
TLS/SSL Support: All PostgreSQL SSL modes (disable, allow, prefer, require, verify-ca, verify-full)
Authentication: Cleartext, MD5, and SCRAM-SHA-256 authentication methods
Streaming Transactions: Support for streaming large transactions (protocol v2+)
Two-Phase Commit: Prepared transaction support (protocol v3+)
Parallel Streaming: Multi-stream parallel replication (protocol v4+)
Zero-Copy Operations: Efficient buffer management using the bytes crate with drain-loop batch queue optimization. The libpq backend copies each COPY message into a reusable BytesMut buffer, then reference-counts it downstream as Bytes (no per-message heap allocation after warmup)
Thread-Safe LSN Tracking: Atomic LSN feedback for producer-consumer patterns
Connection Management: Built-in connection handling with exponential backoff retry logic
Type-Safe API: Strongly typed message parsing with comprehensive error handling
Typed Row Deserialization: Built-in serde deserializer maps WAL rows directly into user-defined Rust structs (numerics, bool, String, Option<T>, enums, bytes)
High-Level Consumption Ergonomics: ReplicationStreamConfig::builder(), an auto-acking EventStream::for_each_event, and a typed by-table WalRouter with an optional #[derive(WalTable)] layer (opt-in derive feature)
Managed Initial Snapshot: LogicalReplicationStream::snapshot() copies the published tables through the slot's exported snapshot and hands off to the stream with no gap and no duplicate window. The ordering invariant is enforced by the type system, and snapshot rows arrive as ordinary ChangeEvents, so one set of handlers (or one WalRouter) serves both the snapshot and the live stream
Bounded Replay: ReplicationStreamConfig::with_stop_at_lsn streams to a target LSN, delivers the crossing transaction in full, then ends cleanly with ReplicationError::StreamStopped
Raw XLogData Access: LogicalReplicationStream::next_raw_event yields the undecoded pgoutput payload plus WAL positions (RawXLogData) for consumers that bring their own decoder — keepalives, feedback, and cancellation still handled, no auto-ack
Replication Slot Management: Create, alter, read, and drop slots with full option support
Hot Standby Feedback: Send hot standby feedback messages for physical replication

Installation

Add this to your Cargo.toml:

[dependencies]
pg_walstream = "0.8"

By default, this uses the pure-Rust rustls-tls backend — no libpq and no OpenSSL, only cmake + a C compiler at build time (for aws-lc-rs).

To use the C libpq backend instead (bound by pq-sys; requires system libpq):

[dependencies]
pg_walstream = { version = "0.8", default-features = false, features = ["libpq"] }

If both backends are enabled, rustls-tls takes priority automatically.

Feature Flags

pg-walstream provides two connection backends plus a std toggle, all selected at compile time. rustls-tls is the default; libpq is opt-in. When both are enabled, rustls-tls takes priority:

Feature	Default	C Dependencies	Description
`std`	Yes	None	Standard library support. Disable with `default-features = false` for a `no_std` plus `alloc` parser-only build (no connection layer) that compiles for `wasm32-unknown-unknown` and embedded targets.
`libpq`	No	`libpq-dev` + OpenSSL	Opt-in. PostgreSQL's C client library via FFI, bound by `pq-sys` (pre-generated bindings — no libclang). Battle-tested. Enable with `--no-default-features --features libpq`. Implies `std`.
`rustls-tls`	Yes	`cmake`, `gcc` (build-time only)	Default. Pure-Rust implementation using `rustls` with `aws-lc-rs` crypto backend for hardware-accelerated TLS. No libpq, no OpenSSL, no runtime C dependencies. Takes priority when both backends are enabled. Implies `std`.
`derive`	No	None	Opt-in proc-macros (pull `syn`/`quote`) that bind a struct to a table — `#[derive(WalTable)] #[wal(table = "...")]` or the one-line attribute form `#[wal_table("...")]` — enabling the `WalRouter::on_insert_of::<T>` / `on_update_of::<T>` / `on_delete_of::<T>` table-inference methods.

Note: The protocol parser, encoder, and types need no backend. Building with default-features = false gives a no_std plus alloc build of just those, suitable for wasm and embedded. A connection backend (libpq or rustls-tls) is required only for the live streaming and connection APIs, and pulls in std.

System Dependencies

System dependencies are only required for the opt-in libpq feature. The default rustls-tls backend requires only cmake and a C compiler at build time (for the aws-lc-rs crypto library), with no runtime dependencies.

For `libpq` feature (opt-in)

Ubuntu/Debian:

sudo apt-get install libpq-dev libssl-dev

CentOS/RHEL/Fedora:

sudo yum install postgresql-devel
# or
sudo dnf install postgresql-devel

For `rustls-tls` feature

Requires cmake and a C compiler at build time for aws-lc-rs (hardware-accelerated cryptography):

Ubuntu/Debian:

sudo apt-get install cmake gcc

Then add to Cargo.toml:

pg_walstream = { version = "0.8", features = ["rustls-tls"] }

TLS trust store

When sslmode is verify-ca or verify-full, the rustls-tls backend builds its root certificate store as follows:

If sslrootcert is set, it loads only those CAs from the PEM file (exclusive).
Otherwise, it loads the Mozilla CA bundle shipped via webpki-roots.

The OS trust store is not consulted. If your PostgreSQL server is signed by a corporate/internal CA that is only present in the OS trust store (e.g. /etc/ssl/certs), you must point sslrootcert at that CA explicitly — for example:

postgresql://user:pass@host/db?sslmode=verify-full&sslrootcert=/etc/ssl/certs/corporate-ca.pem

Quick Start

The examples/ directory contains runnable examples demonstrating various usage patterns:

Example	Description
`basic-streaming`	High-level `futures::Stream` API with stream combinators (`filter`, `take_while`)
`initial-snapshot`	Copy existing rows, then stream changes with no gap and no duplicate — `stream.snapshot()`
`polling`	Manual polling loop using `next_event()` for custom integration scenarios
`safe-transaction-consumer`	Production-grade transaction-aware CDC consumer with ordered commits and safe LSN feedback
`rate-limited-streaming`	Rate-limited consumption using `tokio_stream::StreamExt::throttle`
`tokio-spawn-streaming`	Producer/consumer pattern via `tokio::spawn` with `mpsc` channel (demonstrates `Send` safety)
`typed-deserialization`	Map INSERT/UPDATE/DELETE events directly into user-defined Rust structs via `serde`
`derive-router`	`#[derive(WalTable)]` + `WalRouter` table-inference (`on_*_of::<T>`) — the `derive` feature
`pg-basebackup`	Full physical backup tool using `BASE_BACKUP` with tar extraction and progress reporting
`binary-column-access`	Lossless zero-copy `BYTEA` access via `PgResult::get_bytes`, contrasted with the lossy `get_value`
`raw-xlogdata`	Undecoded pgoutput payload via `next_raw_event()` — bring your own decoder
`arbitrary-fuzzing`	Property-based fuzzing of all protocol types using the `arbitrary` crate

For more control, you can use the traditional polling approach:

use pg_walstream::{
    LogicalReplicationStream, ReplicationStreamConfig, RetryConfig, StreamingMode,
    SharedLsnFeedback, CancellationToken,
};
use std::sync::Arc;
use std::time::Duration;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ReplicationStreamConfig::new(
        "my_slot".to_string(),
        "my_publication".to_string(),
        2, StreamingMode::On,
        Duration::from_secs(10),
        Duration::from_secs(30),
        Duration::from_secs(60),
        RetryConfig::default(),
    );

    let mut stream = LogicalReplicationStream::new(
        "postgresql://postgres:password@localhost:5432/mydb?replication=database",
        config,
    ).await?;

    stream.start(None).await?;

    let cancel_token = CancellationToken::new();

    // Traditional polling loop with automatic retry
    loop {
        match stream.next_event_with_retry(&cancel_token).await {
            Ok(event) => {
                println!("Received event: {:?}", event);
                stream.shared_lsn_feedback.update_applied_lsn(event.lsn.value());
            }
            Err(e) if matches!(e, pg_walstream::ReplicationError::Cancelled(_)) => {
                println!("Cancelled, shutting down gracefully");
                break;
            }
            Err(e) => {
                eprintln!("Error: {}", e);
                break;
            }
        }
    }

    Ok(())
}

Mapping Columns to Struct Fields

Rows are keyed by their real PostgreSQL column names. When a struct field name differs from its column, use serde's #[serde(rename = "...")] — no extra attribute is needed:

use serde::Deserialize;

#[derive(Debug, Deserialize)]
struct User {
    id: i64,
    #[serde(rename = "user_name")] // field `username` ← column `user_name`
    username: String,
    #[serde(rename = "mail")]      // field `email`    ← column `mail`
    email: Option<String>,         // nullable column → Option
}

// let user: User = event.deserialize_insert()?;   // or row.deserialize_into()?

With the derive feature, add #[wal_table("...")] (outermost attribute) to bind the type to its table for WalRouter; it composes with the renames above:

#[wal_table("typed_deser_users")]
#[derive(Debug, Deserialize)]
struct User { /* fields as above */ }

LSN Tracking

Thread-safe LSN tracking for feedback to PostgreSQL:

use pg_walstream::SharedLsnFeedback;
use std::sync::Arc;

let feedback = SharedLsnFeedback::new_shared();

// Producer thread: read LSN from feedback
let (flushed_lsn, applied_lsn) = feedback.get_feedback_lsn();

// Consumer thread: update LSN after processing
feedback.update_applied_lsn(commit_lsn);

Initial Snapshot

Streaming alone tells you what changed, never what was already there. Enable a managed initial snapshot and the library copies the published tables through the replication slot's exported snapshot, then hands off to the stream:

use pg_walstream::SnapshotOutcome;
use std::sync::Arc;

let config = ReplicationStreamConfig::builder("my_slot", "my_publication");

let stream = LogicalReplicationStream::new(conn_str, config).await?;
let sink = Arc::new(sink);

let mut stream = match stream.snapshot().await? {
    // The slot already existed, so there is nothing to copy — the normal
    // "resume an existing subscription" path. Not an error.
    SnapshotOutcome::Unavailable(stream) => stream,

    SnapshotOutcome::Available(snapshot) => {
        println!("copying {} table(s) at {}", snapshot.tables().len(), snapshot.consistent_point());
        snapshot
            .events()
            .run(|event| {
                let sink = Arc::clone(&sink);
                async move { sink.apply(event).await }
            })
            .await?
    }
};

// Resumes exactly at the snapshot's consistent point.
stream.start(None).await?;

snapshot() sets SNAPSHOT 'export' on the slot itself — there is no flag to remember and no builder-ordering rule to get wrong.

Snapshot rows arrive as ordinary ChangeEvent::Inserts carrying the slot's consistent point as their LSN, so an existing WalRouter works across both phases with the same handlers:

let mut router = WalRouter::new();
router.on_insert_of::<User, _>(|user| async move { upsert(user).await });

let mut stream = match stream.snapshot().await? {
    SnapshotOutcome::Unavailable(stream) => stream,
    SnapshotOutcome::Available(snapshot) => {
        let mut events = snapshot.events();
        router.run_snapshot(&mut events).await?;   // snapshot phase
        events.finish().await?
    }
};

stream.start(None).await?;
router.run(&mut stream.into_stream(token)).await?;  // live phase, same handlers

Use snapshot.rows() instead of .events() to iterate SnapshotRows (.data, .relation, .lsn) without the ChangeEvent wrapper. To copy only part of a large publication, bind the snapshot as mut and call snapshot.retain_tables(..) before .events()/.rows().

A complete, runnable version of the above — including a row inserted during the handoff window to prove it arrives exactly once — is in examples/initial-snapshot.

A failed snapshot cannot be resumed. The exported snapshot is REPEATABLE READ; continuing against an expired one would silently mix rows from two points in time. You own the retry policy; the library owns the retry start point, which is always the beginning. A failed or dropped snapshot handle makes a best-effort attempt to drop its slot — a failure to drop is logged, not returned — so a retry normally gets a fresh export instead of finding the slot present and silently proceeding with no baseline.

Holding a snapshot handle blocks VACUUM. The handle keeps a REPEATABLE READ transaction open, pinning the database's xmin — no dead tuple newer than the snapshot can be reclaimed, across the whole database, for as long as you hold it. Copy promptly.

Connection Timeouts

Three conninfo options bound three different things, and they are easy to mistake for one another:

Option	Unit	Bounds
`connect_timeout`	seconds	Only the TCP connect. On the native backend TLS negotiation and startup/auth are not covered; libpq bounds the whole attempt.
`keepalives_idle` / `_interval` / `_count`	seconds	An idle connection. Defaults `120`/`10`/`3` — roughly 150 s to notice a peer that vanished while nothing was in flight.
`tcp_user_timeout`	milliseconds	An unacknowledged write. This is the only one that bounds a blocking round-trip — including the `DROP_REPLICATION_SLOT` that snapshot cleanup runs from `Drop`. Linux-family only; `0` (the default) disables it.

If a stuck destructor or a wedged control-plane command matters to you, tcp_user_timeout is the knob — keepalives will not do it, because a connection waiting for a reply is not idle.

Set it deliberately. An aggressive tcp_user_timeout tears down healthy long-lived replication connections during ordinary transient stalls, which is usually worse than the delay it prevents.

PostgreSQL Setup

Before using this library, you need to configure PostgreSQL for replication:

1. Configure PostgreSQL

Edit postgresql.conf:

wal_level = logical
max_replication_slots = 4
max_wal_senders = 4

Restart PostgreSQL after making these changes.

2. Create a Publication

-- Create a publication for specific tables
CREATE PUBLICATION my_publication FOR TABLE users, orders;

-- Or publish all tables
CREATE PUBLICATION my_publication FOR ALL TABLES;

3. Create Replication User

-- Create a user with replication privileges
CREATE USER replication_user WITH REPLICATION PASSWORD 'secure_password';

-- Grant necessary permissions
GRANT SELECT ON ALL TABLES IN SCHEMA public TO replication_user;
GRANT USAGE ON SCHEMA public TO replication_user;

4. Replication Slot Options

The library provides full control over replication slot creation. The correct SQL syntax is automatically selected based on the connected PostgreSQL version:

PG14: Legacy positional keyword syntax (EXPORT_SNAPSHOT, NOEXPORT_SNAPSHOT, USE_SNAPSHOT, TWO_PHASE, RESERVE_WAL)
PG15+: Modern parenthesized options syntax ((SNAPSHOT 'export', TWO_PHASE true, ...))

Option	Description	PG Version
`temporary`	Temporary slot (not persisted to disk, dropped on disconnect)	14+
`two_phase`	Enable two-phase commit for logical slots	14+
`reserve_wal`	Reserve WAL immediately for physical slots	14+
`snapshot`	Snapshot behavior: `"export"`, `"use"`, or `"nothing"`	14+
`failover`	Enable slot synchronization to standbys for HA	16+

Note: If both two_phase and snapshot are set, two_phase takes priority. An option newer than the connected server is rejected client-side with the required version in the message, before any SQL is sent.

failover slots also need the standby configured for slot sync: sync_replication_slots = on, hot_standby_feedback = on, and a primary_conninfo that includes dbname (pg_basebackup -R omits it). After a promotion the consumer only changes its connection string — the slot is already there.

Message Types

The library supports all PostgreSQL logical replication message types:

Protocol Version 1 Messages

BEGIN: Transaction start
COMMIT: Transaction commit
ORIGIN: Replication origin
RELATION: Table schema definition
TYPE: Data type definition
INSERT: Row insertion
UPDATE: Row update
DELETE: Row deletion
TRUNCATE: Table truncation
MESSAGE: Generic message

Protocol Version 2+ Messages (Streaming)

STREAM_START: Streaming transaction start
STREAM_STOP: Streaming transaction segment end
STREAM_COMMIT: Streaming transaction commit
STREAM_ABORT: Streaming transaction abort

Protocol Version 3+ Messages (Two-Phase Commit)

BEGIN_PREPARE: Prepared transaction start
PREPARE: Transaction prepare
COMMIT_PREPARED: Commit prepared transaction
ROLLBACK_PREPARED: Rollback prepared transaction
STREAM_PREPARE: Stream prepare message

Architecture

┌──────────────────────────────────────────┐
│          Application Layer               │
│  (Your CDC / Replication Logic)          │
└──────────────┬───────────────────────────┘
               │
┌──────────────▼───────────────────────────┐
│    LogicalReplicationStream              │
│  - Connection management & retry         │
│  - Event processing & LSN feedback       │
│  - Managed initial snapshot + handoff    │
└──────────────┬───────────────────────────┘
               │
┌──────────────▼───────────────────────────┐
│  LogicalReplicationParser                │
│  - Protocol v1-v4 parsing                │
│  - Zero-copy message deserialization     │
│  - Streaming transaction support         │
└──────────────┬───────────────────────────┘
               │
┌──────────────▼───────────────────────────┐
│     PgReplicationConnection              │
│  ┌─────────────────┬──────────────────┐  │
│  │  libpq backend  │ rustls-tls       │  │
│  │  (C FFI)        │ (pure Rust)      │  │
│  │                 │                  │  │
│  │  pq-sys         │ rustls +         │  │
│  │                 │ aws-lc-rs +      │  │
│  │                 │ postgres-protocol│  │
│  └─────────────────┴──────────────────┘  │
│  Compile-time feature flag selection     │
└──────────────┬───────────────────────────┘
               │
┌──────────────▼───────────────────────────┐
│     BufferReader / BufferWriter          │
│  - Zero-copy operations (bytes crate)    │
│  - Binary protocol handling              │
│  - Drain-loop batch queue optimization   │
└──────────────────────────────────────────┘

Backend Comparison & Stress Test

Progressive writer concurrency ramp (16 – 192 writers) plus a fixed scenario set, comparing the two connection backends.

Backend A: rustls-tls (aws-lc-rs crypto)
Backend B: libpq (OpenSSL)

Test environment: an 8-vCPU Linux VM (TCP-tuned per Linux VM TCP Tuning: 64 MB buffers, BBR) streaming from a remote Azure PostgreSQL Flexible Server 18.6 across regions (~50 ms RTT). Each scenario ran 10 s warmup + 30 s measure. Process CPU/RSS reflect only the pg-walstream consumer — the write generator runs as a separate OS process.

The sslmode you measure under decides the result. Without TLS the two backends are a statistical tie. With TLS, rustls-tls uses ~2.2x less CPU for the same event rate. Always state the sslmode alongside any backend number.

1. CPU Efficiency (DML events/sec per 1% CPU)

The primary efficiency metric: DML events processed per 1% of consumer CPU. Higher is better.

Scenario	rustls plain	libpq plain	rustls TLS	libpq TLS	libpq CPU vs rustls (TLS)
Baseline	6,670	7,135	3,759	1,739	2.13x
Batch-100	2,080	1,940	1,633	996	1.61x
Batch-5000	7,992	7,126	4,764	1,789	2.63x
4-Writers	6,995	7,093	4,052	2,098	1.84x
Wide-20col	2,162	2,102	1,609	1,067	1.51x
Payload-2KB	2,279	2,165	1,611	1,104	1.46x
Mixed-DML	2,926	2,946	2,458	1,370	1.79x
Stress-16w	6,088	6,437	3,917	1,800	2.62x
Stress-32w	6,237	6,335	3,979	1,810	2.40x
Stress-48w	5,630	6,190	3,854	1,647	2.23x
Stress-64w	5,615	5,737	3,779	1,642	2.12x
Stress-96w	5,546	5,534	3,524	1,568	2.24x
Stress-128w	5,328	5,852	3,356	1,566	2.25x
Stress-192w	5,554	5,457	3,754	1,511	2.33x

Median over CPU-bound scenarios:

`sslmode`	rustls-tls	libpq	Ratio
`disable` (no TLS)	5,859	6,262	0.98x — tie
`require` (TLS)	3,817	1,693	2.22x rustls-tls

Without TLS, per-scenario ratios scatter 0.90x – 1.12x with no directional lean — noise. TLS efficiency retained: rustls-tls 66%, libpq 28%.

2. Why: OpenSSL per-call overhead, not crypto

perf on the consumer during 4-Writers with TLS enabled:

	libpq	rustls-tls
TLS library share of process CPU	20.79% (libcrypto + libssl)	12.48% (aws-lc-rs, static)
...of which is actual AES-GCM	0.84%	most of it

libpq spends ~20 points of CPU on TLS bookkeeping and under 1 point on real crypto. The hot OpenSSL symbols are all per-call setup — ERR_clear_error (1.67%, which libpq calls before every read), BIO_ctrl, EVP_CIPHER_CTX_ctrl. libpq.so itself is only 2.39%, so libpq's own code is cheap; the cost is the OpenSSL call sequence it performs per read. pg-walstream issues one PQgetCopyData per WAL message, so this scales with message count, not bytes.

Measured against libpq 18.6 / OpenSSL 3.0.2. A newer OpenSSL would likely narrow this gap but not close it, since the per-read error-stack clear is libpq's own code.

3. Throughput and Resources

Throughput here is bounded by the write generator and the ~50 ms link, not by the consumer — so both backends land in the same range. The difference is the CPU spent getting there.

Metric	rustls-tls	libpq
Peak DML events/sec (no TLS)	213,392	202,347
Peak DML events/sec (TLS)	161,151	156,813
CPU% at TLS peak	38.9	80.9
Avg RSS (MB)	15.7	17.0
P50 inter-event latency	1 µs	1 µs

RSS and latency show no backend advantage.

4. Choosing a backend

TLS link (any managed Postgres: Azure, RDS, Cloud SQL) → prefer the default rustls-tls; it needs roughly half the consumer CPU.
TLS terminated elsewhere (private network, sidecar/proxy) → the backends are a tie; choose on operational grounds (libpq needs libpq-dev at runtime; rustls-tls is pure Rust with native SCRAM).

For a detailed comparison across PostgreSQL 16 and 18 with different optimizations (binary mode, direct TLS, COPY protocol), see the Load Test Comparison Report.

Linux VM TCP Tuning for Production

When streaming WAL over high-latency links (e.g., cross-region Azure PostgreSQL), the default Linux TCP buffer sizes can become the throughput bottleneck. The kernel's default rmem_max of 208 KB limits the TCP receive window, which — combined with round-trip latency — caps throughput via the Bandwidth-Delay Product (BDP):

Recommended sysctl Settings

# --- TCP buffer sizes ---
# Allow up to 64 MB per-socket receive/send buffers (kernel will auto-tune within this ceiling)
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864

# TCP auto-tuning ranges: min / default / max (bytes)
# The kernel dynamically adjusts each socket's buffer within these bounds
net.ipv4.tcp_rmem = 4096 262144 67108864
net.ipv4.tcp_wmem = 4096 262144 67108864

# --- Congestion control ---
# BBR provides significantly better throughput than cubic on high-latency links
net.ipv4.tcp_congestion_control = bbr

# --- Packet backlog ---
# Increase the NIC receive queue (helps at high packet rates)
net.core.netdev_max_backlog = 5000

Apply immediately:

sudo sysctl --system

Why Each Parameter Matters

Parameter	Default	Recommended	Why
`rmem_max`	208 KB	64 MB	Caps TCP receive window; directly limits throughput on high-RTT links
`wmem_max`	208 KB	64 MB	Caps TCP send window; limits outbound throughput for feedback messages
`tcp_rmem` (max)	6 MB	64 MB	Per-socket auto-tuned receive buffer ceiling
`tcp_wmem` (max)	4 MB	64 MB	Per-socket auto-tuned send buffer ceiling
`tcp_congestion_control`	cubic	bbr	BBR reacts to actual bandwidth, not packet loss; better on cloud networks
`netdev_max_backlog`	1000	5000	Prevents packet drops under burst traffic at NIC level

Note: These settings affect all TCP connections on the VM, not just pg-walstream. The kernel auto-tunes actual buffer usage within the configured ceiling, so idle connections do not consume 64 MB each.

Limitations

Requires PostgreSQL 14 or later for full protocol support
Logical replication slot must be created before streaming
Binary protocol only (no text-based protocol support)
Requires replication permission for the database user
A reconnect that lands on a different cluster or a new timeline (failover, PITR) is rejected as a permanent error rather than resumed: an LSN is only meaningful within one cluster on one timeline. Compare the confirmed LSN against the switchpoint in TIMELINE_HISTORY <tli> to decide whether to resume or re-sync

Resources

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

This project is licensed under the BSD 3-Clause License.

Author

Daniel Shih (dog830228@gmail.com)

Name		Name	Last commit message	Last commit date
.claude		.claude
.github		.github
benches		benches
examples		examples
fuzz		fuzz
integration-tests		integration-tests
load-tests		load-tests
macros		macros
src		src
.gitignore		.gitignore
AGENTS.md		AGENTS.md
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
LOAD_TEST_COMPARISON.md		LOAD_TEST_COMPARISON.md
Makefile		Makefile
README.md		README.md
SECURITY.md		SECURITY.md
_typos.toml		_typos.toml
codecov.yml		codecov.yml

Latest commit

History

Folders and files

Repository files navigation

pg-walstream

Features

Installation

Feature Flags

System Dependencies

For libpq feature (opt-in)

For rustls-tls feature

TLS trust store

Quick Start

Mapping Columns to Struct Fields

LSN Tracking

Initial Snapshot

Connection Timeouts

PostgreSQL Setup

1. Configure PostgreSQL

2. Create a Publication

3. Create Replication User

4. Replication Slot Options

Message Types

Protocol Version 1 Messages

Protocol Version 2+ Messages (Streaming)

Protocol Version 3+ Messages (Two-Phase Commit)

Architecture

Backend Comparison & Stress Test

1. CPU Efficiency (DML events/sec per 1% CPU)

2. Why: OpenSSL per-call overhead, not crypto

3. Throughput and Resources

4. Choosing a backend

Linux VM TCP Tuning for Production

Recommended sysctl Settings

Why Each Parameter Matters

Limitations

Resources

Contributing

License

Author

About

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

Watchers

Forks

Releases

Used by

Contributors

Languages

For `libpq` feature (opt-in)

For `rustls-tls` feature