fix(ci): stabilize PostgreSQL concurrency tests by tee8z · Pull Request #31 · voltagecloud/buzz

tee8z · 2026-10-20T20:31:55Z

PostgreSQL CI failed when tests raced server cleanup, overlapped cluster-wide activity scans, or used wall-clock bounds sensitive to runner load.

Reserve all nextest workers for cluster-wide activity scans, including both floor-lock probes. Ordinary database tests retain eight-worker parallelism.
Wait up to five seconds for the usage-metrics lock to transfer after its owner disconnects. Keep the exclusive-owner assertion.
Observe the readiness query waiting on a server lock before cancelling it, then release the lock and check connection recovery.
Await rollback after deliberately rejected deletion updates before claiming the approved request with SKIP LOCKED.
Advance Tokio time by one reader acquisition budget and require the production route to have selected the writer before resuming real database I/O.

Validation:

All 894 PostgreSQL tests passed against disposable PostgreSQL 16 and Redis 7.
The three additional race fixes passed ten consecutive PostgreSQL test runs.
Mutation checks rejected both a retained usage-metrics lock and a second reader acquisition timeout. Both mutations were removed.
Rust formatting and buzz-db Clippy with warnings denied passed. The earlier workspace Clippy, PostgreSQL discovery and wrapper checks, and file-size checks passed.
just ci passed workspace Clippy and desktop TypeScript/lint, then stopped at Tauri Clippy because this Linux environment lacks glib-2.0.pc.

Stack: layer 4 of 24. Base: #25. Next: #33. The signed fix is propagated through all 20 dependent PRs.

Summary by CodeRabbit

Tests
- Improved reliability and determinism of database integration tests covering cancellation, locking, approval validation, concurrency, and recovery scenarios.
- Added stronger verification for interrupted or rejected database operations.
- Reduced interference by isolating cluster-wide tests from other database tests and ensuring resources are released before subsequent operations.
Documentation
- Clarified database test execution, worker reservation, and shared PostgreSQL resource behavior.

coderabbitai · 2026-10-20T20:32:03Z

Warning

Review limit reached

Run on-demand review

_{This review includes 6 billable files and costs up to $1.50.}

Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 4 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available. Your 53 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository: voltagecloud/buzz/.coderabbit.yaml
Review profile: CHILL
Plan: Essentials
Run ID: 8a01d0f3-95cd-45d7-ac11-3bc3f07d8b57

📥 Commits

Reviewing files that changed from the base of the PR and between 2b99682 and e3f8f9a.

📒 Files selected for processing: 6

.config/nextest.toml
crates/buzz-db/TESTING.md
crates/buzz-db/src/runtime/replica_fence.rs
crates/buzz-db/src/runtime/tests.rs
crates/buzz-db/src/store/deletion.rs
crates/buzz-db/src/store/usage.rs

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: voltagecloud/buzz/.coderabbit.yaml
Review profile: CHILL
Plan: Essentials
Run ID: 66325fb6-e156-4dde-b17b-d2c5f2f69040

📥 Commits

Reviewing files that changed from the base of the PR and between 33c958a and 91cf251.

📒 Files selected for processing: 6

.config/nextest.toml
crates/buzz-db/TESTING.md
crates/buzz-db/src/runtime/replica_fence.rs
crates/buzz-db/src/runtime/tests.rs
crates/buzz-db/src/store/deletion.rs
crates/buzz-db/src/store/usage.rs

Included review availability: This review used your included allowance. 0 included reviews remain after this review. Your included PR review attempts over the past 7 days set your current allowance at 3 reviews per hour.

📝 Walkthrough

The postgres-ci profile reserves all configured test workers for cluster-global tests. Documentation and test names reflect this rule. PostgreSQL integration tests now use deterministic cancellation, lock-release polling, and transaction rollback cleanup.

Changes: PostgreSQL test isolation

Layer / File(s)	Summary
Configure and identify cluster-global tests `.config/nextest.toml`, `crates/buzz-db/src/runtime/replica_fence.rs`, `crates/buzz-db/src/runtime/tests.rs`, `crates/buzz-db/TESTING.md`	The postgres-ci profile reserves all configured test threads for `cluster_global_` tests instead of using a single-thread group. The documentation and integration test names reflect this scope.
Make cancellation tests deterministic `crates/buzz-db/src/runtime/tests.rs`	Cancellation tests confirm pending pool waiters, use advisory locks to detect server-side query waits, and verify reader fallback after one acquire timeout.
Release locks before follow-up operations `crates/buzz-db/src/store/deletion.rs`, `crates/buzz-db/src/store/usage.rs`	Rejected approval mutations run in transactions and roll back after failure. The lock-release test polls for successor acquisition after session disconnect and drops the successor before cleanup.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Unblocks: 19 PRs

Suggested reviewers: thesentinel454

Merge Risk: ⚪ Minimal · up to 91cf2

No actionable merge-blocking risk remains in the reviewed changes. Merge after normal checks.

Pre-merge checks: 5

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title clearly and concisely describes the main change: stabilizing PostgreSQL concurrency tests and related CI races.
Docstring Coverage		Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 4 files. (2 skipped: 2 …
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 6 functions across 4 files. (2 skipped: 2 unsupported.)

✨ Finishing Touches

📝 Generate docstrings

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

tee8z · 2026-10-20T22:17:32Z

@coderabbitai full review

coderabbitai · 2026-10-20T22:17:40Z

⚠️ Action not completed

Pull request base or head changed.

tee8z · 2026-10-20T22:21:43Z

@coderabbitai full review

coderabbitai · 2026-10-20T22:22:07Z

✅ Action performed

Full review finished.

tee8z · 2026-10-20T23:54:31Z

@coderabbitai full review

coderabbitai · 2026-10-20T23:54:36Z

✅ Action performed

Full review finished.

tee8z · 2026-10-21T00:04:54Z

@thebrandonlucas please review this PR, layer 4 of 23 in the remote-agent stack. Assigned to @tee8z.

CodeRabbit's review has no outstanding actionable findings at 1bf528c393faa3e8d880c042d1e96b26a9ce450c. CI and runtime acceptance remain separate checks.

tee8z · 2026-10-21T01:20:59Z

@coderabbitai full review

coderabbitai · 2026-10-21T01:21:05Z

⚠️ Action not completed

Review rate limited.

Your included review limit is currently reached under our Fair Usage Limits Policy. Your current included review allowance is based on your included PR review attempts over the past 7 days. This review may still proceed through usage-based billing if eligible. Your next included review will be available in 34 minutes.

tee8z · 2026-10-21T01:56:36Z

@coderabbitai full review

coderabbitai · 2026-10-21T01:56:42Z

✅ Action performed

Full review finished.

tee8z · 2026-10-21T02:02:56Z

@thebrandonlucas please review this PR, layer 4 of 23 in the remote-agent stack. Assigned to @tee8z.

CodeRabbit's review has no outstanding actionable findings at 91cf2512e06428e0a50cdf52040a8da917622443. CI and runtime acceptance remain separate checks.

Reserve all nextest workers for cluster-wide activity scans, including the floor-lock probes. Wait for server-side cleanup when the usage-metrics leader disconnects. Synchronize readiness cancellation with a blocked server query and await rollback after rejected deletion updates before claiming rows. Drive the reader acquisition budget with Tokio's controlled clock instead of a wall-clock upper bound. Preserve exclusive ownership, cancellation recovery, exact inventory approval, and single-budget fallback assertions. Signed-off-by: Tee8z <tee8z@protonmail.com>

tee8z · 2026-10-21T13:16:24Z

CI note: this stack still hits flaky tests in code it does not change. When one of these fails here, the failure is unrelated to this PR, and a re-run usually passes.

buzz-cli client::retry_policy_tests::exhausted_connect_failures_return_network_retryable: a parallel test can take the freed port and answer with HTTP 405.
Desktop smoke E2E huddle-transcription.spec.ts and empty-edit-delete.spec.ts: upstream fix test: fix recurring CI flakes in desktop and Postgres lanes block/buzz#8343, backported in fix(ci): stabilize fork publishing and desktop smoke fixtures #24.
buzz-db cluster-wide replica_fence PostgreSQL tests: fixed in fix(ci): stabilize PostgreSQL concurrency tests #31 for the layers above it.
buzz-agent steer_rejected_on_run_id_mismatch: fixed in test(agent): wait for both responses in the steer run-id mismatch test #33 for the layers above it.
buzz-acp adapter-spawn tests that fail with Text file busy (ETXTBSY).

Summary jobs from cancelled runs also stay red until the replacement run finishes. A failure in this PR's own code gets a fix in this PR.

@thebrandonlucas it's fine to approve without CI fully passing.

tee8z force-pushed the fix/postgres-cluster-isolation branch from 3cc40b1 to 3bcd9b8 Compare October 20, 2026 21:46

tee8z changed the base branch from main to fix/buzz-security-baseline October 20, 2026 21:47

This was referenced Oct 20, 2026

fix(deps): scope security exceptions and patch desktop archives #25

Merged

feat(backend-kubernetes): add Pod options and identity policy #9

Merged

tee8z added this pull request to stack #32 October 20, 2026 21:51

tee8z force-pushed the fix/postgres-cluster-isolation branch from 3bcd9b8 to c205ddb Compare October 20, 2026 22:21

tee8z changed the title ~~fix(ci): isolate cluster-wide PostgreSQL tests~~ fix(ci): stabilize PostgreSQL test isolation and lock handoff Oct 20, 2026

tee8z self-assigned this Oct 20, 2026

tee8z force-pushed the fix/postgres-cluster-isolation branch from c205ddb to 1bf528c Compare October 20, 2026 23:33

tee8z changed the title ~~fix(ci): stabilize PostgreSQL test isolation and lock handoff~~ fix(ci): stabilize PostgreSQL concurrency tests Oct 20, 2026

tee8z marked this pull request as ready for review October 21, 2026 00:04

tee8z requested a review from thebrandonlucas October 21, 2026 00:04

tee8z force-pushed the fix/postgres-cluster-isolation branch from 1bf528c to d2c063e Compare October 21, 2026 00:20

tee8z force-pushed the fix/postgres-cluster-isolation branch from d2c063e to 91cf251 Compare October 21, 2026 00:49

tee8z added this pull request to stack #34 October 21, 2026 12:21

tee8z mentioned this pull request Oct 21, 2026

test(agent): wait for both responses in the steer run-id mismatch test #33

Merged

tee8z force-pushed the fix/postgres-cluster-isolation branch from 2b99682 to e3f8f9a Compare October 21, 2026 12:55

This was referenced Oct 21, 2026

fix(agents): retry transient executable conflicts #29

Merged

fix(ci): stabilize fork publishing and desktop smoke fixtures #24

Merged

tee8z added the human review wanted Request human review label Oct 21, 2026

thebrandonlucas approved these changes Oct 21, 2026

View reviewed changes

tee8z merged commit a270008 into main Oct 21, 2026
98 of 119 checks passed

tee8z deleted the fix/postgres-cluster-isolation branch October 21, 2026 13:27

Conversation

tee8z commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Uh oh!

tee8z commented Oct 20, 2026

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tee8z commented Oct 20, 2026

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tee8z commented Oct 20, 2026

Uh oh!

coderabbitai Bot commented Oct 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tee8z commented Oct 21, 2026

Uh oh!

tee8z commented Oct 21, 2026

Uh oh!

coderabbitai Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tee8z commented Oct 21, 2026

Uh oh!

coderabbitai Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tee8z commented Oct 21, 2026

Uh oh!

tee8z commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

tee8z commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 20, 2026 •

edited

Loading

coderabbitai Bot commented Oct 21, 2026 •

edited

Loading

coderabbitai Bot commented Oct 21, 2026 •

edited

Loading