Skip to content

RUBY-3946 - Check connection back in when a load-balanced initial command fails - #3104

Merged
comandeo-mongo merged 6 commits into
mongodb:masterfrom
drymar:fix-lb-initial-command-connection-leak
Sep 16, 2026
Merged

comandeo-mongo merged 6 commits into
mongodb:masterfrom
drymar:fix-lb-initial-command-connection-leak

Conversation

@drymar

@drymar drymar commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Fixes https://jira.mongodb.org/browse/RUBY-3946

In load-balanced topology, cursor-returning operations check a connection out of the pool before executing the initial command so the cursor can retain it. When that command fails, no cursor exists to drain and check the connection back in, so the pool permanently loses a slot on every failure; a process accumulating such failures eventually cannot check out any connection (ConnectionCheckOutTimeout with the full pool checked out and nothing pending) and never recovers.

ChangeStream and Database#cursor_command already release the connection on failure (the former since #3041). This applies the same handling to the remaining initial-command sites:

  • find (Iterable)
  • aggregate (Aggregation)
  • mapReduce (MapReduce)
  • parallel scan getMore (Readable)
  • listCollections (Database::View)
  • listIndexes (Index::View)

Connections pinned to a transaction are left checked out, matching Database#cursor_command.

Reproduction:

pool = client.cluster.servers.first.pool
client[:foobar].find(_id: BSON::ObjectId.new).to_a # warm up
pool.available_count #=> 5 (baseline)
3.times { client[:foobar].aggregate([{ '$foo' => 1 }]).to_a rescue nil }
pool.available_count #=> 2
GC.start; sleep 12  # finalizers + reaper
pool.available_count #=> 2, slots never recovered

@drymar
drymar requested a review from a team as a code owner August 24, 2026 10:39
@drymar
drymar requested a review from comandeo-mongo August 24, 2026 10:39
Since the migration from mlaunch to drivers-tools (RUBY-3472) the
load-balanced variants ran as plain single-mongos sharded clusters and
all load-balanced specs were silently skipped:

- LOAD_BALANCED was not exported to the test scripts, so the haproxy
  install and the LOAD_BALANCER translation never ran.
- SINGLE_MONGOS forced the single-mongos.json orchestration file, which
  has no loadBalancerPort, overriding the load-balancer config that
  drivers-tools selects.
- haproxy was never started and the LB URIs were never exported.
- The test suite was started with TOPOLOGY=sharded_cluster, but
  spec_config enables load-balanced mode only for TOPOLOGY=load-balanced.

Export LOAD_BALANCED, keep the load-balancer orchestration file, start
and stop haproxy via drivers-tools run-load-balancer.sh, and point the
suite at the single-mongos LB URI with TOPOLOGY=load-balanced.
Re-enabling the load-balanced Evergreen configuration unmasks 8
pre-existing test failures that are unrelated to the connection-leak
fix and were dark since the drivers-tools migration (RUBY-3472). Skip
them with a reference to RUBY-3959, which tracks the real fixes:

- load_balancers sdam-error-handling (3): CMAP event reason casing
- load_balancers wait-queue-timeouts (2): checkout timeout on maxPoolSize=1
- transaction_pinning lb (2): pool-state expectations
- client_construction (1): LB deployment returns a serviceId

The unified runner gains a skip_descriptions option so the YAML-driven
tests can be skipped without editing the synced fixtures.
@comandeo-mongo
comandeo-mongo merged commit 6375dc6 into mongodb:master Sep 16, 2026
241 checks passed
@comandeo-mongo

Copy link
Copy Markdown
Contributor

@drymar Thank you for your contribution, great work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants