Stuck in Moped::Errors::ConnectionFailure: Could not connect to a primary node for replica set

[dblock]

Moped::Errors::ConnectionFailure: Could not connect to a primary node for replica set #<Moped::Cluster:128953180 @seeds=[<Moped::Node resolved_address="10.95.128.244:27017">, <Moped::Node resolved_address="10.184.156.102:27017">]>

…avity/ruby/2.0.0/gems/moped-2.0.3/lib/moped/cluster.rb: 254:in `with_primary'
…ty/ruby/2.0.0/gems/moped-2.0.3/lib/moped/collection.rb: 124:in `insert'
…by/2.0.0/gems/mongoid-4.0.0/lib/mongoid/query_cache.rb: 117:in `insert_with_clear_cache'
…ems/mongoid-4.0.0/lib/mongoid/persistable/creatable.rb:  79:in `insert_as_root'

Occasionally we see a machine or two stuck in this. I am not sure when this happens, but about 10% of nodes end up in this state every 24 hours. The MongoDB cluster is doing fine.

This issue could probably use more detail, please tell me what to look for next time I have a machine in this state.

[wandenberg]

Hi @dblock could you check if the code on #352 solve this problem?

[niedfelj]

@dblock Please see my PR #338
We were having these errors too, and I'm guessing that you are actually having a pool saturation problem and not primary node connection issues. In general, the logging in mongoid is pretty terrible. Are you running Puma? And have you tuned pool_size and pool_timeout?

[steve-rodriguez]

How do you go about tuning those? How do you know what to set them to? Are there guidelines?

[niedfelj]

In general, you should have a pool_size that is equal to or greater than the number threads you are running. You shouldn't need to tune pool_timeout. Here is an update submitted to mongoid for generating the mongoid.yml giving more details on those configs

https://github.com/mongoid/mongoid/pull/3883/files

[niedfelj]

These PRs might also be useful to you, in adding more/better logging in error situations and giving metrics on per request in rails:

https://github.com/mongoid/mongoid/pull/3885
https://github.com/mongoid/mongoid/pull/3884

[dblock]

#352 has so far been good to us in production (72 hours). So it has improved things I want to say.

[fedenusy]

I'm seeing this error as well.

[ajsharp]

+1. We see this a couple of times per day, seemingly on a random basis.

[wnkz]

+1 also seeing this.

[InvisibleMan]

I think, MOPED also use wrong thread-safe code.

#353 (comment)

[ajsharp]

Interesting. Does anyone see this behavior with unicorn? I've seen it with puma (threads), but don't have anything in production with unicorn.

Wondering if switching the app server to unicorn might be an easy "fix", because it seems like the real fix could take a bit of time.

[ajsharp]

@arthurnn any thoughts on this issue?

[InvisibleMan]

I'm using sidekiq gem and I have not choice.