Segmentation fault when used in parallel threads

[solher]

Hi,

When I try to use the Parallel gem to multi-thread my IO with the Linkedin API, I get a segmentation fault. Sometimes on "net/http.rb:920" and sometimes with "openssl/buffering.rb:326". I put the problem on the Linkedin gem because I don't get the error when I do the same with the Google API.
Do you have any idea where the problem could come from ?

Thanks

2014-11-20T10:21:27.010930+00:00 app[web.1]: App 104 stderr: /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:920: [BUG] Segmentation fault at 0x00000000000000
2014-11-20T10:21:27.010936+00:00 app[web.1]: App 104 stderr: ruby 2.1.3p242 (2014-09-19 revision 47630) [x86_64-linux]
2014-11-20T10:21:27.010938+00:00 app[web.1]: App 104 stderr:
2014-11-20T10:21:27.010945+00:00 app[web.1]: App 104 stderr: -- Control frame information -----------------------------------------------
2014-11-20T10:21:27.010947+00:00 app[web.1]: App 104 stderr: c:0027 p:---- s:0156 e:000155 CFUNC :connect
2014-11-20T10:21:27.010948+00:00 app[web.1]: App 104 stderr: c:0026 p:0008 s:0153 e:000152 BLOCK
2014-11-20T10:21:27.010950+00:00 app[web.1]: App 104 stderr: /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:920
2014-11-20T10:21:27.010952+00:00 app[web.1]: App 104 stderr: c:0025 p:0036 s:0151 E:001c30 BLOCK
2014-11-20T10:21:27.010984+00:00 app[web.1]: App 104 stderr: /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/timeout.rb:91 [FINISH]
2014-11-20T10:21:27.010986+00:00 app[web.1]: App 104 stderr: c:0024 p:----
2014-11-20T10:21:27.010990+00:00 app[web.1]: App 104 stderr: s:0146 e:000145 CFUNC :call
2014-11-20T10:21:27.010992+00:00 app[web.1]: App 104 stderr: c:0023
2014-11-20T10:21:27.010996+00:00 app[web.1]: App 104 stderr: p:0069 s:0142 E:001178 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/timeout.rb:101
2014-11-20T10:21:27.010997+00:00 app[web.1]: App 104 stderr:
2014-11-20T10:21:27.011001+00:00 app[web.1]: App 104 stderr: c:0022 p:0508
2014-11-20T10:21:27.011004+00:00 app[web.1]: App 104 stderr: s:0131 E:001508 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:920
2014-11-20T10:21:27.011008+00:00 app[web.1]: App 104 stderr: c:0021
2014-11-20T10:21:27.011011+00:00 app[web.1]: App 104 stderr: p:0007 s:0120 e:000119 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:863
2014-11-20T10:21:27.011015+00:00 app[web.1]: App 104 stderr:
2014-11-20T10:21:27.011016+00:00 app[web.1]: App 104 stderr: c:0020 p:0036
2014-11-20T10:21:27.011019+00:00 app[web.1]: App 104 stderr: s:0117 e:000116 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:852
2014-11-20T10:21:27.011023+00:00 app[web.1]: App 104 stderr: c:0019
2014-11-20T10:21:27.011026+00:00 app[web.1]: App 104 stderr: p:0017 s:0114 e:000113 METHOD /app/vendor/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1369
2014-11-20T10:21:27.011029+00:00 app[web.1]: App 104 stderr:
2014-11-20T10:21:27.011032+00:00 app[web.1]: App 104 stderr: c:0018 p:0124
2014-11-20T10:21:27.011036+00:00 app[web.1]: App 104 stderr: s:0107 e:000106 METHOD /app/vendor/bundle/ruby/2.1.0/gems/oauth-0.4.7/lib/oauth/consumer.rb:161
2014-11-20T10:21:27.011039+00:00 app[web.1]: App 104 stderr: c:0017
2014-11-20T10:21:27.011043+00:00 app[web.1]: App 104 stderr: p:0018 s:0093 e:000092 METHOD /app/vendor/bundle/ruby/2.1.0/gems/oauth-0.4.7/lib/oauth/tokens/consumer_token.rb:25

[hexgnu]

A segfault is something that is way out of my control.

To cause a segfault involves accessing memory that doesn't technically exist. I think that if you are seeing segfaults while using this gem it's more likely something to do with a misconfigured openssl implementation.

Since the linkedin gem doesn't write any C extensions it's very difficult to cause segfaults or VM crashes.

I'd try posting your issue in the ruby tracker and see what they have to say. Also if you could post more context here I might have some more pointers for fixing.

[solher]

Thanks a lot.
It seems also to me that it's more of a global problem with parallel execution in Rails. I'm experiencing quite a lot of problems with multithreading in Ruby (and so, in Rails). Crashes with Postgres, ActiveRecord, some gems... And I'm not even using JRuby or Rubinius.

So about this crash, here is my setup:

• Ruby 2.1.3
• Rails 4.2.0.beta2
• Parallel gem for the multithreading
• I tested it on Heroku and I got the same problem.

Here is the other error I randomly get:

App 1614 stderr: /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/openssl/buffering.rb:326: [BUG] Segmentation fault at 0x0007feb73c45a1
App 1614 stderr:
App 1614 stderr: ruby 2.1.3p242 (2014-09-19 revision 47630) [x86_64-darwin14.0]
App 1614 stderr:
App 1614 stderr: -- Crash Report log information --------------------------------------------
App 1614 stderr: See Crash Report log file under the one of following:
App 1614 stderr: * ~/Library/Logs/CrashReporter
App 1614 stderr: * /Library/Logs/CrashReporter
App 1614 stderr: * ~/Library/Logs/DiagnosticReports
App 1614 stderr: * /Library/Logs/DiagnosticReports
App 1614 stderr: for more details.
App 1614 stderr:
App 1614 stderr: -- Control frame information -----------------------------------------------
App 1614 stderr: c:0034 p:---- s:0184 e:000183 CFUNC :syswrite
App 1614 stderr: c:0033 p:0164 s:0180 e:000179 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/openssl/buffering.rb:326
App 1614 stderr: c:0032 p:0009 s:0171 e:000170 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/openssl/buffering.rb:344
App 1614 stderr: c:0031 p:0028 s:0167 e:000166 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/protocol.rb:211
App 1614 stderr: c:0030 p:0009 s:0162 e:000161 BLOCK /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/protocol.rb:185
App 1614 stderr: c:0029 p:0027 s:0160 e:000159 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/protocol.rb:202
App 1614 stderr: c:0028 p:0007
App 1614 stderr: s:0156 e:000155 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/protocol.rb:184
App 1614 stderr: c:0027 p:0047 s:0152 e:000151 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http/generic_request.rb:325
App 1614 stderr: c:0026 p:0144 s:0145 e:000144 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http/generic_request.rb:136
App 1614 stderr: c:0025 p:0021 s:0139 e:000138 BLOCK /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1406 [FINISH]
App 1614 stderr: c:0024 p:---- s:0137 e:000136 CFUNC :catch
App 1614 stderr: c:0023 p:0024 s:0133 e:000132 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1405
App 1614 stderr: c:0022 p:0061 s:0126 e:000125 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1378
App 1614 stderr: c:0021 p:0036 s:0119 e:000118 BLOCK /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1371
App 1614 stderr: c:0020 p:0044 s:0117 e:000116 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:853
App 1614 stderr: c:0019 p:0017 s:0114 e:000113 METHOD /Volumes/Data/Users/Solher/.rvm/rubies/ruby-2.1.3/lib/ruby/2.1.0/net/http.rb:1369
App 1614 stderr: c:0018 p:0124 s:0107 e:000106 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/oauth-0.4.7/lib/oauth/consumer.rb:161
App 1614 stderr: c:0017 p:0018 s:0093 e:000092 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/oauth-0.4.7/lib/oauth/tokens/consumer_token.rb:25
App 1614 stderr: c:0016 p:0075 s:0087 e:000086 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/oauth-0.4.7/lib/oauth/tokens/access_token.rb:12
App 1614 stderr: c:0015 p:0017 s:0078 e:000077 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/oauth-0.4.7/lib/oauth/tokens/access_token.rb:27
App 1614 stderr: c:0014 p:0037 s:0073 e:000072 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/linkedin-1.0.0/lib/linked_in/helpers/request.rb:15
App 1614 stderr: c:0013 p:0152 s:0067 e:000065 METHOD /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/linkedin-1.0.0/lib/linked_in/api/query_helpers.rb:27
App 1614 stderr: c:0012 p:0024 s:0058 e:000057 METHOD
App 1614 stderr: /Volumes/Data/Users/Solher/.rvm/gems/ruby-2.1.3/gems/linkedin-1.0.0/lib/linked_in/api/people.rb:22

[hundredwatt]

@solher Looks like the twilio gem had a similar issue: twilio/twilio-ruby#93

We put a global mutex around all calls to the twilio gem twilio/twilio-ruby#93 (comment)

[solher]

So the solution is to put linkedin gem calls in Mutex.new.synchronize {} ?
But, don't we loose the purpose of multithreading ?

[hexgnu]

The only other thing we could do is mutex around the net/http calls which seem to be the problem. When we merge the faraday changes then we can discuss maybe using other http libraries like typhoeus (or make it configurable) that might not have thread safety issues.

[solher]

Ok thanks a lot !

[hundredwatt]

@solher You can try the faraday connection now using the unmerged branch:

 gem 'linkedin', github: 'hexgnu/linkedin', branch: 'faraday_connection_updated'

If you do, please let us know how it works in #248

[solher]

It works perfectly ! No more seg fault.