Ruby and threading

Hello,

Is the current ruby 1.9.2 multithreaded at the OS level?

Regards,

Carter.

On Sun, Oct 16, 2011 at 3:50 AM, Carter C. [email protected]
wrote:

Hello,

Is the current ruby 1.9.2 multithreaded at the OS level?

Regards,

Carter.

For MRI, yes, but it has a global interpreter lock. Here’s a blog that
explains all the nuances.
http://www.engineyard.com/blog/2011/ruby-concurrency-and-you/

Hi Josh,

Thanks for the reply. I was looking over the YARV implementation in
1.9.3rc1
and did notice some support for pthreads and ruby threads (and some
description in the comments of certain models). Do you know if (or does
anyone else) if this code is fully implemented at present? Or is it a
global
lock like situation?

Regards,

Carter.

Hey Carter-

MRI will not be removing the GIL any time soon. For true concurrency,
you should use JRuby or Rubinius.

-Steve

Is there any big projects using jRuby? It seems that jRuby is available
for a long time, but still has not very much attention.

JRuby is being used at several companies including LinkedIn and Square.
It
also powers the HBase console.

I’ve done some consulting work, and JRuby was being used heavily.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 10/16/11 1:57 PM, Steve K. wrote:

MRI will not be removing the GIL any time soon. For true
concurrency, you should use JRuby or Rubinius.

or MacRuby.


All the best, Sandor Sz

On Sun, Oct 16, 2011 at 7:59 PM, Josh C. [email protected]
wrote:

and
then their new portion was written in Rails, and they were coexisting via
JRuby.

In London there are a few big name companies following this MO.

They wrap up a legacy Java app with Ruby based acceptance tests
(cucumber/capybara is popular), and then extend the app with Ruby based
extensions (no new production code with Java).

JRuby glues up a lot of it - companies have put a lot of investment in a
Java-based operations backend. JRuby lets them keep it. Its just the JSP
grunge that they want to be rid of.

I gather then the 1.9.2. implementation at present does not “exploit” OS
level threads (where as JRuby borrows them from the underlying JVM in
some
manner)?

On Sun, Oct 16, 2011 at 12:09 PM, Alexey P. [email protected]
wrote:

Is there any big projects using jRuby? It seems that jRuby is available
for a long time, but still has not very much attention.

Obtiva (who is now owned by Groupon) gave a JRuby workshop at Red Dirt
Ruby
Conf. The presenters, Tyler J., and Noel Rappin, said that they
were
using JRuby on a large client site, which was initially written in Java,
and
then their new portion was written in Rails, and they were coexisting
via
JRuby.

On Mon, Oct 17, 2011 at 2:47 AM, Carter C. [email protected]
wrote:

I gather then the 1.9.2. implementation at present does not “exploit” OS
level threads (where as JRuby borrows them from the underlying JVM in some
manner)?

You gather incorrectly. Ruby 1.9.2 does use underlying OS threads. For
a variety of reasons, however, there is a Global Interpreter Lock as
Josh C. pointed out to you (and provided this link:
http://www.engineyard.com/blog/2011/ruby-concurrency-and-you/).

-a

Maybe it’s a stupid question, but I can’t get it - what’s the point of
using OS threads with GIL?

You’ll anyway never get situation where two thread runs simultaneously,
so, what’s the point of it?

On Mon, Oct 17, 2011 at 1:02 PM, Alexey P. [email protected]
wrote:

Maybe it’s a stupid question, but I can’t get it - what’s the point of
using OS threads with GIL?

You’ll anyway never get situation where two thread runs simultaneously,
so, what’s the point of it?

Threads can still run simultaneously, they just can’t make changes to
Ruby
objects or the Ruby environment.

A native extension can release the GIL and do blocking I/O or perform a
complex computation (e.g. crypto) while Ruby code is running in another
thread.

On Mon, Oct 17, 2011 at 3:02 PM, Alexey P. [email protected]
wrote:

Maybe it’s a stupid question, but I can’t get it - what’s the point of
using OS threads with GIL?

You’ll anyway never get situation where two thread runs simultaneously,
so, what’s the point of it?


Posted via http://www.ruby-forum.com/.

“MRI 1.9 uses the same technique as MRI 1.8 to improve the situation,
namely
the GIL is released if a Thread is waiting on an external event
(normally
IO) which improves responsiveness.”

http://www.engineyard.com/blog/2011/ruby-concurrency-and-you/

On Mon, Oct 17, 2011 at 10:02 PM, Alexey P. [email protected]
wrote:

Maybe it’s a stupid question, but I can’t get it - what’s the point of
using OS threads with GIL?

You’ll anyway never get situation where two thread runs simultaneously,
so, what’s the point of it?

No, with a GIL no two threads can concurrently hold the lock. But a
thread which does not need the lock can execute in parallel (e.g.
while doing a syscall). For specifics you would have to ask Matz or
read the source.

Kind regards

robert

On Sun, Oct 16, 2011 at 4:03 AM, Josh C. [email protected]
wrote:

For MRI, yes, but it has a global interpreter lock. Here’s a blog that
explains all the nuances.
http://www.engineyard.com/blog/2011/ruby-concurrency-and-you/

A great followup to this post, explains why the GIL exists
http://merbist.com/2011/10/18/data-safety-and-gil-removal/

When I ran the code Matt provides under MRI 1.9.3 (has GIL) and
Rubinius,
JRuby, MacRuby (native threads, no GIL):

$ rvm 1.9.3-rc1,rbx-2.0.0pre,jruby-1.6.4,macruby-0.10 do ruby
needs_gil.rb

ruby-1.9.3-rc1
0.064 seconds
400000 elements in array (should be 400000)

rbx-2.0.0pre
0.232 seconds
398877 elements in array (should be 400000)

jruby-1.6.4
0.069 seconds
398709 elements in array (should be 400000)

macruby-0.10
0.076 seconds
366231 elements in array (should be 400000)

$ cat needs_gil.rb
puts ‘’, ENV[‘RUBY_VERSION’]

@array, threads = [], []
start = Time.now
4.times do
threads << Thread.new { (1…100_000).each {|n| @array << n} }
end
threads.each{|t| t.join }
stop = Time.now

puts “%0.3f seconds” % (stop - start), @array.size

Note:

  • Other times I ran it under Rubinius, the array got corrupted or
    something
    “Tuple::copy_from: index 8092 out of bounds for size 5395
    (Rubinius::ObjectBoundsExceededError)”
  • Other times I ran it under JRuby, it detected the corrupt data with
    ‘ConcurrencyError: Detected invalid array contents due to unsynchronized
    modifications with concurrent users’
  • I ran this a whole bunch of times, sometimes MRI was fastest,
    sometimes
    MacRuby, sometimes JRuby (MRI was fastest most consistently, though)

Thoughts:

  • MRI has a GIL, thus keeping the data safe, and still performs
    equivalently
    with other implementations (for this admittedly limited test), so do
    benchmarks to decide if this will be worthwhile. It’s not a fluke that
    Matz
    wants to keep the GIL.
  • I’m glad JRuby notices the corrupt data (though not always) I’m a big
    fan
    of fail-fast
  • Has JRuby fixed their startup time issue? I ran this a lot of times
    and
    didn’t notice any of the lag I used to.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 10/19/11 9:11 AM, Josh C. wrote:

$ cat needs_gil.rb puts ‘’, ENV[‘RUBY_VERSION’]

@array, threads = [], [] start = Time.now 4.times do threads <<
Thread.new { (1…100_000).each {|n| @array << n} } end
threads.each{|t| t.join } stop = Time.now

puts “%0.3f seconds” % (stop - start), @array.size

I think it’s pretty obvious, that implementations without GIL should
behave exactly as you have shown.
You did not use synchronization to append an item to your shared
object, so sometimes items will get lost.


All the best, Sandor Sz

On Wed, Oct 19, 2011 at 7:56 AM, Sandor Szcs
[email protected]wrote:

Thread.new { (1…100_000).each {|n| @array << n} } end
threads.each{|t| t.join } stop = Time.now

puts “%0.3f seconds” % (stop - start), @array.size

I think it’s pretty obvious, that implementations without GIL should
behave exactly as you have shown.
You did not use synchronization to append an item to your shared
object, so sometimes items will get lost.


That’s the point, to show why the GIL is useful.

It’s not useful, it’s hiding a problem: your code isn’t thead-safe.