Forum: JRuby JRuby forkoff replacement?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Shot (Piotr S.) (Guest)
on 2008-12-04 22:38
(Received via mailing list)
In my MRI-targeted code I use the forkoff¹ gem, which adds the
Enumerable#forkoff method that’s basically a #map running its &block’s
contents in separate processes.

¹ http://codeforpeople.com/lib/ruby/forkoff/forkoff-...

In my code, I incorporated #forkoff like this:

module Enumerable
  def parallel &block
    case ArtDecomp::Conf.processes
    when 1 then map(&block)
    else forkoff ArtDecomp::Conf.processes, &block
    end
  end
end

Trying to run #parallel with Conf.processes set to 2 in JRuby gives me
the (understandable) ‘fork is unsafe and disabled by default on JRuby’
error.

What would be the JRuby’s equivalent? In other words:
how would one run such code in parallel properly in JRuby?

Confession 1: I know I should figure the above by myself, but I haven’t
yet dug into JRuby, I’m just evaluating its purpose for my case (PhD
computations) – I need to run my code faster, and I want to see how
fast I can go with JRuby before I give up on pure-Ruby speed and go
the one-way road of RubyInline…

Confession 2: My knowledge about threads and processes is murky to say
the least. All I know is that MRI’s global interpreter lock means two
‘parallel’ threads aren’t really parallel and won’t use multiple cores
while two processes will; also, IIRC, JRuby maps threads to JVM’s
processes and so JRuby threads are really parallel – if that’s true,
I’m ok with running my code in threads under JRuby.

Confession 3: I should be using DRb for the above anyway, and that’s the
plan for the future, but for now I need code that utilises two cores in
full.

-- Shot
Charles Oliver N. (Guest)
on 2008-12-04 23:49
(Received via mailing list)
Shot (Piotr S.) wrote:
> What would be the JRuby’s equivalent? In other words:
> how would one run such code in parallel properly in JRuby?

Just use threads..see below.

> Confession 1: I know I should figure the above by myself, but I haven’t
> yet dug into JRuby, I’m just evaluating its purpose for my case (PhD
> computations) – I need to run my code faster, and I want to see how
> fast I can go with JRuby before I give up on pure-Ruby speed and go
> the one-way road of RubyInline…

You should strongly consider using Java before using RubyInline, since
it's going to be easier to maintain, less prone to C's various
fatalities, just as fast, and trivially callable from JRuby. And if you
are really interested in inlining that code, there's also a java_inline
module I wrote that plugs into RubyInline:

http://projectkenai.com/projects/java-inline

It's very primitive so far, but works pretty well. See the examples. I'm
looking for others to help improve it, if it's desired.

> Confession 2: My knowledge about threads and processes is murky to say
> the least. All I know is that MRI’s global interpreter lock means two
> ‘parallel’ threads aren’t really parallel and won’t use multiple cores
> while two processes will; also, IIRC, JRuby maps threads to JVM’s
> processes and so JRuby threads are really parallel – if that’s true,
> I’m ok with running my code in threads under JRuby.

This is exactly how you should do it. You should take care to protect
common data structures, but under JRuby threads actually do run in
parallel and you can achieve what forkoff does without without using
fork.

I think Ara also released another gem that trivially parallizes using
threads, but the name escapes me. I'm sure it's under codeforpeople
though.

- Charlie

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
Shot (Piotr S.) (Guest)
on 2008-12-05 01:33
(Received via mailing list)
Charles Oliver N.:

> Shot (Piotr S.) wrote:

>> What would be the JRuby’s equivalent? In other words:
>> how would one run such code in parallel properly in JRuby?

> Just use threads..see below.

Ok, thanks.

> You should strongly consider using Java before using RubyInline,
> since it's going to be easier to maintain, less prone to C's
> various fatalities, just as fast, and trivially callable from
> JRuby.

Hm. On one hand, I know a bit of C and almost zero Java, and my
assumption is that in the long run the C solution would be the fastest.
On the other, my gut feeling is that learning Java (and its connections
to JRuby, which, from my quick look at them, are rather intuitive) in
my particular case would be faster and more fun than relearning C with
its VALUE pointers, memory management and MRI’s bindings. On the third,
I’ve yet to try RubyInline+RubyToC combo, but I guess it either works as
advertised (and then there’s no C to learn) or it doesn’t (and I’m back
in C land).

I’m curious about the ‘just as fast’ claim above – my research code
ended up being mostly bitwise operations on variable-length integers
(which are lovely to work with in Ruby due to the Fixnum/Bignum
transparent switching). My base class is a glorified Set of Integers
– hence my assumptions that I might be able to learn a small enough
subset of MRI C API to implement what I need via RubyInline in
a relatively short period of time.

I have no idea about JVM’s performance other than the most popular
myths(?) from both sides of the discussion, but the cases that require
the optimisations are the ones that will end up being computed for hours
or even days, so I can at least safely skip any warm-up disadvantage.

Does JVM indeed approach C speed in this type of computations in my
above case (Sets of Integers, long running code)? I’m just looking
for a ballpark quote, of course – something like ‘most probably yes’,
‘maybe’ or ‘no way, JVM will be at least X times slower’.

> And if you are really interested in inlining that code, there's
> also a java_inline module I wrote that plugs into RubyInline

Cool! I wasn’t aware of it. If I have the time, I might end up
implementing the bottlenecks in both and then switch on runtime
based on the interpreter. :]

I tried a quick google, but it’s surprisingly hard to find if you don’t
know the right words; if you’re replying anyway – what was the constant
that I can use to differentiate between MRI and JRuby?

(Hm, one of the disadvantages of going with JRuby is that my profiling
knowledge is limited to ruby-prof, but then if I learned ruby-prof on
an afternoon then maybe I can learn the basics of a Java profiler in
a weekend.)

> I think Ara also released another gem that trivially parallizes using
> threads, but the name escapes me. I'm sure it's under codeforpeople
> though.

Ah, right, how could I forget! It’s threadify. :)

Also, my crude benchmarks are just in: running an actual, small
‘production’ case with one process took 28m57s on MRI 1.8.6.p287,
12m18s on MRI 1.9.1-preview2 and 9m20s on JRuby 1.1.5. Hmmm. :)

Thanks a lot for your reply! It’s much appreciated.

-- Shot
Shot (Piotr S.) (Guest)
on 2008-12-05 11:54
(Received via mailing list)
Shot (Piotr S.):

> I tried a quick google, but it’s surprisingly hard to find if you
> don’t know the right words; if you’re replying anyway – what was
> the constant that I can use to differentiate between MRI and JRuby?

Ah, RUBY_PLATFORM and RUBY_ENGINE. :)

-- Shot
Vladimir S. (Guest)
on 2008-12-05 12:01
(Received via mailing list)
On Fri, Dec 5, 2008 at 11:00 AM, Shot (Piotr S.) 
<removed_email_address@domain.invalid>
wrote:
> Shot (Piotr S.):
>
>> I tried a quick google, but it's surprisingly hard to find if you
>> don't know the right words; if you're replying anyway – what was
>> the constant that I can use to differentiate between MRI and JRuby?
>
> Ah, RUBY_PLATFORM and RUBY_ENGINE. :)

There is also JRUBY_VERSION, which, if defined, with 100% probability
detects JRuby :)

--Vladimir

---------------------------------------------------------------------
To unsubscribe from this list, please visit:

    http://xircles.codehaus.org/manage_email
This topic is locked and can not be replied to.