Forum: JRuby Benchmarks sorta slow, am I using right options?

Hi,

I just ran a bunch of JSON parsing/generating benchmarks with MRI, JRuby
1.6 and JRuby 1.7 in combination with a bunch of different JSON
libraries.

The JRuby results... well when I saw them I sorta got that look on my
face like when you're at a party, you take a swig of beer and then you
just shit yourself. Everyone around you gets this awkward look, you
look around and avoid eye contact but they know what you did, you know
what you did... oh... you're not relating to that one? Ok, "awkward" is
what I'm going for here. Awkward.

I ran JRuby with the following options:
    --1.9 --server -Xcompile.mode=FORCE --fast

I thought with those options pitting JRuby against MRI would be like
shooting fish in a barrel (or your girlfriend's fish tank) given how I
imagine JSON parsing/generation would work and all the magical
optimisations I *imagine* JRuby would be able to do in those cases.

So, should I be using different options?

Btw sorry if I have unrealistic expectations of JRuby. It _IS_ an
awesome product and I'm really looking forward to seeing how the 1.7
tree comes along.

Finally, benchmark results here if you want a looksee:
http://japgolly.blogspot.com.au/2012/04/ruby-json-...

Cheers guys.
David

Thanks for bringing this to the list! Comments below.

On Wed, Apr 18, 2012 at 12:52 PM, David Barri <lists@ruby-forum.com> 
wrote:
> what you did... oh... you're not relating to that one? Ok, "awkward" is
> what I'm going for here. Awkward.
>
> I ran JRuby with the following options:
>    --1.9 --server -Xcompile.mode=FORCE --fast

These options are fine, though with 1.6+ you don't really need --fast
(almost everything interesting it does became default in 1.6).

JVM version does make a difference; Oracle/OpenJDK 7 (1.7) is faster,
and in combination with JRuby 1.7 it can be even faster for running
Ruby code.

> I thought with those options pitting JRuby against MRI would be like
> shooting fish in a barrel (or your girlfriend's fish tank) given how I
> imagine JSON parsing/generation would work and all the magical
> optimisations I *imagine* JRuby would be able to do in those cases.

Comments on the individual JSON impls you tested...

YAJL:

Anything C extension-based is going to be slow on JRuby (and every
impl except MRI, for various reasons). We discourage using C exts and
our support for them in 1.6 was mostly an olive branch.

Oj:

Same statement as above, I would imagine. I'm amazed either of them
worked as well as they did :)

json-pure:

This is pure Ruby, so it's always going to suffer compared to native
code. However, JRuby 1.7 on Java 7 should do considerably better than
other Rubies, earlier JRubies, or earlier Javas. There are some cases
where JRuby + Java 7 results in *worse* performance than JRuby + Java
6; if that's the case here, I really want to know.

json-jruby:

At this point I believe you should just use the "json" gem, which
includes the json-jruby stuff, or the "json" 1.9.3 library in JRuby
master which is the same thing.

I have looked into the performance of the json gem a few times, and
more recent builds (last few months) should have some perf
improvements I made. Ultimately the remaining performance issue comes
down to problems with the generated json code we can't really do
anything about (since it's generated by Ragel) and really has nothing
to do with JRuby.

I'd love to see a port of Oj (or YAJL) to Java that we could connect
with directly, but I don't know of any plans for someone to do that.

The bottom line is that you're really testing a *library* and the
integration with it rather than individual impls here. MRI + C exts
can be very fast. JRuby + C exts is slow. MRI + Ruby code can be slow.
JRuby + Ruby code can be pretty fast. JRuby + Java libraries can be
fast, if *the libraries themselves are fast*.

I hope that clears things up. I'd love to see the JRuby JSON options
become competitive with the fastest MRI + C options, but it would take
some work.

- Charlie

Complete outsider's possibly dumb comment: Have you considered wrapping
a fast Java JSON library such as Jackson and providing that as a gem? I
wonder how that would compare to the other implementations?

On Wed, Apr 18, 2012 at 9:21 PM, Tim Fox <timvolpe@gmail.com> wrote:
> Complete outsider's possibly dumb comment: Have you considered wrapping a
> fast Java JSON library such as Jackson and providing that as a gem? I wonder
> how that would compare to the other implementations?
>

I have wrapped Jackson for JRuby.  On github it is called jrjackson.
There is also a ruby gem.

https://github.com/guyboertje/jrjackson

It is about a lot faster.

Davids benchmarks on my machine:

Writing
                             user            system       total 
real
jrjackson:            0.585000   0.000000   0.585000 (  0.585000)

Reading
                             user            system      total 
real
jrjackson:            0.621000   0.000000   0.621000 (  0.621000)

vs

1.9.3

Writing
                                user            system        total
         real
oj:                            2.910000   0.000000   2.910000 ( 
2.915350)
yajl:                         2.160000   0.000000   2.160000 ( 
2.166557)
json_gem:              2.190000   0.000000   2.190000 (  2.199133)
json_pure:              2.200000   0.000000   2.200000 (  2.204436)

Reading
                                 user            system        total
         real
oj:                            1.290000   0.000000   1.290000 ( 
1.291839)
yajl:                         2.130000   0.000000   2.130000 ( 
2.138048)
json_gem:              2.400000   0.020000   2.420000 (  2.418907)
json_pure:              2.360000   0.020000   2.380000 (  2.393626)

This is similar to my own benchmarks which use a much larger hash.

The main problem with jrjackson is that it only works with hashes or 
arrays.
I have not found a method to leverage jacksons ability to de/serialise
a ruby object as if it was a java bean or using annotations.

Jrjackson is good for boundary conversions to and from JSON.  It also
includes de/serialisation to Smile binary representations.

I developed it to pass hash data around in ZMQ payloads.   FWIW I
created a wrapper around the JNI based jzmq library too -
unsurprizingly called jrzmq in github.

Guy

On Thu, Apr 19, 2012 at 12:13 PM, Guy Boertje <guyboertje@gmail.com> 
wrote:
>
> It is about a lot faster.
>

Strange. I was curious, since I believe I could gain some milliseconds
on my json generation in my sinatra webapp.

So I downloaded jackson-all-1.9.6.jar to compare it with installed gem
json-1.5.3-java. I ripped a json somewhere, made it a hash and tried
this simple code. I did not try jrjackson, as I wanted to compare java
extensions' performances.

code1: json-1.5.3-java

a = {"name"=>{"first"=>"Joe", "last"=>"Sixpack"}, "gender"=>"MALE",
"verified"=>false, "userImage"=>"Rm9vYmFyIQ=="}
require 'json'
require 'benchmark'
Benchmark.bm do |x|
  x.report { 1000.times { a.to_json } }
end

      user     system      total        real
  0.019000   0.000000   0.019000 (  0.019000)

code2: jackson (maybe my code is not efficient enough? I'm opened to
improvements)

a = {"name"=>{"first"=>"Joe", "last"=>"Sixpack"}, "gender"=>"MALE",
"verified"=>false, "userImage"=>"Rm9vYmFyIQ=="}

require 'java'
java_import org.codehaus.jackson.JsonFactory
java_import org.codehaus.jackson.map.ObjectMapper
class Hash
  def to_json
    factory = JsonFactory.new
    mapper = ObjectMapper.new
    writer = java.io.StringWriter.new
    generator = factory.createJsonGenerator writer
    mapper.writeValue generator, self
    generator.close
    return writer.toString
  end
end
require 'benchmark'
Benchmark.bm do |x|
  x.report { 1000.times { a.to_json } }
end

      user     system      total        real
  3.295000   0.000000   3.295000 (  3.295000)

150x faster on this simple testcase to use json-1.5.3-java than using
jackson. If you've code improvements or if I made obvious mistakes,
thanks to let me know.

Feeling puzzled then by the original report and the latest inputs on 
jackson...

--
Christian

In Java benchmarks I always run them for an extended period of time
(i.e. minutes) to minimise indeterminacies due to gc, and also give time
for the jit to kick in.

I don't think 1000 iterations is anywhere near enough to get a reliable
figure.

I'm not saying that's explains the low results, but it's more of a
general point about conducting micro-benchmarks on the JVM.

On 19/04/12 11:36, Christian MICHON wrote:
>>
> code1: json-1.5.3-java
>    0.019000   0.000000   0.019000 (  0.019000)
> class Hash
> require 'benchmark'
>
> Feeling puzzled then by the original report and the latest inputs on jackson...
>


--
Tim Fox

Vert.x - effortless polyglot asynchronous application development
http://vertx.io
twitter:@timfox

Also... with Jackson creating an JsonFactory/ObjectMapper is an
expensive operation - they should be created once and reused between
iterations.

I also recommend using the method:

String str = mapper.writeValueAsString(obj);

to write the object as a string in one operation

On 19/04/12 11:36, Christian MICHON wrote:
>>
> code1: json-1.5.3-java
>    0.019000   0.000000   0.019000 (  0.019000)
> class Hash
> require 'benchmark'
>
> Feeling puzzled then by the original report and the latest inputs on jackson...
>


--
Tim Fox

Vert.x - effortless polyglot asynchronous application development
http://vertx.io
twitter:@timfox

On Thu, Apr 19, 2012 at 1:06 PM, Tim Fox <timvolpe@gmail.com> wrote:
> Also... with Jackson creating an JsonFactory/ObjectMapper is an expensive
> operation - they should be created once and reused between iterations.
>
> I also recommend using the method:
>
> String str = mapper.writeValueAsString(obj);
>
> to write the object as a string in one operation
>

Thanks Tim. I'll try these...

--
Christian

On Thu, Apr 19, 2012 at 1:18 PM, Christian MICHON
<christian.michon@gmail.com> wrote:
>
> Thanks Tim. I'll try these...
>

Good. Now the 2nd snippet looks like this:

a = {"name"=>{"first"=>"Joe", "last"=>"Sixpack"}, "gender"=>"MALE",
"verified"=>false, "userImage"=>"Rm9vYmFyIQ=="}
require 'java'
$mapper = org.codehaus.jackson.map.ObjectMapper.new # ugly I know...
class Hash
  def to_json
    $mapper.writeValueAsString self
  end
end
require 'benchmark'
Benchmark.bm do |x|
  x.report { 1000.times { a.to_json } }
end

I then increased to 10,000,000 for both snippets to have a decent 
comparison:

c:\jruby>jruby -b json1.rb (json 1.5.3 java)
      user     system      total        real
 23.013000   0.000000  23.013000 ( 23.013000)
Runtime: 23433 ms

c:\jruby>jruby -b json2.rb (jackson)
      user     system      total        real
 65.795000   0.000000  65.795000 ( 65.795000)
Runtime: 66123 ms

json 1.5.3 java is still faster... Seen on jruby 1.6.7
(ruby-1.8.7-p357) (2012-02-22 3e82bc8) (Java HotSpot(TM) Client VM
1.6.0_31) [Windows 7-x86-java]

--
Christian

Hmm I think this forum's email notification is broken.
Anyhoo, wow thanks for all the information guys!

Charlie's explanations made a lot of sense, and the JrJackson lib. WOW! 
It's ridiculously fast! This thing is good! I sent a pull request to 
multi_json. Wish I'd known about it before I did that benchmarking 
blogpost. Glad I know now though :)

Looking forward to JRuby 1.7.0 btw :D
What a solid product!