Forum: JRuby jruby in production

in the upcoming jruby-rubinius-battle [1] i would like to cite some
high-profile uses of jruby in production. are there any, apart from
the "success stories" [2] in the github wiki?

please step forward :)

[1] http://wrocloverb.com/
[2] https://github.com/jruby/jruby/wiki/SuccessStories

--
http://tim.lossen.de

That page could definitely use some updating.

Square runs most of their backend services on JRuby, and seems intent
on moving everything to JRuby.

LinkedIn has done additional apps not listed on that page.

I'll poke around and see what others I can find that are
public...there's a new one every other week, but I always forget to
add them to the wiki.

- Charlie

On Mon, Mar 5, 2012 at 4:03 PM, Tim Lossen <tim@lossen.de> wrote:
> in the upcoming jruby-rubinius-battle [1] i would like to cite some
> high-profile uses of jruby in production. are there any, apart from
> the "success stories" [2] in the github wiki?
>
> please step forward :)
>
> [1] http://wrocloverb.com/
> [2] https://github.com/jruby/jruby/wiki/SuccessStories

The Redcar editor

I work at a game studio within Playdom (owned by Disney ) that uses
jruby.  We just launched our first big title Marvel Avengers Alliance
last week, it's been in development for about a year and a half.
While this is our first jruby game, we have used ruby on a dozen or so
previous titles.  If you go purely by users/requests served, we are
probably near the top.

Chris

awesome! thanks a lot, chris. any numbers that you can share?

(i work at wooga, we also have various game backends in ruby,
and are currently starting to implement the first one in
jruby ... will not launch till autumn, though)

tim


On 2012-03-07, at 12:02 , snacktime wrote:

> On Mon, Mar 5, 2012 at 2:03 PM, Tim Lossen <tim@lossen.de> wrote:
>> http://tim.lossen.de
> To unsubscribe from this list, please visit:
>
>    http://xircles.codehaus.org/manage_email
>
>

--
http://tim.lossen.de

On Wed, Mar 7, 2012 at 3:10 AM, Tim Lossen <tim@lossen.de> wrote:
> awesome! thanks a lot, chris. any numbers that you can share?
>
> (i work at wooga, we also have various game backends in ruby,
> and are currently starting to implement the first one in
> jruby ... will not launch till autumn, though)
>
> tim
>

I seem to remember from somewhere you guys used ruby, not that many of
us in the gaming world yet.

I can't really talk specific numbers too much, but daily requests
served by jruby right now is tens of millions per day, will probably
pass the 100's of millions within a week or so.  We use rails under
tomcat 7.

Performance wise we can push close to 700 requests per second on a
single xeon server.  Of course that number is all relative to the
hardware and the application, but this is not a simple application by
any means.    One interesting thing we found is that in general, jruby
performs closer to pure java performance then I would have thought.
Close enough that it's generally not a big win going with native java
libraries.  There have been a few exceptions, like ditching net-http
to use an http client that uses Grizzly, and making use of some of the
java.util.concurrent stuff, but for the most part we just do
everything in ruby.

The biggest issue we have had is getting used to using threads.  The
jruby guys fixed a couple of performance related threading issues (
and did it really fast I must say), but most of our threading issues
were completely outside jruby.  A gem here and there that was thread
safe but used a giant lock around everything.  Some standard java
libraries that lock aggressively.  I'm still hunting down thread locks
in our app.  But it's not that bad because java gives you the tools to
identify this kind of thing without much effort.

One thing that does hurt is when you need to go to C.  The jruby ffi
is great as far as ease of use, but you take a pretty big performance
hit compared to cruby extensions.  In our case we were looking at
using zeromq for communicating between threads, but the performance
hit was just too much.  On the flip side, there are fewer cases where
you need to go to C because you have access to java, and calling out
into native java libraries doesn't seem to be that expensive.

Chris

On Mar 8, 4:46am, snacktime <snackt...@gmail.com> wrote:
> I seem to remember from somewhere you guys used ruby, not that many of
> us in the gaming world yet.
>
> I can't really talk specific numbers too much, but daily requests
> served by jruby right now is tens of millions per day, will probably
> pass the 100's of millions within a week or so. We use rails under
> tomcat 7.

Wow, that's really excellent. When I started reading, I thought you
must have to be using something light like Sinatra, but this kind of
performance with Rails is great. Which version of Rails are you using?

> Performance wise we can push close to 700 requests per second on a
> single xeon server. Of course that number is all relative to the
> hardware and the application, but this is not a simple application by
> any means.  One interesting thing we found is that in general, jruby
> performs closer to pure java performance then I would have thought.
> Close enough that it's generally not a big win going with native java
> libraries. There have been a few exceptions, like ditching net-http
> to use an http client that uses Grizzly, and making use of some of the
> java.util.concurrent stuff, but for the most part we just do
> everything in ruby.

I'd like to hear about the Grizzly client. This is a common problem
people have with the Ruby HTTP client libraries...they either suck, or
they're a C extension (suck^2). It would be nice to end the problem
for JRuby users once and for all by doing a light JRuby extension
around "the best" Java HTTP client. It should be screaming fast then.

> The biggest issue we have had is getting used to using threads. The
> jruby guys fixed a couple of performance related threading issues (
> and did it really fast I must say), but most of our threading issues
> were completely outside jruby. A gem here and there that was thread
> safe but used a giant lock around everything. Some standard java
> libraries that lock aggressively. I'm still hunting down thread locks
> in our app. But it's not that bad because java gives you the tools to
> identify this kind of thing without much effort.

How are you approaching the external gems? Because threading is a
primary raison d'tre for JRuby, we work hard on issues you report to
us...but I am also interested in helping to "teach the world to sing"
by patching libraries that do too little or too much synchronization.

> One thing that does hurt is when you need to go to C. The jruby ffi
> is great as far as ease of use, but you take a pretty big performance
> hit compared to cruby extensions. In our case we were looking at
> using zeromq for communicating between threads, but the performance
> hit was just too much. On the flip side, there are fewer cases where
> you need to go to C because you have access to java, and calling out
> into native java libraries doesn't seem to be that expensive.

We've heard from others that FFI is actually pretty good in comparison
to C exts, and the MQ libs were one of the prime examples. Do you
have something we can use to reproduce the poor performance? We're
always trying to speed up FFI, since it is (or needs to be) the future
of integrating with C from Ruby.

Thanks for this update...it's great to hear that JRuby's working out
so well for you :)

- Charlie

On 08/03/12 19:46, snacktime wrote:
> I'm still hunting down thread locks
> in our app.  But it's not that bad because java gives you the tools to
> identify this kind of thing without much effort.
>
>

As someone who has started looking at threading in a jruby app but
have not yet delved hugely into it and does not know that much about
the pure Java side I for one would be fascinated about what tools you
  are using and how you go about finding these thread locks.

Blog post? Pretty please!?

Jeff

I don't think our case is that high profile in terms of
throughput/numbers etc. as such (as compared to the 700 req/sec
throughput stated above), but we have deployed JRuby on Rails for
microblogging platform
in a large enterprise. It's in production for over a year now. Though
the production version is not yet as heavily used as we tested it for in
performance testing environment, the performance test results are
encouraging for us. In perf. test env., the application scaled to around
80 primary requests/sec (excludes images, css, js etc.) on 2 dual core
app servers with ~100% transaction passing percentage, 90th percentile
of response time for any transaction < 2 sec and app, db CPUs util <=
60%. I will be talking about this JRoR case study at RubyConf India 2012
on 25-Mar-2012. http://rubyconfindia.org/2012/talks.html

Hope this info will be of some help in your battle...good luck...

-Darshan.

80 requests/second isn't something to brag about :) (I don't mean to
offend..just saying)

On Sun, Mar 11, 2012 at 12:14 PM, Darshan Karandikar

Chris,

Did you try using jetty?

S Ahmed wrote in post #1051206:
> 80 requests/second isn't something to brag about :) (I don't mean to
> offend..just saying)
>
> On Sun, Mar 11, 2012 at 12:14 PM, Darshan Karandikar

Whether a throughput number is something to brag about or not depends on
the complexity of the transactions involved & the size of the h/w you
deploy to support that the throughput. I know 80
req/sec is not something to brag about in terms of sheer "numbers" since
people always love to see BIG numbers without caring much about the
transactions' complexity & deployed h/w :),
but 80 req/sec(excluding images,
js, css) is a good indicator of excellent "scalability" of the JRoR
platform especially because that throughput is achieved with just 2
fully utilized Intel Xeon cores & 4 GB RAM supporting a medium 
complexity app.

-Darshan.

I think the a previous post regarding 700 requests per second was 
something
to brag about.
I don't mean to start a flame war as that isn't my intention, but unless
those pages are doing something pretty heavy 80 requests is on the low 
end,
2 X xeon cores with 4gb ram is not too shaby.
What would be more interesting to know is if the same application 
running
on ruby/phusion (or whatever) was doing 20 rps and jruby provided a
improvement somehow.

We all love numbers, and benchmarks "never lie" :)

yea something sounds fishy unless its a super heavy I/O intensive
controller method in rails.

On a basic hello world EM-Synchrony Jruby Sinatra app i was getting like
2000 req/sec

>We all love numbers, and benchmarks "never lie" :)

I have seen apps seeming to support super throughput when run with 
benchmarking tools like Apache bench, but the same app when tested with 
a perf. test tool covering all transactions and proportionate data 
volume giving considerably poor throughput...point is if you rely on 
only pure benchmarking tools to get throughput numbers without 
subjecting the app to correctly setup load test covering all 
transactions, app usage patterns, data volumes, n/w b/w considerations 
etc. (load testing itself is a mini subject),  chances are that post 
production deployment you may be surprised to know benchmarks lied to 
you...


AD wrote in post #1055886:
> yea something sounds fishy unless its a super heavy I/O intensive
> controller method in rails.
>
> On a basic hello world EM-Synchrony Jruby Sinatra app i was getting like
> 2000 req/sec

We are talking about jruby in "production", so "hello world" app 
benchmark surely doesn't mean much here...the case I shared is of a 
database intensive app with complex queries running on around total 
600,000+ records in database and a lot of I/O involved...

Anyways...this thread can go on like this forever...afterall 
"performance testing, tuning and benchmarking" is an interesting subject 
indeed...

With this, I would like to close this thread from my end by saying that 
if someone feels something is "fishy" with the numbers I shared or it is 
not at all a jruby "success story" when it comes to "scalability", so be 
it...long live JRuby :)

Sorry it took so long to get back on this.  Been really busy the last
few weeks...
>
> I'd like to hear about the Grizzly client. This is a common problem
> people have with the Ruby HTTP client libraries...they either suck, or
> they're a C extension (suck^2). It would be nice to end the problem
> for JRuby users once and for all by doing a light JRuby extension
> around "the best" Java HTTP client. It should be screaming fast then.
>

Actually, Net::Http isn't that bad performance wise in jruby.  The
difference between it and grizzly is barely noticable when it comes to
request/response times.  The main reasons we had for using a java
based client is incomplete support in Net::Http for authorization and
persistent connections, and connection pooling.

>
> How are you approaching the external gems? Because threading is a
> primary raison d'tre for JRuby, we work hard on issues you report to
> us...but I am also interested in helping to "teach the world to sing"
> by patching libraries that do too little or too much synchronization.
>

We try to create patches for this kind of thing and get them into the
gem.  Maintaining local monkey patches or custom gems is a pain.

>
> We've heard from others that FFI is actually pretty good in comparison
> to C exts, and the MQ libs were one of the prime examples. Do you
> have something we can use to reproduce the poor performance? We're
> always trying to speed up FFI, since it is (or needs to be) the future
> of integrating with C from Ruby.
>

I don't have a simple test case lying around.  We abandoned most of
the zeromq stuff we were using for other reasons.

Chris

On Sun, Mar 11, 2012 at 9:14 AM, Darshan Karandikar
<lists@ruby-forum.com> wrote:
> on 25-Mar-2012. http://rubyconfindia.org/2012/talks.html
>

80 requests a second is pretty good for a real world app on that 
hardware.

Chris