Faster running tests in jruby

I’m having a really hard time using TDD with jruby as it’s taking
forever to run my tests. I’m using rspec in a pretty standard way with
rails, running unit and functional tests. No fancy integration tests,
cucumber, browser simulation or any of that.

My biggest problem is the time it takes to load up the vm with each rake
spec that I run. Even for a single spec file, I’m talking at least 30
seconds to spin up the vm each time on a new MBP. It’s pretty much
making me not want to TDD anymore.

Has anyone found a good solution to this so that I can quickly run my
tests and not rip my hair out because it takes so long? I’ve read a bit
about autotest, but thought I read somewhere that it still spins up a
new vm each time it runs the tests, is this still the case?

We have java dependencies in our app so I can’t test in CRuby.

Many thanks,

Brad

On Thu, Nov 18, 2010 at 10:58 PM, Brad Robertson
[email protected] wrote:

I’m having a really hard time using TDD with jruby as it’s taking forever to run
my tests. I’m using rspec in a pretty standard way with rails, running unit and
functional tests. No fancy integration tests, cucumber, browser simulation or any
of that.

which version of Rails ?

My biggest problem is the time it takes to load up the vm with each rake spec
that I run. Even for a single spec file, I’m talking at least 30 seconds to spin
up the vm each time on a new MBP. It’s pretty much making me not want to TDD
anymore.

I’m surprised because to me (I’m not doing TDD with rails on jruby
yet) jruby’s speed is on par with rspec/cucumber on windows MRI.

Has anyone found a good solution to this so that I can quickly run my tests and
not rip my hair out because it takes so long? I’ve read a bit about autotest, but
thought I read somewhere that it still spins up a new vm each time it runs the
tests, is this still the case?

Are you using rakefiles ? Could you share at least these ?


Christian

Rails 2.3.10

It’s not the speed of running the actual tests… it’s the startup that
kills me. I’m quite sure this is just spinning up the VM, jruby is
known for starting up considerably slower than MRI

I’m using the standard rake tasks that the rspec gem gives you, running
rake spec, rake spec:models etc… even running rake spec SPEC=some/spec/file takes 30 seconds just to load everything up. As I
say though, once the vm is up and the rails environment loaded, the
tests themselves then run fast, it’s just that it has to spin up each
time i rake spec, which is what takes so long.

On Thu, Nov 18, 2010 at 11:37 PM, Brad Robertson
[email protected] wrote:

Rails 2.3.10

It’s not the speed of running the actual tests… it’s the startup that kills
me. I’m quite sure this is just spinning up the VM, jruby is known for starting up
considerably slower than MRI

I’m using the standard rake tasks that the rspec gem gives you, running rake spec, rake spec:models etc… even running rake spec SPEC=some/spec/file
takes 30 seconds just to load everything up. As I say though, once the vm is up
and the rails environment loaded, the tests themselves then run fast, it’s just
that it has to spin up each time i rake spec, which is what takes so long.

Interesting. Few more questions:

  • Do you see the same behaviour on projects not based on rails ?
  • How many spec files do you have ?
  • What’s your OS ?

I’ll try some rails apps with jruby soon. If I do not see this slow
behaviour, I’ll let you know.


Christian

Haven’t tried it standalone (ie outside a rails app)
have probably 20 spec files, but as I say it’s slow running a single
spec
OSX 10.6

On Thu, Nov 18, 2010 at 2:42 PM, Christian MICHON
[email protected] wrote:

On Thu, Nov 18, 2010 at 11:37 PM, Brad Robertson
[email protected] wrote:

It’s not the speed of running the actual tests… it’s the startup that kills
me.

Interesting. Few more questions:

Don’t bother – yes, jruby startup is slower than mri:

ripple:~$ time jruby -e “puts ‘foo’”
foo

real 0m4.249s
user 0m0.604s
sys 0m0.262s
ripple:~$ time ruby -e “puts ‘foo’”
foo

real 0m0.373s
user 0m0.002s
sys 0m0.005s
ripple:~$

It’s not a rails/rspec/rake whatever issue. But yes, it makes running
tests more painful :slight_smile:


Hassan S. ------------------------ [email protected]
twitter: @hassan

On Thu, Nov 18, 2010 at 11:52 PM, Hassan S.
[email protected] wrote:

On Thu, Nov 18, 2010 at 2:42 PM, Christian MICHON
[email protected] wrote:

On Thu, Nov 18, 2010 at 11:37 PM, Brad Robertson
[email protected] wrote:

It’s not the speed of running the actual tests… it’s the startup that kills
me.

Interesting. Few more questions:

Don’t bother – yes, jruby startup is slower than mri:

I know this for a fact, but not 30 seconds.

user 0m0.002s
sys 0m0.005s
ripple:~$

It’s not a rails/rspec/rake whatever issue. But yes, it makes running
tests more painful :slight_smile:

I do not mind waiting for 5 sec :slight_smile:


Christian

There is a project called spork meant to alleviate this issue for cruby

  • would be interesting to see if it works (in essence keep the jvm on
    and just reload the ruby and rails subvm) or if the concept could be
    easily ported(my guess is yes)

Hi, all.

There is a project called spork meant to alleviate this issue for cruby - would
be interesting to see if it works (in essence keep the jvm on and just reload the
ruby and rails subvm) or if the concept could be easily ported(my guess is yes)

The Nailgun support that ships with JRuby uses a similar idea, and
works pretty well for running short-lived programs like Rake scripts
and spec files. You wouldn’t want to spin up an entire Rails server
and do functional testing with it, but for unit testing it should work
fine.

In one terminal:

jruby --ng-server

In another:

jruby --ng -S rake spec

Hope this helps…

–Ian

This may or may not do all you need. I’ve not tested it with the latest
RSpec.

But I do know that when I use rspec on my projects I had to modify the
rspec tasks to pass --debug on to any jruby instances. This was do to
the fact that calling jruby -S rake spec launches another jruby
instance

example:
jruby --ng -S rake spec
produces this output
(in /Users/jay/Projects/UBO/Sandbox/Jay/InfoManagement)
/usr/local/Cellar/jruby/1.5.5/jruby/bin/jruby -S bundle exec rspec
“./spec/nodes/asset_spec.rb”

I wonder if there is a way to tell if nailgun is being used here…

in rspec 1.3.1 and earlier I had to add options to my rcov to get the
debug flag to propagate properly…

here is what I had to do…
desc “Run all specs in spec directory with RCov (excluding plugin
specs)”
Spec::Rake::SpecTask.new(:rcov) do |t|
t.spec_opts = [’–options’, “”#{APP_ROOT}/spec/spec.opts""]
t.spec_files = FileList[‘spec/**/*_spec.rb’]
t.ruby_opts = ["–debug"] if RUBY_PLATFORM =~ /java/
t.rcov = true
t.rcov_opts = lambda do
IO.readlines("#{APP_ROOT}/spec/rcov.opts").map {|l|
l.chomp.split " "}.flatten
end
end

I’m trying something similar with Rspec 2. but passing the --ng flag
like so:
RSpec::Core::RakeTask.new(:spec) do |t|
t.ruby_opts = ["–ng"] if RUBY_PLATFORM =~ /java/
end

says that --ng is not a valid argument.
/usr/local/Cellar/jruby/1.5.5/jruby/bin/jruby --ng -S bundle exec
rspec “./spec/nodes/asset_spec.rb”
jruby: unknown option --ng

I wonder if that means Jruby is NOT spinning up a new vm in this
case… ugh… need to noodle on this more.

Jay

I do not mind waiting for 5 sec :slight_smile:

You don’t mind waiting 5s to print out “foo” ?? I personally think
that’s ridiculous. Now try loading the Rails environment… you’ll get
your 30s

$ time rake spec SPEC=spec/models/report_spec.rb

real 0m37.813s
user 0m48.052s
sys 0m3.179s

Looks like nailgun might help out though, i’ll give that a shot.

I have read all of the responses but they do not match up with my own
experiences. I have a large project comprised of 50+ files with a
matching spec file for each. When I run “ruby -S rake spec” using a
slightly modified Rakefile as created by the Bones gem, my specs run in
about 7 seconds with MRI and about 20 with JRuby. I do not see a large
startup time for the JVM on each spec file. As far as I can tell, that
startup time is incurred once at the very beginning when executing the
“spec” command.

It’s odd that each spec file is creating a separate JRuby instance. That
is not my experience. Perhaps a small modification to your Rakefile
would solve your perf issue?

cr

The real startup cost is in loading the Rails environment, not in
starting
the VM. We have observed the same, particularly on Windows; startup
times on
Linux are considerably faster. (MRI on Windows is similarly pokey.)

5 seconds to start the VM in JRuby doesn’t correspond with our
experience,
though. I can “puts foo” in a second or less under Windows. (No “time”
on
Windows so I can’t measure it exactly.) This is on a Core 2 Duo CPU
3.16Ghz,
Windows XP Pro, java 1.6.0_22, jruby 1.4.0.

On Fri, Nov 19, 2010 at 9:45 AM, Brad Robertson <
[email protected]> wrote:

sys 0m3.179s

On Thu, Nov 18, 2010 at 11:37 PM, Brad Robertson <
I know this for a fact, but not 30 seconds.
real 0m0.373s
Christian
To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email


Paul B.
[email protected]

Cyrus Innovation LLC
200 Varick, Suite 902
New York, NY 10014
phone: (212) 647-7186
fax: (212) 591-6043
http://cyrusinnovation.com

Ya you’re right it’s definitely loading the Rails environment that’s
slow… running nailgun appears to offer no performance benefits
whatsoever…

$ time jruby --ng -S rake spec SPEC=spec/models/report_spec.rb

real 0m40.149s
user 0m0.008s
sys 0m0.023s

So the question is, why the difference?

In addition to that, here are some interesting benchmarks… Using a
vanilla rails app -v 2.3.10, loading the environment using:

time ruby -e “require
File.expand_path(File.join(‘config’,‘environment’))”

MRI 1.9.2
real 0m3.299s
user 0m2.527s
sys 0m0.598s

JRuby 1.5.3
real 0m14.241s
user 0m20.566s
sys 0m1.121s

JRuby 1.5.3 -d32
real 0m8.620s
user 0m8.838s
sys 0m0.612s

JRuby 1.5.3 -d32 -client
real 0m8.116s
user 0m8.726s
sys 0m0.593s

Significant gains using 32-bit jvm, but still significantly slower than
MRI. -client VM appears to have no benefits (is -client default ??)

Hi!

On 19.11.2010, at 17:31, Brad Robertson wrote:

Ya you’re right it’s definitely loading the Rails environment that’s slow…
running nailgun appears to offer no performance benefits whatsoever…

That’s exactly what I observed when trying nailgun out the other day. In
most cases it even seemed to slow things down when it comes to running
Rails tests. But nailgun’s speed gain is indeed impressive when doing
one-liners like jruby -e ‘puts “foo”’ (0.02 vs. 0.5 seconds on my MBP)

However I managed to get a pretty impressive speedup with Rails tests by
forcing the JVM into client (32bit) mode:

$ time rake test:units TEST=test/unit/order_test.rb

real 0m34.809s
user 0m46.898s
sys 0m2.891s

$ export JAVA_OPTS=-d32
$ time rake test:units TEST=test/unit/order_test.rb

real 0m25.556s
user 0m25.891s
sys 0m2.060s

still far from great but better than nothing imho.

Cheers,
Jens


Jens Krmer
Finkenlust 14, 06449 Aschersleben, Germany
VAT Id DE251962952
http://www.jkraemer.net/

Luis Landeiro ribeiro wrote in post #962658:

Checkout https://github.com/rdp/faster_rubygems

should improve startup another bit, it’s a must on windows dev.

Cheers
Lus Ribeiro

faster_require might help
spork is what I’d recommend, if possible.

I remember at JRubyConf last year there was a mention of a bazillion
calls
to stat when reading files. Did anyone ever take a look at that?

For reference, re slow Ruby file loading, see:

http://www.ruby-forum.com/topic/182691
http://www.ruby-forum.com/topic/204113 (Roger’s thread.)

There’s some talk in there about UTF-8 encoding and also
universal-newline
causing performance problems for file reads, but I don’t know how
relevant
they are to the JRuby implementation.

On Fri, Nov 19, 2010 at 6:59 PM, Roger P. [email protected]
wrote:


Posted via http://www.ruby-forum.com/.


To unsubscribe from this list, please visit:

http://xircles.codehaus.org/manage_email


Paul B.
[email protected]

Cyrus Innovation LLC
200 Varick, Suite 902
New York, NY 10014
phone: (212) 647-7186
fax: (212) 591-6043
http://cyrusinnovation.com

Thanks for the tip Roger,
I will check out how it works on my *nix systems,
it requires fork, so I guess Windows is out.

Luis Landeiro R.

Luis Landeiro ribeiro wrote in post #962747:

Thanks for the tip Roger,
I will check out how it works on my *nix systems,
it requires fork, so I guess Windows is out.

later versions shouldn’t require fork…