I spent the last few months benchmarking every combination of ruby
server,
runtime, and framework I could find.
I’m really curious what the ruby community at large thinks of the
results.
http://www.madebymarket.com/blog/dev/ruby-web-benchmark-report.html
-Brian
On Tue, Jul 15, 2014 at 5:31 PM, Brian K. [email protected]
wrote:
I spent the last few months benchmarking every combination of ruby server,
runtime, and framework I could find.I’m really curious what the ruby community at large thinks of the results.
http://www.madebymarket.com/blog/dev/ruby-web-benchmark-report.html
Brian, thank you so much for that work! I haven’t seen such a thorough
and well executed as well as well presented benchmark in a while.
I am looking forward to the benchmark with databases. Do you already
have plans what databases you intend to include? Will it only be
relational or also NoSQL? I think those results will be even more
interesting as the typical web app today will have some form of
persistence. Unfortunately you’ll then have to handle another
variable besides runtime, server and framework…
Kind regards
robert
You’re welcome!
I plan on including relational and NoSQL. If you have suggestions, I’d
love
to hear them.
My goal is to isolate the DB performance, so it might be the same web
framework but different servers, frameworks, ORM’s etc.
I want to do with ORM, without ORM, and so on and so forth.
I’ll probably do preliminary tests to see how much web framework impacts
the other things.
It will probably be really ridiculous by the time it’s finally done
later
this year.
-Brian
On Tue, Jul 15, 2014 at 12:30 PM, Robert K.
[email protected]
Tony,
The way I accounted for JVM warmup was I had a warmup task that ran a
ton
of requests over a minute. Then, I would keep running apache bench until
it
topped out. That process often took longer than the warmup if it kept
getting JIT’ed. I figured Reel would end up faster too, but it really
didn’t. It could be a config issue, I’m not sure.
-Brian
Kind of surprised by the Reel results, especially on JRuby. Last I
benchmarked Reel it performed the best on JRuby I was getting over 5000
reqs/s.
How were you accounting for JVM warmup?
On Tue, Jul 15, 2014 at 7:42 PM, Brian K. [email protected]
wrote:
You’re welcome!
I plan on including relational and NoSQL. If you have suggestions, I’d love
to hear them.
For RDBMS I would suggest
MySQL / MariaDB
PostgreSQL
SQLite
maybe Java DB / Apache Derby for JRuby
HyperSQL? (again Java)
If you want to test commercial ones you need (more effort because of the
setup)
Oracle (the free version has limited functionality)
MS SQL Server
DB2
With NoSQL I’m not too familiar, a few come to mind
MongoDB
CouchDB
maybe Redis
maybe even PStore (yes, Marshal)
maybe db4o for JRuby (Java OODB)
maybe Neo4J (a Java graph database)
The biggest issue with NoSQL databases is that they foster quite
different concepts and you can easily end up rewriting your
application because of completely different connectivity and access
code. The situation is much more convenient with RDBMS because SQL is
pretty standardized and ORM usually take care of the rest.
My goal is to isolate the DB performance, so it might be the same web
framework but different servers, frameworks, ORM’s etc.
As you say you’ll have to combine parts from all categories with all
others, i.e. extend the first benchmark with a persistent part and
plug in all RDBMS into the ORM under test.
I want to do with ORM, without ORM, and so on and so forth.
I’ll probably do preliminary tests to see how much web framework impacts the
other things.It will probably be really ridiculous by the time it’s finally done later
this year.
You should especially calculate that efforts will be much higher than
for the tests you did so far. One reason: there are more components.
Another reason is the fact that setup of a relational database is
significantly more complicated especially if you have to install
Oracle (it’s been a while but I assume that the setup is still pretty
complex compared to “apt-get install postgresql”). On the bright side,
with Oracle you get RMAN with which you should be able to restore a
fixed initial state pretty easily. Coming up with a suitable demo
schema (including constraints, foreign keys and indexes) is probably
also not that easy.
Kind regards
robert
Brian K. [email protected] wrote:
I spent the last few months benchmarking every combination of ruby server,
runtime, and framework I could find.I’m really curious what the ruby community at large thinks of the results.
http://www.madebymarket.com/blog/dev/ruby-web-benchmark-report.html
Hello, unicorn (and yahns[1] (and Rainbows!)) author here.
Thank you for posting here, I don’t comment on commercial sites and
cannot stand forms/images/JS in HTML.
I’m a bit surprised unicorn runs at all 
let alone being faster than
Puma/Thin in some cases.
All tests are run on a 2013 Macbook Air 1.3 GHz Intel Core i5 with 8 GB
of DDR3 RAM and a 250 GB SSD.
How many CPU cores is that? Is HT enabled? Which OS?
I have no idea if unicorn even runs on OSX (or any proprietary OSes).
I doubt yahns does, but yahns supports kqueue on FreeBSD since March
2014 or so.
This is not meant to be a good representation of server hardware or
anything like that. It’s the dev machine I have on hand. It’s a great
machine, but understand that all of these performance numbers are
relative to that level of hardware.
There’s not much difference between server and laptop hardware nowadays.
I only develop on GNU/Linux, and occasionally port to FreeBSD as those
are the platforms where deployment usually happens. Multi-threading
performance might be better on Linux or FreeBSD for Rubinius, too.
The fastest framework is Rack. Plain old Rack with no framework at all.
It makes intuitive sense that Rack on its own would be the fastest
possible, but it???s also worth understanding how much you are giving up
by picking a framework like Sinatra or Rails.
Agreed. I mainly develop and test using Rack.
Memory usage comparisons would also be a good bench, since that’s often
a limiting factor for VPS deployments.
unicorn w/4 processes combined = XXX MB
jruby server(s) w/4 threads = YYY MB
Peak JRuby 1.7.9 performance was 10,159 req/sec using Rack and Unicorn
I do not believe that sentence 
I specifically didn???t test Unicorn with more workers because it would
potentially skew the benchmark results in Unicorns favor simply because
it would have access to more resources and processing power.
Thin and Puma also support multi-process, too. So I think they should
be competitive in that config with unicorn on C Ruby.
For JRuby and Rubinius benchmarks, you should specify how many threads
those servers are running (and how many CPUs you have).
Unicorn Isn???t That Fast
Correct. Top performance was never the primary goal of unicorn[2].
unicorn is meant to tolerate imperfect (buggy!) applications and give
acceptable, predictable performance using multiple processes to
workaround limitations in C Ruby, frameworks and developer ability.
In contrast, yahns is completely intolerant of fatal application bugs.
It uses multiple processes on C Ruby, but also uses multiple threads,
and one-shot kqueue/epoll so it should work well with Rubinius.
yahns should be as fast as C Ruby or Rubinius allows.
I’ve long acknowledged unicorn and yahns are limited by Ruby
performance.
So I’ve mainly focused on improving C Ruby performance in the mean time
(and also teaching myself lock-free programming in C).
* Unicorn performance doesn't seem very predictable
No surprise with a single worker and non-concurrent, stop-the-world GC.
* Unicorn and wrk doesn't seem to benchmark well
I suspect this is because of the lack of persistent connections support
unicorn. For real-world use, unicorn is designed to be useless without
nginx (or similar) in front of it.
* I couldn't get rainbows to work
Rainbows! requires a lot of reading and configuration, so I don’t blame
you. There’s too many options, really, and part of the reason I made
yahns[1] into a separate project.
Thank you again for posting here on a non-commercial,
mouseless-user-friendly list so I can respond
Note: I never post benchmarks for my own servers:
a) They’ll inevitably come off as biased
b) I don’t want to make other servers look bad
(but I am always allowed to make my own servers look bad)
Thus independent evaluations are very welcome
[1] http://yahns.yhbt.net/README
a webserver by a guy who can’t even do HTML :P)
[2] I use C if performance is important, but usually Ruby is enough.
Brian K. [email protected] wrote:
You’re welcome!
I plan on including relational and NoSQL. If you have suggestions, I’d love
to hear them.My goal is to isolate the DB performance, so it might be the same web
framework but different servers, frameworks, ORM’s etc.I want to do with ORM, without ORM, and so on and so forth.
I suggest using Rack::MockRequest to test ORMs, even. No need to have
the web server variable in there when testing an ORM.
For web server performance, I simulate a DB with:
sleep(expected_db_request_time)
It’s pretty uninteresting from a server point-of-view, and
the unicorn (by design) completely sucks with slow databases.
yahns is OK here, as MRI releases the GVL on sleep
(as do the pg and mysql2 drivers, at least).
Hello world is a pretty good benchmark for servers themselves.
Other benchmarks relevant for servers:
Maybe relevant for servers, but you should also use Rack::MockRequest
as a baseline:
On Tue, Jul 15, 2014 at 8:37 PM, Brian K. [email protected]
wrote:
Tony,
The way I accounted for JVM warmup was I had a warmup task that ran a ton of
requests over a minute. Then, I would keep running apache bench until it
topped out. That process often took longer than the warmup if it kept
getting JIT’ed. I figured Reel would end up faster too, but it really
didn’t. It could be a config issue, I’m not sure.
Did you fiddle with GC settings? For server applications there is G1
which is supposed to give better response times and probably also
throughput.
Kind regards
robert
Eric,
I’m a bit surprised unicorn runs at all
Puma/Thin in some cases.
Well at least now there are public benchmarks to prove that unicorn
runs.
How many CPU cores is that? Is HT enabled? Which OS?
It’s 2 core and I think HT enabled. It’s OS X 10.9.4
I have no idea if unicorn even runs on OSX (or any proprietary OSes).
I doubt yahns does, but yahns supports kqueue on FreeBSD since March
2014 or so.
I haven’t tried yahns, but I’ll give it a shot and see if it works or
not.
There’s not much difference between server and laptop hardware nowadays.
That’s a good point, I just didn’t want people looking at the raw
numbers
and saying, “Hey, my server is way faster than that. This benchmark is
wrong!” Which seems common in most benchmarks. So, I added lots of
disclaimers.
I only develop on GNU/Linux, and occasionally port to FreeBSD as those
are the platforms where deployment usually happens. Multi-threading
performance might be better on Linux or FreeBSD for Rubinius, too.
Yeah, I will probably run a future test suite against Linux so that it’s
closer to a real server setup.
Agreed. I mainly develop and test using Rack.
Why don’t more programmers write against Rack directly?
Memory usage comparisons would also be a good bench, since that’s often
a limiting factor for VPS deployments.
unicorn w/4 processes combined = XXX MB
jruby server(s) w/4 threads = YYY MB
That’s a good point and I think I’ll add that dimension to a future
benchmark.
I do not believe that sentence
Me either, I fixed it. Thanks for finding that.
Thin and Puma also support multi-process, too. So I think they should
be competitive in that config with unicorn on C Ruby.
For JRuby and Rubinius benchmarks, you should specify how many threads
those servers are running (and how many CPUs you have).
That’s a very sensible idea. It gets tricky balancing CPU and Memory to
have truly comparable benchmarks. I’ll have to see how best to fairly
compare the different servers.
Correct. Top performance was never the primary goal of unicorn[2].
unicorn is meant to tolerate imperfect (buggy!) applications and give
acceptable, predictable performance using multiple processes to
workaround limitations in C Ruby, frameworks and developer ability.
Yes, and I totally understand that, but I don’t think a lot of people do
the research to understand why Unicorn is different and where the
performance it does have comes from.
In contrast, yahns is completely intolerant of fatal application bugs.
It uses multiple processes on C Ruby, but also uses multiple threads,
and one-shot kqueue/epoll so it should work well with Rubinius.
yahns should be as fast as C Ruby or Rubinius allows.
Well now I have to try yahns!
I’ve long acknowledged unicorn and yahns are limited by Ruby performance.
So I’ve mainly focused on improving C Ruby performance in the mean time
(and also teaching myself lock-free programming in C).
It’s great that you are working on improving C Ruby performance. I don’t
have the C chops to contribute. Perhaps someday…
No surprise with a single worker and non-concurrent, stop-the-world GC.
Again, you’d be surprised how few people know about this stuff. Well,
maybe
you wouldn’t be surprised, but a lot of people read blogs and pick
whatever
the flavor of the week is. After the RapGenius blog fight with Heroku,
I’m
sure interest in Unicorn jumped a bit. I’m not sure a lot of people
understand when and why you would use Unicorn vs. a different server.
I suspect this is because of the lack of persistent connections support
unicorn. For real-world use, unicorn is designed to be useless without
nginx (or similar) in front of it.
That seems like a reasonable tradeoff to me.
Rainbows! requires a lot of reading and configuration, so I don’t blame
you. There’s too many options, really, and part of the reason I made
yahns[1] into a separate project.
Well that explains the difficulty I had then.
Thank you again for posting here on a non-commercial,
mouseless-user-friendly list so I can respond
You are welcome. I’m just glad to be able to contribute something useful
to
the Ruby community at large.
Note: I never post benchmarks for my own servers:
a) They’ll inevitably come off as biased
b) I don’t want to make other servers look bad
(but I am always allowed to make my own servers look bad)
Thus independent evaluations are very welcome
That is a sensible stance to have. All benchmarks are biased, if only by
our lack of knowledge of what we are benchmarking. Some projects go too
far
to show that you should use them because of their awesome benchmarks.
Benchmarks are an important dimension, but they aren’t the only
dimension.
[1] http://yahns.yhbt.net/README
a webserver by a guy who can’t even do HTML :P)
I bet your web servers would be faster if you COULD do HTML
(Also, nobody can really do HTML)
Eric, I’d also just like to say thank you for all the effort you put in
unicorn, rainbows, and yahns. They are excellent projects and we use
unicorn in production at work every day. At my previous job we used
unicorn
to power a site that raised millions of dollars to cure children’s
cancer,
so your work is doing a lot to help make the world a better place.
-Brian
Robert,
Those are all awesome suggestions. I plan on probably testing most of
them.
I don’t know if I will mess with Oracle and MS SQL or not. Open source
projects are a lot easier to get ahold of, test, and so on. Also, some
commercial DB’s have restrictions on what you are allowed to benchmark
and
publish the results. I know that’s silly, but I’m pretty sure that’s
still
a thing.
Also, no I didn’t fiddle too much with GC settings because I didn’t know
what to fiddle with and most people probably don’t know what to fiddle
with. I’m always open to suggestion on things to fiddle with
specifically.
-Brian
On Tue, Jul 15, 2014 at 4:20 PM, Robert K.
[email protected]
That interesting. I was planning on writing a minimal Rack app to do
this,
but Rack::MockRequest might be a better way to go. I’ll look into that
Hello world is a pretty good benchmark for servers themselves.
Other benchmarks relevant for servers:
- Large requests (uploads)
- Large (dynamically-generated) response handling
- zlib/gzip response compression (recent MRI release the GVL)
That’s interesting. I guess those start to be sort of application
dependent
and it really depends on your system load (read or write heavy, lots of
uploads, etc.)
A lot of this is really not obvious to test I guess unless you develop
the
servers.
Maybe relevant for servers, but you should also use Rack::MockRequest
as a baseline:
- Large view rendering (erb/haml/…),
this translates to a large response.
I’ll look into Rack::MockRequest more. Seems like a useful thing to
know.
Thanks!
-Brian
Hi Brian,
that’s very interesting, thank you for doing this. I’m a little suprised
though that padrino [1] is missing on the list of frameworks tested, as
it is quite mature, and has a good userbase afaics. Is there a specific
reason it didn’t make it into the benchmark?
Regards,
Am 15.07.14 17:31, schrieb Brian K.:
Brian K. [email protected] wrote:
Why don’t more programmers write against Rack directly?
Rack is probably too low-level for most folks and requires reading the
spec + docs to know what’s going on. I had to know the Rack and HTTP
specs for implementing servers, so it was natural to me.
Well now I have to try yahns!
Great! Please let us know if you have any questions/comments at
[email protected] (no HTML email, please)
If you have many (16?) cores doing “Hello world”, you may hit contention
problems in the Linux kernel. 3.13 and later kernels should be better,
but I think there’s further room for improvement in the kernel
(but always more room for improvement in Ruby!).
No surprise with a single worker and non-concurrent, stop-the-world GC.
Again, you’d be surprised how few people know about this stuff. Well, maybe
you wouldn’t be surprised, but a lot of people read blogs and pick whatever
the flavor of the week is. After the RapGenius blog fight with Heroku, I’m
sure interest in Unicorn jumped a bit. I’m not sure a lot of people
understand when and why you would use Unicorn vs. a different server.
I’m not aware of what went on with RapGenius in particular. Based on
what I saw on the unicorn mailing list from Heroku users, Heroku was
completely misusing unicorn since their proxy did not completely buffer
requests before forwarding them to unicorn. I’m not sure if that
situation ever improved.
nginx is currently the only proxy I know to work with unicorn,
though I suppose I could whip an equivalent together with yahns in
case people don’t feel like installing nginx.
(Also, nobody can really do HTML)
Then get your mailer to stop sending HTML portions
Eric, I’d also just like to say thank you for all the effort you put in
unicorn, rainbows, and yahns. They are excellent projects and we use
unicorn in production at work every day. At my previous job we used unicorn
to power a site that raised millions of dollars to cure children’s cancer,
so your work is doing a lot to help make the world a better place.
Great to know!
Lars,
I really don’t know how padrino slipped through the cracks. I’ll add it
in
the next round.
-Brian
On Wed, Jul 16, 2014 at 2:13 PM, Brian K. [email protected]
wrote:
I really don’t know how padrino slipped through the cracks.
Maybe too large cracks. 
SCNR
robert
This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.
Sponsor our Newsletter | Privacy Policy | Terms of Service | Remote Ruby Jobs