Long running query causes Mongrel/Rails to block completely?


#1

Hi.

I’m troubleshooting a problem, and I would love some help
understanding what is going on.

The problem is that Mongrel/Rails appears to hang or completely block
if one of the users initiates a very slow, long-running query (via an
ad-hoc report generation tool from within the app).

For example, user A does a big query that takes minutes to complete.
User B tries to do anything, but Rails seems completely dead.

Should Rails be blocking on that query, effectively a single-user
server? (I certainly wouldn’t expect this…)

Secondly, for what it’s worth, I’m running a mongrel cluster, but that
isn’t helping right now.

More info:

“top” shows the host machine completely idle.
development.log shows only user A’s query SQL, but it does not show
user B’s request for a new page.
When user A’s query finally completes, then suddenly Rails comes back
to life, and user B gets a response.

This is a serious problem for me right now. It must be a
configuration problem, because there’s no way Rails is designed to
behave this way…?

Oh also, the database is hosted on another machine, and it’s Oracle 9.

Thanks!
Michael


#2

On 29 Oct 2008, at 14:31, michael_teter wrote:

For example, user A does a big query that takes minutes to complete.
User B tries to do anything, but Rails seems completely dead.

Should Rails be blocking on that query, effectively a single-user
server? (I certainly wouldn’t expect this…)

Yup that’s the way it is. load balancing across several mongrels helps
a bit, but not massively because most load balancers just spread the
load equally - they don’t prioritize mongrels that aren’t processing a
request.
Rails 2.2 is thread safe, but MRI’s less than stellar threading means
that won’t make a lot of difference if you are using MRI. For jruby
things could get very interesting.

Fred


#3

On Oct 30, 2:00 am, Frederick C. removed_email_address@domain.invalid
wrote:

Yup that’s the way it is. load balancing across several mongrels helps
a bit, but not massively because most load balancers just spread the
load equally - they don’t prioritize mongrels that aren’t processing a
request.
Rails 2.2 is thread safe, but MRI’s less than stellar threading means
that won’t make a lot of difference if you are using MRI. For jruby
things could get very interesting.

Wow, Looks like I’m going to have to try out jruby/rails 2.2 what
with jruby using java/os threads. Anyone started looking at this with
2.2 RC or whatever the latest is?

http://blog.headius.com/2008/08/qa-what-thread-safe-rails-means.html


Daniel B.


#4

I don’t know how practical this is, but I can envision a system
whereby each process wraps a request/response with a Start End message
to a load balancer so it will know which processes are busy and which
are not. Further, the load balancer could keep statistics that might
be useful in optimizing the balance behavior.


#5

A slow SQL query woudn’t freeze rails 2.2 as it freezes 2.1 now as the
MRI removes threads that are waiting for IO handles to be available
from the running list, so this should not be a big issue with the new
rails, but JRuby is a definitely better option today with the latest
rails.

Maurício Linhares
http://alinhavado.wordpress.com/ (pt-br) | http://blog.codevader.com/
(en)
João Pessoa, PB, +55 83 8867-7208


#6

I was actually running in JRuby earlier on in my development process,
but “something changed” and it appeared that the Oracle driver and
JRuby stopped playing nicely together.

Perhaps I should revisit JRuby…

Thanks for the responses!

On Oct 29, 10:29 am, “Maurício Linhares” removed_email_address@domain.invalid


#7

On Oct 30, 2:33 am, michael_teter removed_email_address@domain.invalid wrote:

I was actually running in JRuby earlier on in my development process,
but “something changed” and it appeared that the Oracle driver and
JRuby stopped playing nicely together.

Perhaps I should revisit JRuby…

Thanks for the responses!

If it’s a report you could pass it off to a separate process. You’ll
have to add some extra ui so the user can go back and check or get
notified when it finishes.

http://backgroundrb.rubyforge.org/rails/


Daniel B.


#8

On 29 Oct 2008, at 15:29, Maurício Linhares wrote:

A slow SQL query woudn’t freeze rails 2.2 as it freezes 2.1 now as the
MRI removes threads that are waiting for IO handles to be available
from the running list, so this should not be a big issue with the new

It depends. C extensions can freeze the entire ruby interpreter, right
now the native mysql driver is one of those (but see the work the
neverblock guys have been doing). I don’t recall what other database
drivers do.

Fred

rails, but JRuby is a definitely better option today with the latest
rails.


#9

On Oct 29, 7:43 pm, Lionel B. removed_email_address@domain.invalid
wrote:

global lock to make sure only one was running at one time (maybe to
protect itself against race conditions in some Ruby code or parts of the
implementation that isn’t fully threadsafe yet). Did it change or are

That’s just wrong (YARV/ruby 1.9 is a bit like that - you may have got
the idea from there). See the Charles Nutter’s blog post (http://
blog.headius.com/2008/08/qa-what-thread-safe-rails-means.html ) which
Daniel already linked to.

Fred


#10

Maurício Linhares a écrit, le 10/29/2008 04:29 PM :

A slow SQL query woudn’t freeze rails 2.2 as it freezes 2.1 now as the
MRI removes threads that are waiting for IO handles to be available
from the running list, so this should not be a big issue with the new
rails, but JRuby is a definitely better option today with the latest
rails.

I’m not sure if the threading in Rails 2.2 brings any benefit for JRuby
vs MRI. Last time I checked, even if JRuby used OS threads it used a
global lock to make sure only one was running at one time (maybe to
protect itself against race conditions in some Ruby code or parts of the
implementation that isn’t fully threadsafe yet). Did it change or are
there some implementation details making it more thread friendly than
MRI ?

That said, JRuby could be better than MRI in many cases with Rails 2.1
already, I’m just wondering if it recently got even better :slight_smile:

Lionel


#11

Frederick C. wrote:

On Oct 29, 7:43�pm, Lionel B. removed_email_address@domain.invalid
wrote:

global lock to make sure only one was running at one time (maybe to
protect itself against race conditions in some Ruby code or parts of the
implementation that isn’t fully threadsafe yet). Did it change or are

That’s just wrong (YARV/ruby 1.9 is a bit like that - you may have got
the idea from there). See the Charles Nutter’s blog post (http://
blog.headius.com/2008/08/qa-what-thread-safe-rails-means.html ) which
Daniel already linked to.

Fred

Just benchmarking JRuby against Ruby 1.8 in Rails 2.2 right now. JRuby
is slower.