Long-running pages cause mongrel time-outs

#1

Hi all,

We use mongrel behind apache (using mod_proxy_balancer) to run an
internal
app for our business. Lately we’ve been seeing frequent errors from some
of
our mongrel processes.

[Sun Dec 16 08:48:47 2007] [error] [client ] (70007)The timeout
specified has expired: proxy: error reading status line from remote
server
127.0.0.1, referer: http://
[Sun Dec 16 08:48:47 2007] [error] [client ] proxy: Error
reading
from remote server returned by /, referer:
http://
[Sun Dec 16 08:49:08 2007] [error] [client ] (70007)The timeout
specified has expired: proxy: error reading status line from remote
server
127.0.0.1
[Sun Dec 16 08:49:08 2007] [error] [client ] proxy: Error
reading
from remote server returned by /

The problem seems to be that the long-running request hogs a mongrel
process, and when apache sends the next request to it (after a
round-robin
tour through the other 9 mongrels), that request times out too. The
original
action eventually comes back, but sometimes not for several minutes, so
we
can timeout on many requests in a row.

I’ve seen problems like this mentioned in the context of file uploads
(which
explain some of our long-running requests), but I’ve never seen a fix
posted. Maybe I haven’t looked hard enough? Does anyone know of a config
fix
that will skip over these blocked processes?

Thanks!
Mike

#2

On Dec 17, 2007, at 4:10 PM, Michael Van Wie wrote:

reading from remote server returned by /,
too. The original action eventually comes back, but sometimes not

The best thing to do is to move any of your long running tasks off
into a work queue and out of the mongrel/rails request response cycle.
Something like Bj woudl help:

http://codeforpeople.rubyforge.org/svn/bj/trunk/README

But I think there has been apache mod_proxy_balancer configs posted
to this list in the past few months that will only send one request to
a mongrel at a time, search the archives.

or if you can use nginx instead there is a new fair proxy balancer
for nginx that works well.

http://brainspl.at/articles/2007/11/09/a-fair-proxy-balancer-for-nginx-and-mongrel

Cheers-

  • Ezra Z.
    – Founder & Software Architect
    – removed_email_address@domain.invalid
    – EngineYard.com
#3

Hi Ezra –

I found it interesting that you recommended Bj (backgroundjobs) and not
BackgrounDrb. I hadn’t heard of Bj before, and I’ll check it out. Thanks
for
the link. Given that you initiated the development BackgrounDrb, I’m
curious
when and where you would recommend Bj over BackgrounDrb.

Mason

#4

There’s a library called Spawn, too, which is nice for doing
asynchronous but still request-triggered background processing. It
might get less nice once 1.9 comes around, though, but… bridges when
we come to them.

Evan

#5

Evan W. wrote:

There’s a library called Spawn, too, which is nice for doing
asynchronous but still request-triggered background processing. It
might get less nice once 1.9 comes around, though, but… bridges when
we come to them.

My Google-fu is eluding me. Can someone point me to somewhere I can
learn more about Spawn?

#6

http://rubyforge.org/projects/spawn/

#7

On Dec 18, 2007, at 1:38 PM, Jacob A. wrote:

–
Cheers,

  • Jacob A.

http://rubyforge.org/projects/spawn/

  • Jason
#8

On Dec 19, 2007 10:31 PM, Steve M. removed_email_address@domain.invalid wrote:

Hi Ezra –

to address this issue.
Thats just one side of the story. BackgrounDRb no longer depends
slave.rb. Its stable as hell. There is not a single bug in
BackgrounDRb that has been reported and not solved. You can checkout
activity on BackgrounDRb mailing list.

BackgrounDRb is not Bj however, It can be if you want to use it like
that. Please check your facts before making any such claims.

Mongrel-users mailing list
removed_email_address@domain.invalid
http://rubyforge.org/mailman/listinfo/mongrel-users

–
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org

#9

At 02:42 AM 12/19/2007, removed_email_address@domain.invalid wrote:

I found it interesting that you recommended Bj (backgroundjobs) and
not
BackgrounDrb. I hadn’t heard of Bj before, and I’ll check it out.
Thanks for
the link. Given that you initiated the development BackgrounDrb, I’m
curious
when and where you would recommend Bj over BackgrounDrb.

Mason

Hi Mason,

I found this link which clearly illuminates your question.

http://groups.google.com/group/ruby-talk-google/browse_frm/thread/144e9b3644ffe88e#38b99ec30098b956

In short: BackgroundRb relies on slave.rb (maintained by Ara T Howard,
the author of Bj), and Ara claims only one person (Eric H.)
understands how to use slave.rb properly: so BackgroundRb is just too
complex to maintain (too much flexibility). Bj is specifically designed
to address this issue.

Erza’s company EngineYard is one of the sponsors of Bj - so it’s
designed to meet some business needs of theirs (I’m a customer so that
includes me!).

From my analysis, Bj looks good b/c it puts the jobs into a database
table so things are both persistent and scalable with very little
additional infrastructure.

Steve

#10

On Dec 19, 2007, at 9:01 AM, Steve M. wrote:

Hi Ezra –

I found it interesting that you recommended Bj (backgroundjobs) and
not
BackgrounDrb. I hadn’t heard of Bj before, and I’ll check it out.
Thanks for
the link. Given that you initiated the development BackgrounDrb, I’m
curious
when and where you would recommend Bj over BackgrounDrb.

Mason

Backgroundrb and Bj serve different purposes. The new backgroundrb is
a complete rewrite by Hemant that throws out drb and uses an event
driven styel of programming. It really is very stable now and it’s
main focus is on doing jobs that require reporting status in your
rails app. Like if you want to fetch some news feeds and report a
progress bar via ajax polling in your app. Bdrb works very well for
this and other more real time background jobs.

Bj is a work queue. You just shove jobs into the queue which is a
database table and the Bj daemon will run through them and call each
job one at a time. But you can also have Bj daemons on each node in a
cluster of servers and have them all pulling from the same queue. Bj
jobs are just shell commands. So your actual jobs can be ruby scripts,
shell scripts, or script/runner calls. The main idea behind Bj is
that you could throw 100K jobs at it and it will just do them as fast
as it can one at a time without overwhelming the system all at once.

So both of these tools are very solid now, they just serve different
purposes. Backgroiundrb is more for real time jobs and status reports
as well as having a very nice cron like scheduler for jobs. Bj is
more of a work queue that is persistent.

Hope that clears things up, Both of these tools are great for
different circumstances.

Cheers-

  • Ezra Z.
    – Founder & Software Architect
    – removed_email_address@domain.invalid
    – EngineYard.com