Forum: Rails deployment mongrel goes unresponsive after 30 minutes

Greetings!

I'm in the process of moving my app from a shared hosting arrangement to 
a
VPS and am seeing some mongrel behavior I haven't seen before.
Specifically, after about 30 minutes of inactivity mongrel goes 
unresponsive
and has to be restarted.  When I restart it writes the messages below.
Before that, there's nothing in the logs that shows any problem at all.

In mongrel.log:

** USR2 signal received.
Thu Feb 08 08:43:51 EST 2007: Reaping 2 threads for slow workers because 
of
'shutdown'
Thread # is too old, killing.
Thu Feb 08 08:43:51 EST 2007: Error calling Dispatcher.dispatch #Thread 
# is
too old, killing.

In production.log:

Mongrel timed out this thread: shutdown

A set of trace messages follow this which I'm happy to post if it would 
be
helpful.

Can anybody help me understand what's going on and what I need to do to 
fix
it?

Thanks much,
Bill

>From Q: Mongrel stops working if it's left alone for a long time.

If you find that Mongrel stops working after a long idle time and
you're using MySQL then you're hitting a bug in the MySQL driver that
doesn't properly timeout connections. What happens is the MySQL server
side of the connection times out and closes, but the MySQL client
doesn't detect this and just sits there.

What you have to do is set:

ActiveRecord::Base.verification_timeout = 14400

Or to any value that is lower than the MySQL server's
interactive_timeout setting. This will make sure that ActiveRecord
checks the connection often enough to reset the connection.

Q: Mongrel stops working if it's left alone for a long time.

If you find that Mongrel stops working after a long idle time and
you're using MySQL then you're hitting a bug in the MySQL driver that
doesn't properly timeout connections. What happens is the MySQL server
side of the connection times out and closes, but the MySQL client
doesn't detect this and just sits there.

What you have to do is set:

ActiveRecord::Base.verification_timeout = 14400

Or to any value that is lower than the MySQL server's
interactive_timeout setting. This will make sure that ActiveRecord
checks the connection often enough to reset the connection.

--
Aníbal Rojas
http://www.rubycorner.com
http://www.hasmanydevelopers.com

The previous snippet is from:

http://mongrel.rubyforge.org/faq.html

--
Aníbal Rojas
http://www.rubycorner.com
http://www.hasmanydevelopers.com

Hi Anibal,

Thanks!  I've got the admin looking into now.  I really appreciate the
response.

Best regards,
Bill

You are welcome, good luck. Zed has a checklist in Mongrel's site for
some other conditions that could cause this errors.

--
Aníbal Rojas
http://www.rubycorner.com
http://www.hasmanydevelopers.com

Bill Walton wrote:
> ** USR2 signal received.
> A set of trace messages follow this which I'm happy to post if it would be 
> helpful.
> 
> Can anybody help me understand what's going on and what I need to do to fix 
> it?

Do you have a cluster of mongrels running or just the one?

-Robby


--
Robby Russell
http://www.robbyonrails.com/
http://www.planetargon.com/

Hi Robby,

Robby Russell wrote:
>
> Do you have a cluster of mongrels running or just the one?

I'm not sure and asked similar question of my hosting service today. 
I'll
report back on that as soon as I find out.  For development purposes I
started with a shared hosting plan with the same provider and the app 
I'm
having problems with is running just fine on that site.  I haven't even
browsed to it in almost two weeks and when I did this afternoon I found 
that
it's just 'hummin along'.  It's not _exactly_the same app, but there are
very few differences.  My first instinct is to look for differences in 
the
code between the two but, IME, that's really only going to be a 
productive
approach if the underlying architecture is comparable.  Any debugging 
advice
/ experience is tremendously appreciated.

Thanks!
Bill

On Thu, 8 Feb 2007 09:07:47 -0600
"Bill Walton" <bill.walton@charter.net> wrote:

> 
> Greetings!
> 
> I'm in the process of moving my app from a shared hosting arrangement to a 
> VPS and am seeing some mongrel behavior I haven't seen before. 
> Specifically, after about 30 minutes of inactivity mongrel goes unresponsive 
> and has to be restarted.  When I restart it writes the messages below. 
> Before that, there's nothing in the logs that shows any problem at all.
> 

Restart (USR2) isn't that reliable because of how ruby/rails needs to
reload.  You should use a full stop/start cycle instead.

Once you start doing that and you still get no response then run strace
on the mongrel process to see what it's doing.  It's most likely that
you have a library that's messing things up.

Also, don't use pstore, don't set Logger to rotate, don't do file
locking, and don't use a crappy DNS.

--
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

Hi Zed,

Thanks very much for your reply.  I appreciate it very much.  More 
below.

Zed A. Shaw wrote:

> Restart (USR2) isn't that reliable because of
> how ruby/rails needs to reload.  You should
> use a full stop/start cycle instead.

I've been doing that.  Should have said it that way.

> Once you start doing that and you still get no
> response then run strace on the mongrel process
> to see what it's doing.

The folks at the hosting service (a2hosting) have done that I think and 
they
tell me it hasn't given them any clues.

> It's most likely that you have a library that's messing
> things up.

If by 'library' you mean gems / plugins, I do have a couple installed:
PDF::Writer and BackgroundRb.  But BackgroundRb is stopped and there are 
no
calls to PDF:Writer being used in the test scenerio I've been using.  I
thought Rails didn't load code before it needed it.  Anyway, here's what 
I'm
doing.

1) Start mongrel using mongrel_rails start -d -p 3002 -e production <
/dev/null >& /dev/null
2) Open a browser and browse to the index action of the Rails app (the 
index
action is empty and the view renders a page with a form on it that has a
group of 3 radio buttons and a submit button.)
3) Close the browser.
4) Wait 40 minutes, then do step 2 again.

The result of step 4 is a blank white screen, not an error message. 
Neither
production.log nor mongel.log have anything written to them at that 
point
other than the initial startup messages in mongrel.log and the initial
request from step 2 in production.log.  When I stop mongrel, it writes a
message to both logs saying its killing a slow worker.  That's the only 
clue
I get.  Steps 1-4 can be repeated ad nauseum.

After their initial investigation the sys admins concluded the problem 
had
to be in my code (surprise, surprise ;-) )  I had a slightly different
version that was running flawlessly on my shared account with them, so I
decided to 'dig in.'  I copied the entire Rails application directory 
for
both apps to my PC, then copied the one I was having problems with on 
the
VPS over to the shared account.  It ran without a problem (and, a week
later, is *still* running without a problem.)

Yesterday they set up a cron job outside my space to invoke the index 
method
every 10 minutes.  That's 'solved' the problem.  It doesn't sit well 
with me
though.  My experience leads me to fear that whatever the real cause of 
the
problem is will eventually manifest itself again.  At the worst possible
time.

If you (or anyone else reading this) have any interest in understanding
what's really going on (since mongrel's behavior is the only clue from 
an
end user perspective), please let me know.  I'll give you any access you
need.  If not, I understand completely.

Thanks again for your time.

Best regards,
Bill

On Fri, 16 Feb 2007 09:20:11 -0600
"Bill Walton" <bill.walton@charter.net> wrote:

> Yesterday they set up a cron job outside my space to invoke the index method 
> every 10 minutes.  That's 'solved' the problem.  It doesn't sit well with me 
> though.  My experience leads me to fear that whatever the real cause of the 
> problem is will eventually manifest itself again.  At the worst possible 
> time.

Interesting, make sure you've got the compiled mysql gem and you aren't
using the rails mysql.rb file.

> If you (or anyone else reading this) have any interest in understanding 
> what's really going on (since mongrel's behavior is the only clue from an 
> end user perspective), please let me know.  I'll give you any access you 
> need.  If not, I understand completely.

Yeah, actually I'd be interested.  Email me off list to coordinate it.

--
Zed A. Shaw, MUDCRAP-CE Master Black Belt Sifu
http://www.zedshaw.com/
http://www.awprofessional.com/title/0321483502 -- The Mongrel Book
http://mongrel.rubyforge.org/
http://www.lingr.com/room/3yXhqKbfPy8 -- Come get help.

On 2/16/07, Zed A. Shaw <zedshaw@zedshaw.com> wrote:
>
> make sure you've got the compiled mysql gem and you aren't
> using the rails mysql.rb file.


Zed,

Can you point me to the reason for this recommendation? Is there a known
problem with pure-Ruby MySQL driver?

Best regards,
Alexey Verkhovsky

The problem has been resolved.  I've included the 'bread crumbs' below 
to
both recognize the contributors and to document it for future Railers.
Thank you to all.

Best regards,
Bill

Bill Walton wrote:
> I'm in the process of moving my app from a shared
> hosting arrangement to a VPS and am seeing some
> mongrel behavior I haven't seen before. Specifically,
> after about 30 minutes of inactivity mongrel goes
> unresponsive and has to be restarted.

Zed A. Shaw wrote:
> make sure you've got the compiled mysql gem and
> you aren't using the rails mysql.rb file.

Ezra Zygmuntowicz wrote:
> Also if your host is running linux then you can tell for
> sure if it's the compiled driver from irb:
>
> ey00-s00059 ~ # irb
> irb(main):001:0> require 'mysql.so'
> => true

Bill Walton wrote:
> irb(main):002:0>
Aníbal Rojas wrote:
> As I undertand the ruby-mysql library 2.7 does
> require the libmysqlclient to be properly installed,
> and depending on the location of mysql.so it is
> possible that rubygems is unable to load it, while
> the library is actually using it...

Support at hosting service wrote:
>This issue has been rectified:
>
> bwalton@yourtimematters.com [~]# irb
> irb(main):001:0> require 'mysql.so'
> => true
> irb(main):002:0>

And that fixed the behavior mongrel was exhibiting.