Ruby Forum Ruby on Rails > Webbrick and/or Mongrels Dies Overnight

I have a SuSE 10.0 server that was first running Webbrick, and now
Mongrel.  In both cases, I can have them running and used all day.
However, either one of them will go unresponsive in the middle of the
night for seemingly no reason.  Being that I only use Mongrel now, I
check the mongrel.log, and nothing unusual there.  I check /var/log/
messages, and nothing there.

Still, a request to http://myhost:3000/mycontroller/list just times
out.  So I stop Mongrel, and restart as a daemon like:
/usr/local/ruby/bin/mongrel_rails start -d --environment production

...and all is fine until the next morning, where the app again goes
unresponsive sometime during the night.  In my mongrel.log this is the
first line of me stopping Mongrel once it is unresponsive and through
to the point where I have to stop/start the next day:

** TERM signal received.
** Daemonized, any open files are closed.  Look at log/mongrel.pid and
log/mongrel.log for info.
** Starting Mongrel listening at 0.0.0.0:3000
** Starting Rails with production environment...
** Rails loaded.
** Loading any Rails specific GemPlugins
** Signals ready.  TERM => stop.  USR2 => restart.  INT => stop (no
restart).
** Rails signals registered.  HUP => reload (without restart).  It
might not work well.
** Mongrel available at 0.0.0.0:3000
** Writing PID file to log/mongrel.pid

What else could I look for?  I have Mongrel running on a another
server too, and it stays up fine.  In theory, they are set up the
exact same way.

Ideas?
Rob

mazurr@gmail.com wrote:
> I have a SuSE 10.0 server that was first running Webbrick, and now
> Mongrel.  In both cases, I can have them running and used all day.
> However, either one of them will go unresponsive in the middle of the
> night for seemingly no reason.  Being that I only use Mongrel now, I
> check the mongrel.log, and nothing unusual there.  I check /var/log/
> messages, and nothing there.

It could be a number of reasons, either in your code or elsewhere. But 
the same thing happened to me a while back and I'm pretty certain it was 
an exploit of Ruby's CGI vulnerability (which was patched in later 
version on Mongrel). SPAM bots were setting it off, not intentionally 
though I think...

If you're not running the latest version of Mongrel, try upgrading it 
and see how you go. In the mean time you could write a short script to 
kill/restart a mongrel when it stops responding.

RYan.
http://yeahnah.org/

There's a daemon specifically designed for this purpose (restarting
unresponsive or otherwise dead services), called monit.

   http://www.tildeslash.com/monit/

n

On May 9, 9:09 pm, Ryan Allen <rails-mailing-l...@andreas-s.net>
wrote:
>
> RYan.http://yeahnah.org/

Thanks for the response, Ryan.

Ya, this app is an internal application, not exposed to the outside
world.  So I don't think it is a bot exploit (though you never know).
There would be a number of security/firewall hoops to jump through for
an outside resource to access this app.

I was starting to write a sript to restart Mongrel nightly.  But then
stopped, thinking, I am fixing the symptom instead of the problem.  I
was hoping to find the source of the issue.  With this same app
running successfully, without interuption, on our development box, I
don't think is it code releated.

Hmmm....still thinking though.
Rob

On May 9, 10:08 pm, Nathan Fiedler <nfied...@bluemarsh.com> wrote:
> There's a daemon specifically designed for this purpose (restarting
> unresponsive or otherwise dead services), called monit.
>
>    http://www.tildeslash.com/monit/
>
> n

Thanks Nathan.  Ya, if I go the route of auto restarting every night,
I may use that.

I still hope to find the source of the problem instead of just band-
aiding the bleeding though.  I am guessing that most folks don't have
to restart their Mongrel daemon every night.

Rob

On 5/11/07, mazurr@gmail.com <mazurr@gmail.com> wrote:

> I was starting to write a sript to restart Mongrel nightly.  But then
> stopped, thinking, I am fixing the symptom instead of the problem.  I
> was hoping to find the source of the issue.  With this same app
> running successfully, without interuption, on our development box, I
> don't think is it code releated.

Have you checked the potential issues in the Mongrel FAQ?

J.

Is Mongrel locking up? Are you using pure Ruby MySQL driver?

There is a known issue with pure Ruby MySQL driver that causes it to
lock up after a period of inactivity (what actually happens is that
MySQL drops an idle connection, and pure Ruby MySQL driver doesn't
handle that situation well.

Another regular reason for Mongrel hangups is logs rotation.

Mongrel FAQ lists a number of other possible causes of this behavior.

--
Alex Verkhovsky

mazurr@gmail.com wrote:
> I have a SuSE 10.0 server that was first running Webbrick, and now
> Mongrel.  In both cases, I can have them running and used all day.
> However, either one of them will go unresponsive in the middle of the
> night for seemingly no reason.
   [...restart app...]
> ...and all is fine until the next morning, where the app again goes
> unresponsive sometime during the night.

I'm having the same problem with WEBrick 
(http://www.ruby-forum.com/topic/126840#new), and both Google and this 
forum searches seem to turn up a lot of wild guessing, a few hacks, but 
not actual solutions.  (This is most likely an inability on my part to 
use Google/forums, and not an indictment against either of those fine 
resources :)

The closest I've been able to find is that it seems that the MySQL 
connection times out, and there're some settings you can make in the 
ActiveRecord::Base to help it reconnect when this happens.

*MY* problem is that I'm not exactly sure which thing to set in 
ActiveRecord (one suggestion was along the lines of 
"ActiveRecord::Base.verify_all_connections if 
ActiveRecord::Base.connection.nil?" -- except I don't remember the exact 
syntax, and I'm not sure exactly where in the app that line would go), 
or where to put them.

This seems to be a reasonably common problem (again, judging by the 
number of Google/forum pages), I'm a bit surprised that there isn't a 
more explicit solution...

If I ever figure it out (probably my main goal, next week), I'll be sure 
to update this and similar threads :)