Fcgi problems - rails and dreamhost

Rails
1

Hi folks,

First day my site goes live and it’s down. Fun fun.

I’m on dreamhost, and the problem seems to happen when I update the code
and kill the fcgi processes to pick up the changes. It’s happened twice
today; first time it came back after an hour or so; this time not so
far.

here’s the restart script:
killall -USR1 dispatch.fcgi
echo ‘done’

Here’s the error in error.log:
[Tue Jan 02 12:13:41 2007] [error] [client 66.118.143.90] FastCGI: comm
with (dynamic) server
“/home/wmnf_admin/wmnf.org/wmnf/public/dispatch.fcgi” aborted: (first
read) idle timeout (60 sec)
[Tue Jan 02 12:13:41 2007] [error] [client 66.118.143.90] FastCGI:
incomplete headers (0 bytes) received from server
“/home/wmnf_admin/wmnf.org/wmnf/public/dispatch.fcgi”

Here’s the last few lines of fastcgi.crash.log:

[02/Jan/2007:10:58:38 :: 8165] starting
[02/Jan/2007:11:55:26 :: 24649] starting
[02/Jan/2007:12:03:11 :: 24649] asked to terminate ASAP
[02/Jan/2007:12:03:11 :: 8165] asked to terminate ASAP
[02/Jan/2007:12:03:11 :: 24649] terminated gracefully
[02/Jan/2007:12:03:12 :: 8165] terminated gracefully

So, it terminated but isn’t restarting.

I’ve seen recommendations elsewhere to clear out the rails session dir;
I’ve done that with no change.

Any ideas? So far no response from dreamhost support…

thanks much

2

Matt C. wrote:

here’s the restart script:
incomplete headers (0 bytes) received from server
[02/Jan/2007:12:03:12 :: 8165] terminated gracefully

So, it terminated but isn’t restarting.

I’ve seen recommendations elsewhere to clear out the rails session dir;
I’ve done that with no change.

Any ideas? So far no response from dreamhost support…

I have the opposite problem - on the server Garvey, since Saturday,
killing (-9) dispatch.fcgi processes is causing one new dispatch.fcgi
process to start immediately (without waiting for an HTTP request to
come in). This never happened before. For a couple of days the
application wouldn’t start under FastCGI, but would still run fine (but
slowly) under CGI. Now it’s working under FastCGI again. The only
changes I have been making are the one-character changes in .htaccess to
switch between FastCGI and CGI.

The server has just been rebooted, but it’s still resurrecting
dispatch.fcgi. DreamHost support say that nothing has changed on the
server, but this looks like a change in mod_fastcgi configuration to me.

regards

Justin F.

3

It doesn’t seem like I’m getting any fcgi errors until I kill the
processes. When I do, I get the errors (and the site is down) for about
two hours, and then it restarts.

Any ideas why this would happen? I’ve seen reports of dreamhost’s
over-zealous process-killers causing sites to return 500 errors, but I
don’t seem to be getting them in the normal operation; only when I try
to refresh the fcgi and ruby so changes are picked up.

My site’s gone down 4 times in a day in a half…help would be most
appreciated.

Thanks

4

On 1/4/07, Matt C. [email protected] wrote:

My site’s gone down 4 times in a day in a half…help would be most
appreciated.

My approach has been to killall -9 dispatch.fcgi and killall -9
ruby1.8. It seems to work.

5

avocado wrote:

On 1/4/07, Matt C. [email protected] wrote:

My site’s gone down 4 times in a day in a half…help would be most
appreciated.

My approach has been to killall -9 dispatch.fcgi and killall -9
ruby1.8. It seems to work.

Hmm…I’ve been using
killall -USR1 ruby
killall -USR1 dispatch.fcgi

I’ll try your suggestion. Thanks much.

6

Matt C. wrote:

killall -USR1 ruby
killall -USR1 dispatch.fcgi

I’ll try your suggestion. Thanks much.

Use

ps ax

to see if they have gone away (or if they have resurrected themselves
with different PIDs). I agree with avocado’s advice, and it’s consistent
with this DreamHost wiki page:

http://wiki.dreamhost.com/index.php/Ruby_on_Rails

regards

Justin F.

7

I’m on dreamhost and when I began developing on it I had many problems
with this exact nature. I had to manually kill the processes.

Dreamhost gives you a free SVN service, so what I do now is just use
Capistrano. It allows me to very easily deploy my applications using
less time, giving me more time to develop. It also restarts the FCGI
processes using the reaper script.

I love it.

8

Use

ps ax

to see if they have gone away

Thanks. avocado’s script seems to be working better than what I had, so
far.

I also reduced the number of fcgi processes from 10 to 6, after getting
a response from DH support that I was using too much memory (I had 10
fcgis running, each between 30 and 70mb).

I’ve been monitoring the site today with top, and have noticed zombie
fcgi processes popping up. So, I’d have 3 normal and 3 zombies. I’m
looking into a script to kill those off every hour or so, which might
help.

9

Hmm. I thought I read that that line set something about the GC, I’ll
look
into that.

RSL

10

Russel - Don’t know if you’re still looking at this thread, but here’s
what
I found out.

I think you can edit the last line in your dispatch.fcgi file to do what
you’re asking about.

I believe that this line will limit you to 6 processes:

RailsFCGIHandler.process! nil, 6

So, I’m guessing you can change it from 6 to any other number you’d
like.

Terry

On 1/7/07, Russell N. [email protected] wrote:

anything further.

Six still sounds a lot.

regards

Justin F.

–
Terry (TAD) Donaghe

http://www.tadspot.com

11

Please do tell how you did this. I need nowhere near the six dispatchers
that get started every time my piddlin’ blog starts up. I wasn’t aware
we
had any control over the number of dispatchers. Someone should blog this
and
get it on the internets if you can control them. I’ll do it if no one
else
does [though no one would probably see it].

RSL

12

Guys I have been experiencing similar frustrations for sometime and
DreamHost confirmed that their procwatcher is killing my processes.
The only
way I have been able to get my application back up has been consistent
with what I’ve read on these posts

killall -9 ruby1.8
killall -9 dispatch.fcgi

But I have been doing this manually, does anyone have a blog(like
suggested) or some other systematic tips on tuning and or avoiding this
problem?

From Dream Host…

"It looks like the problem is indeed your scripts being killed by our
procwatcher:

2007-01-14 17:46:11 kill 5198 kreyze dispatch.fcgi “Ruby
process puts user over the fcgi memory limit (170.19 used + 30.58
additional would be over 200.00 limit)”

2007-01-14 17:46:42 kill 10243 kreyze dispatch.fcgi “Ruby
process puts user over the fcgi memory limit (170.19 used + 30.58
additional would be over 200.00 limit)”