Mongrel Hanging (Apache, mod_proxy, Mongrel Cluster, Rails)

We have been having a problem with our mongrel servers hanging at
seemingly random times.

The problem can be most often reproduced by accessing a page, leaving
the browser open for some time (60+ min) then making another request.
The browser will receive a “Proxy Error” from Apache, If we look at
mongrel via “ps aux” it says

“mongrel_rails [8000/1/253]: handling 127.0.0.1: GET /”

Using strace we pulled this:

14:04:15 poll([{fd=12, events=POLLIN|POLLPRI}], 1, 0) = 0
14:04:15 write(12, “\1\0\0\0\t”, 5) = 5
14:04:15 read(12,

SEVER IS LOCKED HERE…

Then 15 min later…

14:04:15 read(12, 0x8eab758, 16384) = -1 ETIMEDOUT (Connection timed
out)
14:19:40 shutdown(12, 2 /* send and receive */) = -1 ENOTCONN
(Transport endpoint is not connected)
14:19:40 close(12) = 0
14:19:40 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
14:19:40 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 12
14:19:40 fcntl64(12, F_SETFL, O_RDONLY) = 0
14:19:40 fcntl64(12, F_GETFL) = 0x2 (flags O_RDWR)
14:19:40 connect(12, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(“XXX.XXX.XXX.XXX”)}, 16) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0”,
8) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_SNDTIMEO, “\2003\341\1\0\0\0\0”,
8) = 0
14:19:40 setsockopt(12, SOL_IP, IP_TOS, [8], 4) = 0
14:19:40 setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
14:19:40 read(12, “F\0\0\0\n5.0.60sp1-enterprise-gpl\0\3679”…, 16384)
= 74

We have tried all the results we can find on the internet regarding
“Mongrel Hanging”

For Example:

We are NOT using ruby log rolling
We ARE using the c compiled mysql Adapter
We have tried setting “ActiveRecord::Base.verification_timeout = 14400”
The timeout setting on MySQL are default (interactive_timout,
wait_timeout) = 28800

We have tried the TCP keepalive setting found in here:
http://wooga.drbacchus.com/rails-and-mysql-timeouts

We have tried setting an apache setting:
#Fix for Apache bug 39499
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1

We have even tried many mysql_adapter.rb patches that “correctly”
reconnect the connection on an error.

None of these have made any difference.

We are currently using:
MySQL 5.0.60sp1
Rails 2.0.2
Mongrel 1.1.5

Thanks you very much for any help!

Justin

Hi, Justin. Were you ever able to get this resolved? I’m currently
experiencing the same behavior you describe in your post. Any help is
much appreciated.

Hi Daniel,

I had the same issue recently and detailled it here :
http://groups.google.ca/group/rubyonrails-talk/browse_thread/thread/a53e75680a98f014/81dfbf8cbb0e7c69?lnk=raot

In my case, as in Justin’s IIRC , the firewall between the mongrel and
the
database caused the connection to drop after some time.

Hope this was helpful

Damien

2010/2/8 Daniel S. [email protected]

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs