We have been having a problem with our mongrel servers hanging at
seemingly random times.
The problem can be most often reproduced by accessing a page, leaving
the browser open for some time (60+ min) then making another request.
The browser will receive a “Proxy Error” from Apache, If we look at
mongrel via “ps aux” it says
“mongrel_rails [8000/1/253]: handling 127.0.0.1: GET /”
Using strace we pulled this:
14:04:15 poll([{fd=12, events=POLLIN|POLLPRI}], 1, 0) = 0
14:04:15 write(12, “\1\0\0\0\t”, 5) = 5
14:04:15 read(12,
SEVER IS LOCKED HERE…
Then 15 min later…
14:04:15 read(12, 0x8eab758, 16384) = -1 ETIMEDOUT (Connection timed
out)
14:19:40 shutdown(12, 2 /* send and receive */) = -1 ENOTCONN
(Transport endpoint is not connected)
14:19:40 close(12) = 0
14:19:40 rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0
14:19:40 socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 12
14:19:40 fcntl64(12, F_SETFL, O_RDONLY) = 0
14:19:40 fcntl64(12, F_GETFL) = 0x2 (flags O_RDWR)
14:19:40 connect(12, {sa_family=AF_INET, sin_port=htons(3306),
sin_addr=inet_addr(“XXX.XXX.XXX.XXX”)}, 16) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_RCVTIMEO, “\2003\341\1\0\0\0\0”,
8) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_SNDTIMEO, “\2003\341\1\0\0\0\0”,
8) = 0
14:19:40 setsockopt(12, SOL_IP, IP_TOS, [8], 4) = 0
14:19:40 setsockopt(12, SOL_TCP, TCP_NODELAY, [1], 4) = 0
14:19:40 setsockopt(12, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
14:19:40 read(12, “F\0\0\0\n5.0.60sp1-enterprise-gpl\0\3679”…, 16384)
= 74
We have tried all the results we can find on the internet regarding
“Mongrel Hanging”
For Example:
We are NOT using ruby log rolling
We ARE using the c compiled mysql Adapter
We have tried setting “ActiveRecord::Base.verification_timeout = 14400”
The timeout setting on MySQL are default (interactive_timout,
wait_timeout) = 28800
We have tried the TCP keepalive setting found in here:
http://wooga.drbacchus.com/rails-and-mysql-timeouts
We have tried setting an apache setting:
#Fix for Apache bug 39499
SetEnv force-proxy-request-1.0 1
SetEnv proxy-nokeepalive 1
We have even tried many mysql_adapter.rb patches that “correctly”
reconnect the connection on an error.
None of these have made any difference.
We are currently using:
MySQL 5.0.60sp1
Rails 2.0.2
Mongrel 1.1.5
Thanks you very much for any help!
Justin