Mongrels Hanging in mysql_adapter.rb

Hi everyone,

I have been trying to solve this problem for quite some time now and I
really don’t have any further clue to investigate.

I am running a mongrel_cluster 1.0.5-2 (mongrel verison : 1.1.5-5) on
a ubuntu box with apache2 2.2.12-1ubuntu2.1

I have 4 different apps on the same box and each app has two mongrel
instances.

After a short period of time ( < 2 hours ) some of the 8 mongrel
instances just stop responding.

I read the mongrel FAQ and tried to set
ActiveRecord::Base.verification_timeout = 14400 (the mysql parameter
is set to the default 28800) in myapp/config/environments/
production.rb

The problem kept happening.

I decided to lower that number, making it 60.
No more success.

I installed gdb and the ruby macros to make sure i was in the mysql
driver while hanging and sure enough, here is the top of the BT from
GDB :

/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/mysql_adapter.rb:264:in active?'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract_adapter.rb:136:inverify!’",
“/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:256:in
checkout_and_verify'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:252:incheckout_existing_connection’”,
“/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:186:in checkout'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:184:inloop’”,
“/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:184:in checkout'", "/usr/lib/ruby/1.8/monitor.rb:242:insynchronize’”,
“/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:183:in checkout'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_pool.rb:98:inconnection’”,
“/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_pool.rb:326:in
retrieve_connection'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/ connection_adapters/abstract/connection_specification.rb:121:inretrieve_connection’”,
“/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/
connection_adapters/abstract/connection_specification.rb:113:in
connection'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb: 2934:inquoted_table_name’”,
“/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:
1626:in construct_finder_sql'", "/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb: 1490:infind_every’”,
“/var/lib/gems/1.8/gems/activerecord-2.2.2/lib/active_record/base.rb:
589:in `find’”,

I figured that since I was running a packaged version of
mongrel_cluster (not the gem installed one), I should use the package
version of mysql driver for ruby.
So i did
gem uninstall mysql

and installed :
apt-get install libmysql-ruby1.8

I then restarted the mongrels and sure enough after 1h25 one of them
was hanging in the mysql_adapter.rb, showing the same backtrace as
before.

I really am out of clue here and any advice / help would be much
appreciated.

I have to note that I am far from being an ROR guru and maybe the
solution to my problem is simpler than what I have been looking for,
but I just don’t know where to look anymore.

Thanks for any advice / answer

Damien

For the record, our problem was sitting with the firewall between the
app
box and the mysql box.
There seems to be a 4000 seconds timeout on any TCP connection passing
through the firewall…
I followed the steps detailed here :
https://boxpanel.blueboxgrp.com/public/the_vault/index.php/Mongrel_/_MySQL_Timeout
https://boxpanel.blueboxgrp.com/public/the_vault/index.php/Mongrel_/_MySQL_TimeoutMaking
it 900.times and the timeout = 1800

It seems to have done the trick.
I don’t like this solution, but hell, it’s the better of two evils I
think.

It’ll do for now but if anyone has another way of seing things, be sure
to
let me know :wink:

Damien

2010/1/26 serty [email protected]