Forum: Ferret MySQL has gone away

3fb5c540c91cfd02dc2c1dbbb7e6ff5c?d=identicon&s=25 Matthew Langham (silentpenguin)
on 2007-09-03 15:42
Hi,

in a production environment we are running into "MySQL server has gone
away" errors (see below) when we call rebuild_index in aaf. This happens
sporadically.

Any suggestions for what might be wrong or workarounds? I guess we try a
reconnect before we call rebuild_index?

Matthew



ActiveRecord::StatementInvalid (Mysql::Error: MySQL server has gone
away: SELECT count(*) AS count_all FROM bands ):
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/abstract_adapter.rb:128:in
`log'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/mysql_adapter.rb:243:in
`execute'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/mysql_adapter.rb:399:in
`select'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/abstract/database_statements.rb:13:in
`select_one'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/connection_adapters/abstract/database_statements.rb:19:in
`select_value'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/calculations.rb:212:in
`execute_simple_calculation'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/calculations.rb:121:in
`calculate'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/calculations.rb:117:in
`catch'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/calculations.rb:117:in
`calculate'
    (druby://localhost:3009)
/usr/lib/ruby/gems/1.8/gems/activerecord-1.15.3/lib/active_record/calculations.rb:45:in
`count'
    (druby://localhost:3009)
/var/www/web1/oms/current/script/../config/../vendor/plugins/acts_as_ferret/lib/ferret_extensions.rb:22:in
`index_model'
    (druby://localhost:3009)
/var/www/web1/oms/current/script/../config/../vendor/plugins/acts_as_ferret/lib/ferret_extensions.rb:9:in
`index_models'
    (druby://localhost:3009)
/var/www/web1/oms/current/script/../config/../vendor/plugins/acts_as_ferret/lib/ferret_extensions.rb:9:in
`each'
    (druby://localhost:3009)
/var/www/web1/oms/current/script/../config/../vendor/plugins/acts_as_ferret/lib/ferret_extensions.rb:9:in
`index_models'
    (druby://localhost:3009)
/var/www/web1/oms/current/script/../config/../vendor/plugins/acts_as_ferret/lib/ferret_server.rb:93:in
`rebuild_index'
    (druby://localhost:3009)
/var/www/web1/oms/current/script/../config/../vendor/plugins/acts_as_ferret/lib/ferret_server.rb:113:in
`with_class'
    (druby://localhost:3009)
/var/www/web1/oms/current/script/../config/../vendor/plugins/acts_as_ferret/lib/ferret_server.rb:88:in
`rebuild_index'
    /vendor/plugins/acts_as_ferret/lib/remote_index.rb:16:in `send'
    /vendor/plugins/acts_as_ferret/lib/remote_index.rb:16:in
`method_missing'
    /vendor/plugins/acts_as_ferret/lib/class_methods.rb:15:in
`rebuild_index'
3fb5c540c91cfd02dc2c1dbbb7e6ff5c?d=identicon&s=25 Matthew Langham (silentpenguin)
on 2007-09-03 15:52
I need to add that we are running the integrated aaf Ferret Drb Server,
so I guess it is the Drb Server that is losing the connection? How can I
get it to reconnect without throwing the error?
D148c5959e4eccc97f33244f1c5396e9?d=identicon&s=25 Benjamin Krause (Guest)
on 2007-09-03 16:12
(Received via mailing list)
On 2007-09-03, at 3:52 PM, Matthew Langham wrote:

> I need to add that we are running the integrated aaf Ferret Drb
> Server,
> so I guess it is the Drb Server that is losing the connection? How
> can I
> get it to reconnect without throwing the error?

Hey ..

we've got the same error with our non-aaf backgroundrb solution ..

that's what we're doing:

begin
   index_object( record )
rescue ActiveRecord::RecordNotFound
   @logger.warn "Object gone"
rescue ActiveRecord::StatementInvalid
   @logger.warn "MySQL is gone.."
   clazz.connection.reconnect!
   enqueue_object( record )
end

So basically just catch the StatementInvalid Exception and
try again after you reconnected to the database..

I guess Jens can add something like that to AAF quite easily :)

Ben
C9dd93aa135988cabf9183d3210665ca?d=identicon&s=25 Jens Kraemer (Guest)
on 2007-09-03 16:22
(Received via mailing list)
Hi!

On Mon, Sep 03, 2007 at 03:42:07PM +0200, Matthew Langham wrote:
> Hi,
>
> in a production environment we are running into "MySQL server has gone
> away" errors (see below) when we call rebuild_index in aaf. This happens
> sporadically.

Does it happen after a long time of inactivity on the DRb server's side?

If yes, a look at mysql's wait_timeout setting, and
ActiveRecord::Base.verification_timeout, wich should be less than the
former, might help.

Setting ActiveRecord::Base.allow_concurrency to true solved this
problem for me once, however I'm not sure why and it was not
ferret-related.

Cheers,
Jens

--
Jens Krämer
webit! Gesellschaft für neue Medien mbH
Schnorrstraße 76 | 01069 Dresden
Telefon +49 351 46766-0 | Telefax +49 351 46766-66
kraemer@webit.de | www.webit.de

Amtsgericht Dresden | HRB 15422
GF Sven Haubold, Hagen Malessa
D148c5959e4eccc97f33244f1c5396e9?d=identicon&s=25 Benjamin Krause (Guest)
on 2007-09-03 16:28
(Received via mailing list)
>
> Setting ActiveRecord::Base.allow_concurrency to true solved this
> problem for me once, however I'm not sure why and it was not
> ferret-related.

We set the following .. but still - the sql connection is lost from time
to time..

ActiveRecord::Base.allow_concurrency = true
ActiveRecord::Base.verification_timeout = 10

Ben
F0bf457915a11e25d6744d02362511aa?d=identicon&s=25 Julio Cesar Ody (Guest)
on 2007-09-04 00:23
(Received via mailing list)
Are you running Ubuntu?

If yes, try installing the libmysqlclient-dev package (latest one you
have), then run "sudo gem install mysql". You get my meaning. If
you're not running Ubuntu, get the mysql client development headers,
which are required by the mysql gem, then install the gem.

I had a similar issue to yours, then I installed the C based
extensions. Problem went away.
D148c5959e4eccc97f33244f1c5396e9?d=identicon&s=25 Benjamin Krause (Guest)
on 2007-09-04 09:34
(Received via mailing list)
> I had a similar issue to yours, then I installed the C based
> extensions. Problem went away.

Julio,

i got the mysql bindings (version 2.7) installed. But there is a newer
version (2.7.3). I will update them and take a look. Thanks for the
suggestion :)

Ben
0de532978ea562e7a6d794ce7afac55a?d=identicon&s=25 Andreas Korth (Guest)
on 2007-09-04 13:35
(Received via mailing list)
On 03.09.2007, at 16:27, Benjamin Krause wrote:

>>
>> Setting ActiveRecord::Base.allow_concurrency to true solved this
>> problem for me once, however I'm not sure why and it was not
>> ferret-related.
>
> ActiveRecord::Base.allow_concurrency = true
> ActiveRecord::Base.verification_timeout = 10

ActiveRecord just isn't reentrant. Period ;)

There are supposedly ways to make it work using allow_concurrency,
reconnect and whatever. But...

>  .. but still - the sql connection is lost from time to time..

Exactly.

I rebuild my index in a separate process and directory. When it's
done, I tell the index server via DRb that a new index is available
and it'll switch to the new directory. Works like a charm.

Cheers,
Andy
44896977650aef424270590e60b39964?d=identicon&s=25 Ed -- (epo001)
on 2007-09-05 18:40
Andreas Korth wrote:

>
> I rebuild my index in a separate process and directory. When it's
> done, I tell the index server via DRb that a new index is available
> and it'll switch to the new directory. Works like a charm.

Do you use backgroundrb for this?. I'm coming up against similar
problems with long running processes which need an ActiveRecord
connection. I briefly tried to use fork() to background it but that
seemed to give me problems with acts-as-authenticated.
36feb4959db6ab8259a44962f0fa761f?d=identicon&s=25 Jens Kraemer (Guest)
on 2007-09-05 23:10
(Received via mailing list)
On Mon, Sep 03, 2007 at 04:11:34PM +0200, Benjamin Krause wrote:
>
>    clazz.connection.reconnect!
>    enqueue_object( record )
> end
>
> So basically just catch the StatementInvalid Exception and
> try again after you reconnected to the database..
>
> I guess Jens can add something like that to AAF quite easily :)

Okay, I just committed this. According to my limited testing it works as
intended - close db connection with Model.connection.disconnect, and it
will reconnect again.

Please give it a try and report the outcome here :-)

cheers,
Jens

--
Jens Krämer
http://www.jkraemer.net/ - Blog
http://www.omdb.org/     - The new free film database
0de532978ea562e7a6d794ce7afac55a?d=identicon&s=25 Andreas Korth (Guest)
on 2007-09-06 12:36
(Received via mailing list)
On 05.09.2007, at 18:40, Ed -- wrote:

>> I rebuild my index in a separate process and directory. When it's
>> done, I tell the index server via DRb that a new index is available
>> and it'll switch to the new directory. Works like a charm.
>
> Do you use backgroundrb for this?. I'm coming up against similar
> problems with long running processes which need an ActiveRecord
> connection. I briefly tried to use fork() to background it but that
> seemed to give me problems with acts-as-authenticated.

I use fork without problems. Here's a snippet from my controller
which triggers the rebuild:

     def exec_command(command)
       dbconfig = ActiveRecord::Base.remove_connection
       fork do
         begin
           ActiveRecord::Base.establish_connection(dbconfig)
           `cd #{RAILS_ROOT} && script/runner -e #{ENV['RAILS_ENV']}
'#{index_class}.#{command}'`
         ensure
           ActiveRecord::Base.remove_connection
         end
       end
       ActiveRecord::Base.establish_connection(dbconfig)
     end

As you can see, I'm actually running a shell script which starts a
whole new Rails process via script/runner. Note that
ActiveRecord::Base.allow_concurrency is NOT set to true.

I don't remember why this remove/establish connection was necessary,
but it's definitely needed.

Cheers,
Andy
62599364f87417832d81f9cfc81e6d2b?d=identicon&s=25 Ken Wei (kenwei)
on 2007-09-11 09:12
Attachment: active_record_fix.rb (358 Bytes)
I did a hack in the abstract connection code to enable the
auto-reconnection property.
A7dbd84f5f91810f6158e7428d941fe6?d=identicon&s=25 Bruno Miranda (brupm)
on 2008-08-27 05:55
Ken Wei wrote:
> I did a hack in the abstract connection code to enable the
> auto-reconnection property.
Hi Ken,

Where would this hack go? I am not sure I understand what you did there.
62599364f87417832d81f9cfc81e6d2b?d=identicon&s=25 Ken Wei (kenwei)
on 2008-08-27 06:00
Bruno Miranda wrote:
> Ken Wei wrote:
>> I did a hack in the abstract connection code to enable the
>> auto-reconnection property.
> Hi Ken,
>
> Where would this hack go? I am not sure I understand what you did there.

module ActiveRecord
  class Base

    class << self
      alias :old_connection :connection
      def connection
        self.verify_active_connections!
        old_connection
      end
    end

  end
end
This topic is locked and can not be replied to.