Forum: Ruby Strange interference between LSAPI, popen3 and MySQL

3a83969376c805ef5b6042191fdb0ff3?d=identicon&s=25 Andreas S. (andreas)
on 2007-03-04 19:39
Hi,

I am experiencing a very strange problem in my Rails app. I use popen3
to execute an external program. Everything works fine in Mongrel, but
when I use the app with LSAPI, after popen3 all MySQL queries fail with
"Mysql::Error (MySQL server has gone away)". Once I comment out popen3
it works fine again.

Does anyone have the slightest idea about what goes wrong here?

Thanks
Andreas
3a83969376c805ef5b6042191fdb0ff3?d=identicon&s=25 Andreas S. (andreas)
on 2007-03-04 19:57
The problem is easy to reproduce in any Rails app using LSAPI and MySQL.
Add the following to a controller action:

    require 'open3'
    Open3.popen3('/usr/bin/ls') {|stdout, stdin, stderr| stdin.read }

The stdin.read is important.
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2007-03-04 20:50
(Received via mailing list)
On Mon, 5 Mar 2007, Andreas Schwarz wrote:

> Hi,
>
> I am experiencing a very strange problem in my Rails app. I use popen3
> to execute an external program. Everything works fine in Mongrel, but
> when I use the app with LSAPI, after popen3 all MySQL queries fail with
> "Mysql::Error (MySQL server has gone away)". Once I comment out popen3
> it works fine again.
>
> Does anyone have the slightest idea about what goes wrong here?

guessing, but popen3 uses fork.  forking with and open db handle is
undefined
for almost any db lib - of course people ignore this.  i did, and had
all
sorts of issues.  you can read about them here

   http://www.linuxjournal.com/article/7922

i bet what happens is that the parent (rails) has setup an at_exit
handler to
close the db on exit.  when popen forks and does it's thing in the
child, the
child's death kills the db handle (at_exit handlers being inherited - of
course).

this shows what i'm talking about


   harp:~ > cat a.rb
   at_exit{ puts 'close connection __A__' }
   fork{ }
   Process.wait

   class Object
     alias_method '__fork__', 'fork'
     def fork *a, &b
       at_exit{ exit! }
       __fork__ *a, &b
     end
   end
   module Kernel
     alias_method '__fork__', 'fork'
     def fork *a, &b
       at_exit{ exit! }
       __fork__ *a, &b
     end
   end

   at_exit{ puts 'close connection __B__' }
   fork{ }
   Process.wait


   harp:~ > ruby a.rb
   close connection __A__


i'd try putting this


   class Object
     alias_method '__fork__', 'fork'
     def fork *a, &b
       at_exit{ exit! }
       __fork__ *a, &b
     end
   end
   module Kernel
     alias_method '__fork__', 'fork'
     def fork *a, &b
       at_exit{ exit! }
       __fork__ *a, &b
     end
   end

in environment.rb or something.  another idea, i'm grasping, is some
sort of
buffer flushing thing.

you could try


   4242.times do |fd|
     begin
       IO.for_fd(fd).flush
     rescue Errno::EBADF
     end
   end

right before the popen.  wild ass guess though.

good luck.

-a
3a83969376c805ef5b6042191fdb0ff3?d=identicon&s=25 Andreas S. (andreas)
on 2007-03-04 21:43
>> I am experiencing a very strange problem in my Rails app. I use popen3
>> to execute an external program. Everything works fine in Mongrel, but
>> when I use the app with LSAPI, after popen3 all MySQL queries fail with
>> "Mysql::Error (MySQL server has gone away)". Once I comment out popen3
>> it works fine again.
>>
>> Does anyone have the slightest idea about what goes wrong here?
>
> guessing, but popen3 uses fork.  forking with and open db handle is undefined
> for almost any db lib - of course people ignore this.  i did, and had all
> sorts of issues.  you can read about them here

OK, good to know. I can close the MySQL connection for the popen3 call
and reopen it after it to avoid problems.

> i bet what happens is that the parent (rails) has setup an at_exit handler to
> close the db on exit.

That makes sense, although I couldn't find any reference to at_exit in
Rails or ruby-lsapi.

> i'd try putting this
>
>
>    class Object
>      alias_method '__fork__', 'fork'
>      def fork *a, &b
>        at_exit{ exit! }
>        __fork__ *a, &b
>      end
>    end
>    module Kernel
>      alias_method '__fork__', 'fork'
>      def fork *a, &b
>        at_exit{ exit! }
>        __fork__ *a, &b
>      end
>    end
>
> in environment.rb or something.

This fixes the MySQL problem (as does manually
disconnecting/reconnecting), but now I have another problem: popen3 just
doesn't do anything when used within LSAPI. The program is never
executed. Any ideas?

Who would have thought that simply calling an external program could be
so difficult...

Thanks
Andreas
Cb48ca5059faf7409a5ab3745a964696?d=identicon&s=25 unknown (Guest)
on 2007-03-04 22:13
(Received via mailing list)
On Mon, 5 Mar 2007, Andreas Schwarz wrote:

> and reopen it after it to avoid problems.
>
>> i bet what happens is that the parent (rails) has setup an at_exit handler to
>> close the db on exit.
>
> That makes sense, although I couldn't find any reference to at_exit in
> Rails or ruby-lsapi.

huh - maybe it's the actuall c mysql driver.

>>      def fork *a, &b
> executed. Any ideas?
hmmm. no.  must be related though.  i'm staring to thinks it's lsapi
which has
setup all the funky at_exit handlers, and that this is causing you
grief...

> Who would have thought that simply calling an external program could be
> so difficult...

me ;-)

ultimately, in ruby queue, i had to setup a drb process before setting
up db
connections.  then i used that drb process to do all forking/waiting for
me.

here's another idea, try systemu, it does not use fork at all

   gem install systemu

   http://rubyforge.org/frs/?group_id=1024&release_id=7721
   http://codeforpeople.com/lib/ruby/systemu/systemu-...

unless you need realtime control over stdin/stdout/stderr it'll do the
trick.

regards.


-a
3a83969376c805ef5b6042191fdb0ff3?d=identicon&s=25 Andreas S. (andreas)
on 2007-03-04 22:33
Ara wrote:
> On Mon, 5 Mar 2007, Andreas Schwarz wrote:
>
>>>      def fork *a, &b
>> executed. Any ideas?
> hmmm. no.  must be related though.  i'm staring to thinks it's lsapi
> which has
> setup all the funky at_exit handlers, and that this is causing you
> grief...

I grepped for at_exit in the LSAPI source code, but didn't find
anything. Maybe it's the stdout/stderr/stdin redirection stuff...

>> Who would have thought that simply calling an external program could be
>> so difficult...
>
> me ;-)
>
> ultimately, in ruby queue, i had to setup a drb process before setting
> up db
> connections.  then i used that drb process to do all forking/waiting for
> me.
>
> here's another idea, try systemu, it does not use fork at all
>
>    gem install systemu
>
>    http://rubyforge.org/frs/?group_id=1024&release_id=7721
>    http://codeforpeople.com/lib/ruby/systemu/systemu-...

It seems to use popen, isn't this based on fork, too?

The reason I want to use popen3 is not that I need stdin/stderr, but
that there is a clean way to pass arguments without having to build a
command line (potential escaping problems etc.).
6076c22b65b36f5d75c30bdcfb2fda85?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2007-03-05 00:17
(Received via mailing list)
Hi~

On Mar 4, 2007, at 1:33 PM, Andreas Schwarz wrote:

> I grepped for at_exit in the LSAPI source code, but didn't find
>> up db
>
> It seems to use popen, isn't this based on fork, too?
>
> The reason I want to use popen3 is not that I need stdin/stderr, but
> that there is a clean way to pass arguments without having to build a
> command line (potential escaping problems etc.).
>
> --
> Posted via http://www.ruby-forum.com/.
>


  LSAPI does its own preforking of Rails processes and disconnects and
reconnects the database handles after every fork. The way they manage
ruby processes causes weird issues like the one you are seeing. The
manager code that does the fork management is closed source so it's
hard to know exactly what is going on in there.

Cheers-
-- Ezra Zygmuntowicz
-- Lead Rails Evangelist
-- ez@engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)
24d2f8804e6bb4b7ea6bd11e0a586470?d=identicon&s=25 Jeremy Kemper (Guest)
on 2007-03-05 00:46
(Received via mailing list)
On 3/4/07, Ezra Zygmuntowicz <ezmobius@gmail.com> wrote:
>
>         LSAPI does its own preforking of Rails processes and disconnects
> and
> reconnects the database handles after every fork. The way they manage
> ruby processes causes weird issues like the one you are seeing. The
> manager code that does the fork management is closed source so it's
> hard to know exactly what is going on in there.


It's all under a BSD-looking license:
  gem install ruby-lsapi

jeremy
6076c22b65b36f5d75c30bdcfb2fda85?d=identicon&s=25 Ezra Zygmuntowicz (Guest)
on 2007-03-05 01:16
(Received via mailing list)
On Mar 4, 2007, at 3:45 PM, Jeremy Kemper wrote:

>
> It's all under a BSD-looking license:
>  gem install ruby-lsapi
>
> jeremy


  You're right, my bad. I didn't realize they distributed the C code
that does the management as well.


-- Ezra Zygmuntowicz
-- Lead Rails Evangelist
-- ez@engineyard.com
-- Engine Yard, Serious Rails Hosting
-- (866) 518-YARD (9273)
1651dafcf287d2fbfbeeb5efc53d9c3f?d=identicon&s=25 George Wang (mistwang)
on 2007-03-09 21:37
OK, we figured out what is wrong. "reopen" method need to be defined for
"LSAPI" class since popen3 calls "reopen". LSAPI new release will
address this.

However, the root cause of broken MySQL connection is not LSAPI's fault,
actually a bug in ruby's "open3.rb", the fix is like:

def popen3(*cmd)
    pw = IO::pipe   # pipe[0] for read, pipe[1] for write
    pr = IO::pipe
    pe = IO::pipe

    pid = fork{
      # child
      fork{
  # grandchild
+     begin

  pw[1].close
  STDIN.reopen(pw[0])
  pw[0].close

  pr[0].close
  STDOUT.reopen(pr[1])
  pr[1].close

  pe[0].close
  STDERR.reopen(pe[1])
  pe[1].close

        exec(*cmd)
+     ensure
+  exit!(0)
+     end
      }
      exit!(0)
    }

    pw[0].close
    pr[1].close
    pe[1].close
    Process.waitpid(pid)
    pi = [pw[1], pr[0], pe[0]]
    pw[1].sync = true
    if defined? yield
      begin
  return yield(*pi)
      ensure
  pi.each{|p| p.close unless p.closed?}
      end
    end
    pi
  end
  module_function :popen3
end

Whenever an error raise in the grandchild process, this process will not
be terminated properly, and DB connection was closed in this process,
thus the DB connection error in the main process. It happens to both
mongrel and LSAPI.

With above fix, DB connection is in good shape even without the LSAPI
fix, just the command was not  getting executed.

Best Regards,
George Wang
http://www.litespeedtech.com/
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.