Forum: Ruby zombie invasion - six ways to invoke shell processes

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Ce8b03e5750097942c58e12b46724312?d=identicon&s=25 Giles Bowkett (Guest)
on 2007-07-05 21:26
(Received via mailing list)
I've got code somebody else wrote. The code uses exec() to invoke
shell processes. It spawns zombies by the thousands.

I need to redo this over the long term to stop using shell processes at
all.

I need it to stop spawning zombies immediately.

This blog post gives six ways to invoke a shell process:

http://pasadenarb.com/2007/03/ruby-shell-commands.html

How do I choose which of these six methods to use?

All I remember off the top of my head is that backticks is the most
terse and system() is the one I use the most. I had forgotten exec()
even existed. "The Ruby Way" says that exec() doesn't use a subshell
and system() does. I think that's *why* I forgot exec() existed - I
think I blacklisted the method from the moment I discovered it.

I'm switching the legacy code to use system(). Way too much
superstition and guessing in this process for my taste though. How do
you choose the best (least zombie-prone) method to invoke a shell
process?

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org
B9b5ff40232c1dfd61238c2a90467f84?d=identicon&s=25 Wayne E. Seguin (Guest)
on 2007-07-05 21:46
(Received via mailing list)
On Jul 05, 2007, at 15:24 , Giles Bowkett wrote:

> http://pasadenarb.com/2007/03/ruby-shell-commands.html
> superstition and guessing in this process for my taste though. How do
> you choose the best (least zombie-prone) method to invoke a shell
> process?
>
> --
> Giles Bowkett
>
> Blog: http://gilesbowkett.blogspot.com
> Portfolio: http://www.gilesgoatboy.org
>

It sounds like your use cases are rather simple and really only need
to be focusing on `` and system().
In this case use `` when you need to capture the output and system()
otherwise.

Of course I'm assuming that your need is simple here.
Ce8b03e5750097942c58e12b46724312?d=identicon&s=25 Giles Bowkett (Guest)
on 2007-07-05 22:11
(Received via mailing list)
> It sounds like your use cases are rather simple and really only need
> to be focusing on `` and system().
> In this case use `` when you need to capture the output and system()
> otherwise.
>
> Of course I'm assuming that your need is simple here.

The need is simple, I think. But the only difference between using
exec() and system() in this case is that the zombies now have a
different listing for CMD in ps.

On examining the code it looks as if these external processes require
something to be configured which is not configured. So it appears that
every one of these zombies is failing to even do anything.

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org
31e038e4e9330f6c75ccfd1fca8010ee?d=identicon&s=25 Gregory Brown (Guest)
on 2007-07-05 22:23
(Received via mailing list)
On 7/5/07, Giles Bowkett <gilesb@gmail.com> wrote:

> All I remember off the top of my head is that backticks is the most
> terse and system() is the one I use the most. I had forgotten exec()
> even existed. "The Ruby Way" says that exec() doesn't use a subshell
> and system() does. I think that's *why* I forgot exec() existed - I
> think I blacklisted the method from the moment I discovered it.

I wonder if exec() might be useful in forking. Just playing around:

fork do
  exec 'ls'
end
puts "I can has ls?"
Process.wait
B9b5ff40232c1dfd61238c2a90467f84?d=identicon&s=25 Wayne E. Seguin (Guest)
on 2007-07-05 22:30
(Received via mailing list)
On Jul 05, 2007, at 16:10 , Giles Bowkett wrote:
> The need is simple, I think. But the only difference between using
> exec() and system() in this case is that the zombies now have a
> different listing for CMD in ps.
>
> On examining the code it looks as if these external processes require
> something to be configured which is not configured. So it appears that
> every one of these zombies is failing to even do anything.

Giles,

That is a bit odd, I suppose a snippet of the code that produces the
zombies would be necessary to help further.

exec() would kill your ruby process and not return.
system() should execute the command then return to the ruby process
with exit status in $?.
E50adabd92510bf226ad0c3a2adca452?d=identicon&s=25 Wes Sheldahl (Guest)
on 2007-07-05 23:34
(Received via mailing list)
On 7/5/07, Giles Bowkett <gilesb@gmail.com> wrote:
> different listing for CMD in ps.
>
>
In that case, it sounds like you could just comment out the code that's
trying to call the external processes, and you would immediately
eliminate
the zombies without losing any functionality that you have at the
moment. :-) I'd probably start there.

Next figure out whether you really need those external processes to
run. If not then you're done; maybe the reason to run them has gone away
since this was first set up. If you do need them, then configure them so
they run properly in 'standalone' mode, then uncomment the original code
and
chances are you won't have the same zombie problem anymore.

As for exec() vs. system(), the biggest difference is whether you want
any
code after that call to get executed. Since exec replaces the current
process with the new one, any code after the exec() call will never get
run.
If it's the last line of the program anyway, then this won't matter to
you.
E0526a6bf302e77598ef142d91bdd31c?d=identicon&s=25 Daniel DeLorme (Guest)
on 2007-07-06 01:11
(Received via mailing list)
Giles Bowkett wrote:
> I've got code somebody else wrote. The code uses exec() to invoke
> shell processes. It spawns zombies by the thousands.
>
> I need to redo this over the long term to stop using shell processes at
> all.

why? it's one of ruby's strengths that it can act as glue between
external components rather than having to code everything yourself.

> I need it to stop spawning zombies immediately.

Look for the cause. Zombie processes are child processes whose return
response has not yet been collected by the parent. Since you say that
the code uses exec() extensively (when normally it can only be used once
and then the program exits), it leads me to think you have something
like this:
   Process.fork{ exec(cmd) }
which of course would spawn a lot of zombies. You need to detach the
child process from the parent:
   pid = Process.fork{ exec(cmd) }
   Process.detach(pid)

Daniel
Ce8b03e5750097942c58e12b46724312?d=identicon&s=25 Giles Bowkett (Guest)
on 2007-07-06 01:54
(Received via mailing list)
> > On examining the code it looks as if these external processes require
> > something to be configured which is not configured. So it appears that
> > every one of these zombies is failing to even do anything.
> >
> In that case, it sounds like you could just comment out the code that's
> trying to call the external processes, and you would immediately eliminate
> the zombies without losing any functionality that you have at the
> moment. :-) I'd probably start there.

In fact, that's what I did. The code didn't do anything, so I deleted
it.

> Next figure out whether you really need those external processes to
> run. If not then you're done; maybe the reason to run them has gone away
> since this was first set up. If you do need them, then configure them so
> they run properly in 'standalone' mode, then uncomment the original code and
> chances are you won't have the same zombie problem anymore.

The external processes actually called the app itself, via curl.

The external processes are now considered evil in my book. Anything
which spawns zombies is evil; anything which calls itself recursively
via HTTP is also evil. Therefore, evil.

QED.

(Actually, one day we'll probably have apps that call themselves via
HTTP, and it'll be higher-order HTTP, and that'll be great, but for
now, evil.)

> As for exec() vs. system(), the biggest difference is whether you want any
> code after that call to get executed. Since exec replaces the current
> process with the new one, any code after the exec() call will never get run.
> If it's the last line of the program anyway, then this won't matter to you.

Well, it wasn't the last line of code, but it wasn't really being
called properly. Ordinarily I use system() rather than exec(), on the
assumption that in most cases I want the program to continue its end.

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org
Ce8b03e5750097942c58e12b46724312?d=identicon&s=25 Giles Bowkett (Guest)
on 2007-07-06 01:56
(Received via mailing list)
> > I've got code somebody else wrote. The code uses exec() to invoke
> > shell processes. It spawns zombies by the thousands.
> >
> > I need to redo this over the long term to stop using shell processes at
> > all.
>
> why? it's one of ruby's strengths that it can act as glue between
> external components rather than having to code everything yourself.

Yeah, but given the userbase etc. it should be using BackgrounDRb
rather than external shell processes, if it needs external processes.
In fact, it didn't have such a need; the shell process was calling the
app itself via HTTP over curl, and the code was already within the
same app.

Yay rapid prototyping.

--
Giles Bowkett

Blog: http://gilesbowkett.blogspot.com
Portfolio: http://www.gilesgoatboy.org
This topic is locked and can not be replied to.