Executing system commands in threads under Ruby 1.8.6

Ruby
1

I tried to write a script that makes use of external binaries. Each
external binary is called from a different thread, but, under 1.8.6,
this doesn’t seem to work. Everything is executing in a sequential-like
manner.
The guys from StackOverflow said this was because of Ruby’s thread
implementation ( and when I tried the same code under JRuby, it worked
).

Is there some way to make my script work under 1.8.6, or is upgrading to
1.9 the only solution?

2

vhaerun vh wrote:

I tried to write a script that makes use of external binaries. Each
external binary is called from a different thread, but, under 1.8.6,
this doesn’t seem to work. Everything is executing in a sequential-like
manner.
The guys from StackOverflow said this was because of Ruby’s thread
implementation ( and when I tried the same code under JRuby, it worked
).

Is there some way to make my script work under 1.8.6, or is upgrading to
1.9 the only solution?

What do you mean by external binary? Something you invoke with #system
or the like?

$ ruby -e ‘t=Thread.new {system “echo foo; sleep 1; echo bar”}; system
“echo baz; sleep 1; echo quux”; t.join’
foo
baz
bar
quux

Note the order of output: baz before bar. So it’s not sequential.

3

2009/9/7 vhaerun vh [email protected]:

I tried to write a script that makes use of external binaries. Each
external binary is called from a different thread, but, under 1.8.6,
this doesn’t seem to work. Everything is executing in a sequential-like
manner.
The guys from StackOverflow said this was because of Ruby’s thread
implementation ( and when I tried the same code under JRuby, it worked
).

Is there some way to make my script work under 1.8.6, or is upgrading to
1.9 the only solution?

I don’t think that is necessary. Can you provide a example of the
phenomenon you describe? Normally, external programs are separate
processes and work independently. It may be though that if you do the
IO handling for external processes not properly that it looks like
they are executed sequentially because they are blocked in IO
operations.

Kind regards

robert

4

Here’s a link to the question I asked on SO:

I’m executing the calls using backticks.

5

On 7 Sep 2009, at 12:06, Eleanor McHugh wrote:

On 7 Sep 2009, at 09:55, vhaerun vh wrote:

Here’s a link to the question I asked on SO:

windows - Why is this running like it isn't threaded? - Stack Overflow

I’m executing the calls using backticks.

Backticks are blocking calls which return the entirety of their
stdout as a single string on completion of the subprocess.

You may also find my “Ruby Plumber’s Guide to *nix” presentation
useful. It’s available from the link in my sig.

Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

raise ArgumentError unless @reality.responds_to? :reason

6

Hi,

Am Montag, 07. Sep 2009, 20:06:32 +0900 schrieb Eleanor McHugh:

On 7 Sep 2009, at 09:55, vhaerun vh wrote:

Here’s a link to the question I asked on SO:

windows - Why is this running like it isn't threaded? - Stack Overflow

I’m executing the calls using backticks.

Backticks are blocking calls which return the entirety of their stdout as a
single string on completion of the subprocess.

I cannot open the above link. So I rewrite Joel’s code snippet
using backticks:

t = Thread.new {
puts echo foo; sleep 1; echo bar
}
puts echo baz; sleep 1; echo quux
t.join

Stricly spoken it’s not blocking. The “baz” is written to its pipe
before “bar”. But it is stored until “quux” is echoed and puts is
asked to write it out.

Bertram

7

The commands executed in the script at the SO link was simply pinging
the host on an ip range from 1 to 254 like in the following line :

ping #{ip_addr}

This happened from inside each individual thread. How could I replace
the backticks so that they won’t block ?

8

On 7 Sep 2009, at 09:55, vhaerun vh wrote:

Here’s a link to the question I asked on SO:

windows - Why is this running like it isn't threaded? - Stack Overflow

I’m executing the calls using backticks.

Backticks are blocking calls which return the entirety of their stdout
as a single string on completion of the subprocess.

Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

raise ArgumentError unless @reality.responds_to? :reason

9

2009/9/7 geo ssscripting [email protected]:

The commands executed in the script at the SO link was simply pinging
the host on an ip range from 1 to 254 like in the following line :

ping #{ip_addr}

This happened from inside each individual thread. How could I replace
the backticks so that they won’t block ?

You can use one of the popen methods, e.g.

IO.popen [“ping”, ip_addr] do |io|
io.each {|l| puts l}
end

Example

15:37:19 ~$ ruby <<EOF

th.value
end
EOF
1
1 Pinging 192.168.110.74 with 32 bytes of data:
1
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2
2 Pinging 192.168.110.74 with 32 bytes of data:
2
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
1
1 Ping statistics for 192.168.110.74:
1 Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
1 Approximate round trip times in milli-seconds:
1 Minimum = 0ms, Maximum = 0ms, Average = 0ms
2 Reply from 192.168.110.74: bytes=32 time<1ms TTL=128
2
2 Ping statistics for 192.168.110.74:
2 Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
2 Approximate round trip times in milli-seconds:
2 Minimum = 0ms, Maximum = 0ms, Average = 0ms
15:37:38 ~$ ruby -version
ruby 1.8.7 (2008-08-11 patchlevel 72) [i386-cygwin]
-e:1: undefined local variable or method `rsion’ for main:Object
(NameError)
15:37:53 ~$

Cheers

robert

10

On 7 Sep 2009, at 12:56, Bertram S. wrote:

before “bar”. But it is stored until “quux” is echoed and puts is
asked to write it out.

From Ruby’s perspective the backtick is definitely a blocking IO
operation, just as it is in shell script.

Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

raise ArgumentError unless @reality.responds_to? :reason

11

Here’s the rewrite using IO.popen:

threads = []

(1…254).each do |i|
puts “pinging #{i}”
threads << Thread.new {
content = “”
IO.popen(“ping 192.168.0.#{i}”) do |io|
io.each { |l| content << l }
end
content
}
end

threads.each do |t|
t.join
puts t.value
end

This doesn’t behave different. My Ruby -v outputs this :

ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]

I’m running Windows XP.

12

Hi,

Am Dienstag, 08. Sep 2009, 01:11:23 +0900 schrieb Eleanor McHugh:

asked to write it out.

From Ruby’s perspective the backtick is definitely a blocking IO operation,
just as it is in shell script.

Yes, of course. I just examined the interpreter’s source. The
waitpid function is called without the WNOHANG flag. Before that
the child processes output is read and appended to a string until
the stream is closed what usually happens when the program
terminates.

What I meant was just that the child process is really running and
writing “foo” and “baz” to the pipes 1 second before the output
can by noticed in the parent.

Bertram

13

Hi,

Am Montag, 07. Sep 2009, 21:04:53 +0900 schrieb geo ssscripting:

The commands executed in the script at the SO link was simply pinging
the host on an ip range from 1 to 254 like in the following line :

ping #{ip_addr}

This happened from inside each individual thread. How could I replace
the backticks so that they won’t block ?

You could fork-exec-wait as described in
http://www.ruby-doc.org/docs/ProgrammingRuby/html/ref_m_process.html#Process.waitpid.

To retrieve the exit status call waitpid2 as described below
waitpid.

Just fill an array with pids and you don’t even need threading.

Bertram

14

On 07.09.2009 18:58, geo ssscripting wrote:

end
content

}
end

threads.each do |t|
t.join

#join is superfluous when using #value.

puts t.value
end

This doesn’t behave different. My Ruby -v outputs this :

What exactly do you mean? What do you expect? If you refer to seeing
the output of both ping commands sequentially: with the code you
presented you always will get your output sequentially simply because
you wait until all threads finish and then you’ll iterate them and print
the output of one thread at a time. Your ping commands will run in
parallel.

ruby 1.8.6 (2008-08-11 patchlevel 287) [i386-mswin32]

I’m running Windows XP.

robert

15

Hi,

Am Dienstag, 08. Sep 2009, 02:05:03 +0900 schrieb Bertram S.:

What I meant was just that the child process is really running and
writing “foo” and “baz” to the pipes 1 second before the output
can by noticed in the parent.

In other words: the child processes don’t block each other.
Is that ok?

Bertram

16

The first 4 threads or so are created almost instantly, and after that,
everything runs as if only one thing is executed at a time. I get the
same behaviour no matter how many times I try to run it.

17

On 8 Sep 2009, at 03:13, Bertram S. wrote:

Am Dienstag, 08. Sep 2009, 02:05:03 +0900 schrieb Bertram S.:

What I meant was just that the child process is really running and
writing “foo” and “baz” to the pipes 1 second before the output
can by noticed in the parent.

In other words: the child processes don’t block each other.
Is that ok?

Rather let’s say that child processes won’t block each other due to
the underlying implementation of backtick. It is however still
possible to introduce blocking behaviour by using operations in those
children which would cause blocking, such as accessing a semaphore or
other blocking system resource.

Ellie

Eleanor McHugh
Games With Brains
http://slides.games-with-brains.net

raise ArgumentError unless @reality.responds_to? :reason