# Shell pipeline in Ruby?

**URL:** <https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889>\
**Category:** Ruby\
**Created:** [May 12, 2011, 6:07pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889 "2011-05-12T18:07:05Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [May 12, 2011, 6:07pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/1 "2011-05-12T18:07:05Z")

</div>

Hello,

how do you write an equivalent of

$ cmdA | cmdB | cmdC

in Ruby?

Specifically, I would like to see the PID, return value and stderr of  
each of these commands but I would like cmdB to read the stdout of  
cmdA directly, on its stdin, and similarily for cmdC and cmdB.

Here I am not interested in feeding cmdA some particular input (eg. it  
can read /dev/null for all I care) but in general gluing arbitrary fd  
to its stdin might be desirable in other cases.

Thanks

Michal

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 12, 2011, 7:25pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/2 "2011-05-12T19:25:41Z")

</div>

Have you checked out Open3.pipeline?

Jos

–  
Peace cannot be achieved through violence, it can only be attained  
through  
understanding.

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 12, 2011, 8:06pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/3 "2011-05-12T20:06:34Z")

</div>

Michal S. wrote in post #998275:

> Hello,
> 
> how do you write an equivalent of
> 
> $ cmdA | cmdB | cmdC

How about:

puts `cmdA | cmdB | cmdC`

or

puts %x{cmdA | cmdB | cmdC}

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 12, 2011, 9:42pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/4 "2011-05-12T21:42:20Z")

</div>

On 12 May 2011 19:22, Jos B. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Have you checked out Open3.pipeline?

I have checked all the stuff I could find once but did not find a  
solution.

That’s why I am asking, perhaps I overlooked something.

While the pipeline gives the status of all the involved processes and  
connects them properly which is more than I ever got from Ruby it is  
not obvious how to capture the output - the stdout of the last  
process can be captured with pipeline\_r but the stderrs get sinked  
somewhere in the pipeline\_run.

Also the in and out options used in the example are not documented  
afaict.

Thanks

Michal

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 12, 2011, 9:57pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/5 "2011-05-12T21:57:48Z")

</div>

There is also popen4 which I believe also gives you the pid with std -  
in/out/err

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 12, 2011, 10:20pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/6 "2011-05-12T22:20:02Z")

</div>

On 12 May 2011 21:54, Stu [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> There is also popen4 which I believe also gives you the pid with std -  
> in/out/err

It provides stderr but not the pipeline which cannot be constructed in  
pure ruby (unless a popen taking fd:s for stdio is implemented).

Thanks

Michal

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 5:13am UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/7 "2011-05-13T05:13:53Z")

</div>

In the spirit of the recent call for better ruby documentation, I wrote  
a  
writeup:

[http://en.wikibooks.org/w/index.php?title=Ruby\_Programming/Running\_Multiple\_Processes&stable=0#Chaining\_processes](http://en.wikibooks.org/w/index.php?title=Ruby_Programming/Running_Multiple_Processes&stable=0#Chaining_processes)

Enjoy.  
-roger-

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 10:45am UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/8 "2011-05-13T10:45:29Z")

</div>

On Thu, May 12, 2011 at 6:05 PM, Michal S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Here I am not interested in feeding cmdA some particular input (eg. it  
> can read /dev/null for all I care) but in general gluing arbitrary fd  
> to its stdin might be desirable in other cases.

$ ri Open3.pipeline\_r Open3.pipeline\_start

Cheers

robert

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 10:36am UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/9 "2011-05-13T10:36:51Z")

</div>

On 13 May 2011 05:13, Roger P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> In the spirit of the recent call for better ruby documentation, I wrote

Unfortunately, this does not work.

First, these pipelines don’t always work as one would expect so  
testing is _required_ before posting a “solution”.

Second, afaict the reopen method only changes the IOs at Ruby level,  
not the fd:s at C level so it’s useless in this context.

Thanks

Michal

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 3:33pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/10 "2011-05-13T15:33:35Z")

</div>

On 13 May 2011 10:44, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > Here I am not interested in feeding cmdA some particular input (eg. it  
> > can read /dev/null for all I care) but in general gluing arbitrary fd  
> > to its stdin might be desirable in other cases.
> 
> $ ri Open3.pipeline\_r Open3.pipeline\_start

Again, like in Open3.pipeline the :err argument used in the example is  
not documented.

From the description of the example it seems it captures the stderr of  
all the commands together.

Are there more arguments like that?

What do they mean?

Also this is available in ruby 1.9 only.

Thanks

Michal

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 3:43pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/11 "2011-05-13T15:43:55Z")

</div>

> Unfortunately, this does not work.

Try it now, fixed the bug.

> Second, afaict the reopen method only changes the IOs at Ruby level,  
> not the fd:s at C level so it’s useless in this context.

Try it out. This is, AFAIK, the “standard” way to pipeline processes,  
and works in 1.8 even.

GL!  
-r

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 11:39am UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/12 "2011-05-13T11:39:55Z")

</div>

+1 for Robert’s answer. From the Open3.pipeline\_start doc, an example is  
given:

# convert pdf to ps and send it to a printer.

# collect error message of pdftops and lpr.

pdf\_file = “paper.pdf”  
printer = “printer-name”  
err\_r, err\_w = IO.pipe  
Open3.pipeline\_start([“pdftops”, pdf\_file, “-”],  
[“lpr”, “-P#{printer}”],  
:err=\>err\_w) {|ts|  
err\_w.close  
p err\_r.read # error messages of pdftops and lpr.  
}

Cheers  
Lionel

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 3:47pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/13 "2011-05-13T15:47:17Z")

</div>

Roger P. wrote in post #998434:

> In the spirit of the recent call for better ruby documentation, I wrote  
> a  
> writeup:

> **[Chaining processes - Ruby Programming/Running Multiple Processes - Wikibooks,...](https://en.wikibooks.org/w/index.php?title=Ruby_Programming%2FRunning_Multiple_Processes&stable=0#Chaining_processes)**
>
> So say you want to chain some processes together in ruby, the equivalent of bashes \`a | b | c\`?

Unfortunately it appears the wiki formatting is mangled in firefox, but  
works well in chrome 🙂

Here’s the complete code snippet:

pipe\_me\_in, pipe\_peer\_out = IO.pipe  
pipe\_peer\_in, pipe\_me\_out = IO.pipe

fork do  
STDIN.reopen(pipe\_peer\_in)  
STDOUT.reopen(pipe\_peer\_out)  
Kernel.exec(“echo 33”)

# this line is never executed because exec moves the process

end  
pipe\_peer\_out.close # file handles have to all be closed in order for  
the “read” method, below, to be able to know

# that it’s done reading data, so it can return. See also

> **[Beware of pipe duplication in subprocesses](https://devver.wordpress.com/2009/10/22/beware-of-pipe-duplication-in-subprocesses/)**
>
> I’ve been writing a lot lately about managing subprocesses in Ruby, and in that vein I thought I’d write up an interesting subprocess gotcha we encountered recently. I won’t go in…

pipe\_me\_in.read

There is a link in there to a blog describing ruby’s shell class,  
perhaps you did not see it?

-r

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 5:34pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/14 "2011-05-13T17:34:08Z")

</div>

On 12 May 2011 18:05, Michal S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> cmdA directly, on its stdin, and similarily for cmdC and cmdB.
> 
> Here I am not interested in feeding cmdA some particular input (eg. it  
> can read /dev/null for all I care) but in general gluing arbitrary fd  
> to its stdin might be desirable in other cases.
> 
> Thanks
> 
> Michal

OK, thanks for all the replies.

It seems this is not doable with ruby 1.8 but spawn and similar  
methods in 1.9 grew the named arguments which include the file  
descriptors.

Here is a working (although not polished) solution for ruby 1.9.

First a short testing script called cmd:

#!/usr/bin/ruby

prefix = ARGV[0] + “:”  
append = ARGV[1]

while l = STDIN.gets do  
STDOUT \<\< l  
STDERR \<\< prefix \<\< l  
end

STDOUT.puts append

And the long script with pipes that calls the above:

#!/usr/bin/ruby1.9.1

# Debian calls ruby 1.9 ruby1.9.1

pipes =   
(1..4).each {pipes \<\< IO.pipe}  
errs =   
(1..3).each {errs \<\< IO.pipe}

writer = pipes[0][1]  
reader = pipes[3][0]  
i = 1

# Careful about pipe direction, it only works one way.

pids = [  
spawn(“cmd”, “a”, “aa”, :in =\> pipes[0][0], :out =\> pipes[1][1], :err  
=\> errs[0][1] ),  
spawn(“cmd”, “b”, “bb”, :in =\> pipes[1][0], :out =\> pipes[2][1], :err  
=\> errs[1][1] ),  
spawn(“cmd”, “c”, “cc”, :in =\> pipes[2][0], :out =\> pipes[3][1], :err  
=\> errs[2][1] ),  
]

# We don’t really want to see these pipes, perhaps some

Open3.pipeline\_\* method would create them out of sight  
pipes.flatten.select{|p| p != writer && p != reader}.each{|p|p.close}\

maxlen = 128 # some arbitrary size for chunk of data read at once

outputs = errs.collect{|r,w|r}  
outputs \<\< reader  
wsel = [writer]

while true do  
ios = IO.select(outputs, wsel, nil, 1)  
STDERR \<\< ‘.’

```
string = "%4i\n" % i
if writer then
  begin
    result = writer.write_nonblock(string)
    i = i + 1
    if ( i > 1000 ) then
      writer.close
      writer = nil
      wsel = nil
    end
  rescue IO::WaitWritable, Errno::EINTR
  end
end

```

outputs.each{|io|  
begin  
result = io.read\_nonblock(maxlen)  
STDOUT \<\< result  
rescue IO::WaitReadable, Errno::EINTR, EOFError  
if $!.is\_a? EOFError then  
STDERR.puts “EOF”  
outputs.reject!{|o| o == io }  
end  
end  
}

(0…3).each{|i| p = pids[i]  
if p and Process.waitpid(p, Process::WNOHANG) then  
STDOUT.puts $?.inspect  
pids[i] = nil  
end  
}  
break if !ios && pids.select{|pid| pid} ==   
end

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 7:44pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/15 "2011-05-13T19:44:34Z")

</div>

On 13 May 2011 16:23, Robert K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > > > each of these commands but I would like cmdB to read the stdout of
> 
> [env, cmdname, arg1, …, opts] command name and one or  
> more arguments (no shell)  
> [env, [cmdname, argv0], arg1, …, opts] command name and  
> arguments including argv[0] (no shell)
> 
> Note that env and opts are optional, as Process.spawn.
> 
> Then
> 
> $ ri Process.spawn

It’s not quite clear that Process.spawn description also applies to  
options outside the commands and how it applies but maybe it makes  
sense, they would be applied to the whole pipeline.

Thanks

Michal

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 4:26pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/16 "2011-05-13T16:26:08Z")

</div>

On Fri, May 13, 2011 at 3:27 PM, Michal S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> > > cmdA directly, on its stdin, and similarily for cmdC and cmdB.
> > > 
> > > Here I am not interested in feeding cmdA some particular input (eg. it  
> > > can read /dev/null for all I care) but in general gluing arbitrary fd  
> > > to its stdin might be desirable in other cases.
> > 
> > $ ri Open3.pipeline\_r Open3.pipeline\_start
> 
> Again, like in Open3.pipeline the :err argument used in the example is  
> not documented.

Not true. From ri Open3.pipeline\_r in 1.9.2p180:

Each cmd is a string or an array. If it is an array, the elements are  
passed  
to Process.spawn.

cmd:  
commandline command line string which  
is passed to a shell  
[env, commandline, opts] command line string which  
is passed to a shell  
[env, cmdname, arg1, …, opts] command name and one or  
more arguments (no shell)  
[env, [cmdname, argv0], arg1, …, opts] command name and  
arguments including argv[0] (no shell)

Note that env and opts are optional, as Process.spawn.

Then

$ ri Process.spawn

> From the description of the example it seems it captures the stderr of  
> all the commands together.
> 
> Are there more arguments like that?
> 
> What do they mean?

See above.

> Also this is available in ruby 1.9 only.

Well?

Cheers

robert

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 13, 2011, 8:37pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/17 "2011-05-13T20:37:01Z")

</div>

On 13 May 2011 15:48, Roger P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> works well in chrome 🙂  
> Kernel.exec(“echo 33”)
> 
> # this line is never executed because exec moves the process
> 
> end  
> pipe\_peer\_out.close # file handles have to all be closed in order for  
> the “read” method, below, to be able to know
> 
> # that it’s done reading data, so it can return. See also

> **[Beware of pipe duplication in subprocesses](https://devver.wordpress.com/2009/10/22/beware-of-pipe-duplication-in-subprocesses/)**
>
> I’ve been writing a lot lately about managing subprocesses in Ruby, and in that vein I thought I’d write up an interesting subprocess gotcha we encountered recently. I won’t go in…

> pipe\_me\_in.read

Changing the last line to

puts ‘"’ + pipe\_me\_in.read.chomp + ‘"’

Really writes 33 quoted so IO.reopen works for setting up the file  
descriptors.

It is not clear from the docs if it would work or not.

It does work, though:

out = File.open(“testfile”,“w”)

STDOUT.reopen(out)

system(“echo 33”)

STDERR.puts ‘"’ + File.read(“testfile”).chomp + ‘"’

=\> “33”

I vaguely recall some issues with using STDIN and STDOUT for  
redirection earlier but it might be due to using them incorrectly.

> There is a link in there to a blog describing ruby’s shell class,  
> perhaps you did not see it?

Yes, I missed the link labeled just [1]

Thanks

Michal

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 14, 2011, 5:05pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/18 "2011-05-14T17:05:45Z")

</div>

Michal S. wrote in post #998548:

> And the long script with pipes that calls the above:

Since you are doing this on a real operating system, there’s no need to  
use ruby to copy the output of one process to the input of the next  
process (unless you want to capture the communication between the  
processes as well).

OTOH, making sure all the right FDs are closed in each child can be  
tricky in 1.8, because there’s no IO.close\_on\_exec= (although 1.9 has  
this).

Here is an example which works in 1.8.7. Obviously it’s possible to  
refactor this to N children, although I expect you’ll end up with  
something like the ‘pipeline’ method mentioned before if you do.

* * *

#Thread.abort\_on\_exception = true # for debugging

ruby\_to\_a\_rd, ruby\_to\_a\_wr = IO.pipe  
a\_to\_b\_rd, a\_to\_b\_wr = IO.pipe  
a\_err\_rd, a\_err\_wr = IO.pipe

# open: r2a\_r, r2a\_w, a2b\_r, a2b\_w, ae\_r, ae\_w

pid1 = fork do  
ruby\_to\_a\_wr.close  
a\_to\_b\_rd.close  
a\_err\_rd.close  
STDIN.reopen(ruby\_to\_a\_rd)  
STDOUT.reopen(a\_to\_b\_wr)  
STDERR.reopen(a\_err\_wr)  
exec(“cat; echo done cat 1\>&2”)  
STDERR.puts “Whoops! #{$!}”  
end  
ruby\_to\_a\_rd.close  
a\_to\_b\_wr.close  
a\_err\_wr.close

# open: r2a\_w, a2b\_r, ae\_r

b\_to\_c\_rd, b\_to\_c\_wr = IO.pipe  
b\_err\_rd, b\_err\_wr = IO.pipe

# open: r2a\_w, a2b\_r, ae\_r, b2c\_r, b2c\_w, be\_r, be\_w

pid2 = fork do  
ruby\_to\_a\_wr.close  
b\_to\_c\_rd.close  
a\_err\_rd.close  
b\_err\_rd.close  
STDIN.reopen(a\_to\_b\_rd)  
STDOUT.reopen(b\_to\_c\_wr)  
STDERR.reopen(b\_err\_wr)  
exec(“tr ‘a-z’ ‘A-Z’; echo done tr 1\>&2”)  
STDERR.puts “Whoops! #{$!}”  
end  
a\_to\_b\_rd.close  
b\_to\_c\_wr.close  
b\_err\_wr.close

# open: r2a\_w, ae\_r, b2c\_r, be\_r

c\_to\_ruby\_rd, c\_to\_ruby\_wr = IO.pipe  
c\_err\_rd, c\_err\_wr = IO.pipe

# open: r2a\_w, ae\_r, b2c\_r, be\_r, c2r\_r, c2r\_w, ce\_r, ce\_w

pid3 = fork do  
ruby\_to\_a\_wr.close  
c\_to\_ruby\_rd.close  
a\_err\_rd.close  
b\_err\_rd.close  
c\_err\_rd.close  
STDIN.reopen(b\_to\_c\_rd)  
STDOUT.reopen(c\_to\_ruby\_wr)  
STDERR.reopen(c\_err\_wr)  
exec(“sed ‘s/O/0/g’; echo done sed 1\>&2”)  
STDERR.puts “Whoops! #{$!}”  
end  
b\_to\_c\_rd.close  
c\_to\_ruby\_wr.close  
c\_err\_wr.close

# open: r2a\_w, ae\_r, be\_r, c2r\_r, ce\_r

Thread.new do  
ruby\_to\_a\_wr.puts “Here is some data”  
ruby\_to\_a\_wr.puts “And some more”  
ruby\_to\_a\_wr.close  
end

Thread.new do  
while line = a\_err\_rd.gets  
puts “A err: #{line}”  
end  
a\_err\_rd.close  
end

Thread.new do  
while line = b\_err\_rd.gets  
puts “B err: #{line}”  
end  
b\_err\_rd.close  
end

Thread.new do  
while line = c\_err\_rd.gets  
puts “C err: #{line}”  
end  
c\_err\_rd.close  
end

## while line = c\_to\_ruby\_rd.gets puts line end c\_to\_ruby\_rd.close

This can be simplified a bit if you don’t want three separate error  
streams; they can all share the same one. i.e. each child would have

err\_rd.close  
STDERR.reopen(err\_wr)

Regards,

Brian.

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 14, 2011, 5:14pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/19 "2011-05-14T17:14:42Z")

</div>

Here is a ruby 1.9.2 version with close\_on\_exec=. I leave it as an  
exercise to refactor this to an arbitrary pipeline of N commands - it  
would return one input pipe, one output pipe, N pids and N error pipes.

Regards,

Brian.

#Thread.abort\_on\_exception = true # for debugging

ruby\_to\_a\_rd, ruby\_to\_a\_wr = IO.pipe  
ruby\_to\_a\_wr.close\_on\_exec = true # no children should have this

a\_to\_b\_rd, a\_to\_b\_wr = IO.pipe

a\_err\_rd, a\_err\_wr = IO.pipe  
a\_err\_rd.close\_on\_exec = true # no children should have this

pid1 = fork do  
a\_to\_b\_rd.close  
STDIN.reopen(ruby\_to\_a\_rd)  
STDOUT.reopen(a\_to\_b\_wr)  
STDERR.reopen(a\_err\_wr)  
exec(“cat; echo done cat 1\>&2”)  
STDERR.puts “Whoops! #{$!}”  
end

# we don’t want these fds, nor any of the further children

ruby\_to\_a\_rd.close  
a\_to\_b\_wr.close  
a\_err\_wr.close

b\_to\_c\_rd, b\_to\_c\_wr = IO.pipe  
b\_err\_rd, b\_err\_wr = IO.pipe  
b\_err\_rd.close\_on\_exec = true # no children should have this

pid2 = fork do  
b\_to\_c\_rd.close  
STDIN.reopen(a\_to\_b\_rd)  
STDOUT.reopen(b\_to\_c\_wr)  
STDERR.reopen(b\_err\_wr)  
exec(“tr ‘a-z’ ‘A-Z’; echo done tr 1\>&2”)  
STDERR.puts “Whoops! #{$!}”  
end

# we don’t want these fds, nor any of the further children

a\_to\_b\_rd.close  
b\_to\_c\_wr.close  
b\_err\_wr.close

c\_to\_ruby\_rd, c\_to\_ruby\_wr = IO.pipe  
c\_err\_rd, c\_err\_wr = IO.pipe  
c\_err\_rd.close\_on\_exec = true # no children should have this  
pid3 = fork do  
c\_to\_ruby\_rd.close  
STDIN.reopen(b\_to\_c\_rd)  
STDOUT.reopen(c\_to\_ruby\_wr)  
STDERR.reopen(c\_err\_wr)  
exec(“sed ‘s/O/0/g’; echo done sed 1\>&2”)  
STDERR.puts “Whoops! #{$!}”  
end

# we don’t want these fds, nor any of the further children

b\_to\_c\_rd.close  
c\_to\_ruby\_wr.close  
c\_err\_wr.close

Thread.new do  
ruby\_to\_a\_wr.puts “Here is some data”  
ruby\_to\_a\_wr.puts “And some more”  
ruby\_to\_a\_wr.close  
end

Thread.new do  
while line = a\_err\_rd.gets  
puts “A err: #{line}”  
end  
a\_err\_rd.close  
end

Thread.new do  
while line = b\_err\_rd.gets  
puts “B err: #{line}”  
end  
b\_err\_rd.close  
end

Thread.new do  
while line = c\_err\_rd.gets  
puts “C err: #{line}”  
end  
c\_err\_rd.close  
end

while line = c\_to\_ruby\_rd.gets  
puts line  
end  
c\_to\_ruby\_rd.close

---

<div class="post-metadata">

**Author:** ![Michal\_S](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/779978/32.png) [@Michal\_S](https://www.ruby-forum.com/u/Michal_S)\
**Post date:** [May 16, 2011, 1:17pm UTC](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889/20 "2011-05-16T13:17:04Z")

</div>

On 14 May 2011 17:05, Brian C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Michal S. wrote in post #998548:
> 
> > And the long script with pipes that calls the above:
> 
> Since you are doing this on a real operating system, there’s no need to  
> use ruby to copy the output of one process to the input of the next

I don’t use Ruby for that.

And I used a select loop because I had issues with threads in the past.

However, your example works perfectly for me, even when updated to put  
more data through each of the pipes to make sure they don’t end up all  
in a buffer and the threads have to be actually switched for the  
pipeline to work.

I think there might still be issues if the produced data was without  
line endings because none of the gets would finish but that can be  
solved by using read\_nonblock or somesuch instead.

Thanks

Michal

* * *

#Thread.abort\_on\_exception = true # for debugging

ruby\_to\_a\_rd, ruby\_to\_a\_wr = IO.pipe  
a\_to\_b\_rd, a\_to\_b\_wr = IO.pipe  
a\_err\_rd, a\_err\_wr = IO.pipe

# open: r2a\_r, r2a\_w, a2b\_r, a2b\_w, ae\_r, ae\_w

pid1 = fork do  
ruby\_to\_a\_wr.close  
a\_to\_b\_rd.close  
a\_err\_rd.close  
STDIN.reopen(ruby\_to\_a\_rd)  
STDOUT.reopen(a\_to\_b\_wr)  
STDERR.reopen(a\_err\_wr)  
exec(‘while read x ; do echo “$x” ; echo “$x” \>&2 ; done’)  
STDERR.puts “Whoops! #{$!}”  
end  
ruby\_to\_a\_rd.close  
a\_to\_b\_wr.close  
a\_err\_wr.close

# open: r2a\_w, a2b\_r, ae\_r

b\_to\_c\_rd, b\_to\_c\_wr = IO.pipe  
b\_err\_rd, b\_err\_wr = IO.pipe

# open: r2a\_w, a2b\_r, ae\_r, b2c\_r, b2c\_w, be\_r, be\_w

pid2 = fork do  
ruby\_to\_a\_wr.close  
b\_to\_c\_rd.close  
a\_err\_rd.close  
b\_err\_rd.close  
STDIN.reopen(a\_to\_b\_rd)  
STDOUT.reopen(b\_to\_c\_wr)  
STDERR.reopen(b\_err\_wr)  
exec(‘while read x ; do x=`echo "$x" | tr a-z A-Z` ; echo “$x” ;  
echo “$x” \>&2 ; done’)  
STDERR.puts “Whoops! #{$!}”  
end  
a\_to\_b\_rd.close  
b\_to\_c\_wr.close  
b\_err\_wr.close

# open: r2a\_w, ae\_r, b2c\_r, be\_r

c\_to\_ruby\_rd, c\_to\_ruby\_wr = IO.pipe  
c\_err\_rd, c\_err\_wr = IO.pipe

# open: r2a\_w, ae\_r, b2c\_r, be\_r, c2r\_r, c2r\_w, ce\_r, ce\_w

pid3 = fork do  
ruby\_to\_a\_wr.close  
c\_to\_ruby\_rd.close  
a\_err\_rd.close  
b\_err\_rd.close  
c\_err\_rd.close  
STDIN.reopen(b\_to\_c\_rd)  
STDOUT.reopen(c\_to\_ruby\_wr)  
STDERR.reopen(c\_err\_wr)  
exec(‘while read x ; do x=`echo "$x" | sed -e s/O/0/g` ; echo “$x”  
; echo “$x” \>&2 ; done’)  
STDERR.puts “Whoops! #{$!}”  
end  
b\_to\_c\_rd.close  
c\_to\_ruby\_wr.close  
c\_err\_wr.close

# open: r2a\_w, ae\_r, be\_r, c2r\_r, ce\_r

Thread.new do  
ruby\_to\_a\_wr.puts “Here is some data”  
(1..1000).each{|i|  
ruby\_to\_a\_wr.puts “#{i} And some more”  
}  
ruby\_to\_a\_wr.close  
end

Thread.new do  
while line = a\_err\_rd.gets  
puts “A err: #{line}”  
end  
a\_err\_rd.close  
end

Thread.new do  
while line = b\_err\_rd.gets  
puts “B err: #{line}”  
end  
b\_err\_rd.close  
end

Thread.new do  
while line = c\_err\_rd.gets  
puts “C err: #{line}”  
end  
c\_err\_rd.close  
end

while line = c\_to\_ruby\_rd.gets  
puts line  
end  
c\_to\_ruby\_rd.close

[Next page](https://www.ruby-forum.com/t/shell-pipeline-in-ruby/206889.md?page=2)
