# Working with a slow pipe (IO.popen)

**URL:** <https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264>\
**Category:** Ruby\
**Created:** [December 16, 2009, 5:33am UTC](https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264 "2009-12-16T05:33:15Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![bwv549](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/ac8455/32.png) [@bwv549](https://www.ruby-forum.com/u/bwv549)\
**Post date:** [December 16, 2009, 5:33am UTC](https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264/1 "2009-12-16T05:33:15Z")

</div>

I’m doing some work with pymol and having trouble getting all the  
pymol output out:

# I open it in quiet/commandline mode with a pipe:

IO.popen(“pymol -cq -p”, ‘w+’) do |pipe|  
pipe.puts “load file.pdb, mymodel\n”  
pipe.puts “run my\_script.py”

# this command will generate a whole bunch of output to stdout

pipe.puts “my\_script mymodel”  
sleep(5) # \<— this is what I have to do in order to get the  
output  
pipe.close\_write  
output = pipe.read  
end

What is annoying is that unless I sleep for 2-5 seconds I don’t get  
output or (even worse) my output is cut off. There has to be a way  
besides arbitrarily sleeping to ensure that the command is finished.  
I’ve tried things like piping in “quit”, but that doesn’t seem to  
work. Anybody know how to solve this? I’m happy to use a different  
approach if it means I can be sure I get all of my output out. –  
Thanks!

---

<div class="post-metadata">

**Author:** ![bwv549](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/ac8455/32.png) [@bwv549](https://www.ruby-forum.com/u/bwv549)\
**Post date:** [December 16, 2009, 7:51am UTC](https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264/2 "2009-12-16T07:51:22Z")

</div>

bwv549 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> output  
> Thanks!  
> Hi,

I don’t know about pymol, but some apps do not respond well to having  
its stdin closed on it with close\_write. pipe.read should just block  
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

IO.popen(“pymol -cq -p”, ‘w+’) do |pipe|  
pipe.puts “load file.pdb, mymodel\n”  
pipe.puts “run my\_script.py”  
pipe.puts “my\_script mymodel”

```
# IO.select([pipe]) # you shouldn't need to uncomment this...

output = pipe.read

```

end

---

<div class="post-metadata">

**Author:** ![bwv549](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/ac8455/32.png) [@bwv549](https://www.ruby-forum.com/u/bwv549)\
**Post date:** [December 16, 2009, 8:46am UTC](https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264/3 "2009-12-16T08:46:59Z")

</div>

bwv549 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > > pipe.puts “my\_script mymodel”  
> > > work. Anybody know how to solve this? I’m happy to use a different
> 
> both of those hang indefinitely. It doesn’t hang if I add the line:
> 
> pipe.puts “quit”
> 
> but that doesn’t give me all the output either. So maybe pymol  
> behaves differently than many commandline programs?

Even IO.select([pipe]) hangs indefinitely? Oh, I guess pymol  
expects an explicit quit of some sort… What if you did:

IO.select([pipe])  
pipe.readpartial(whatever\_size\_you\_are\_comfortable\_with)

> Here is what I’ve worked out for the time being. It opens a thread  
> for reading the output and asks if there is any output every 1/2  
> second. Once we go 1/2 second without output we kill the thread.

Here’s a version without threads, should work the same:

my\_string = “”  
Open3.popen3(“pymol -cq -p”) do |si, so, se|  
si.puts “load file.pdb, mymodel\n”  
si.puts “run my\_script.py\n”  
si.puts “myscript mymodel\n”

```
# await input for 0.5 seconds, will return nil and
# break the loop if there is nothing to read from so after 0.5s
while ready = IO.select([so], nil, nil, 0.5)
  # ready.first == so # in this case

  # read until the current pipe buffer is empty
  begin
    my_string << so.read_nonblock(4096)
  rescue Errno::EAGAIN
    break
  end while true
end

```

end

---

<div class="post-metadata">

**Author:** ![bwv549](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/ac8455/32.png) [@bwv549](https://www.ruby-forum.com/u/bwv549)\
**Post date:** [December 16, 2009, 8:15am UTC](https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264/4 "2009-12-16T08:15:23Z")

</div>

On Dec 15, 11:32 pm, Eric W. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> > sleep(5) # \<— this is what I have to do in order to get the  
> > approach if it means I can be sure I get all of my output out. –  
> > IO.popen(“pymol -cq -p”, ‘w+’) do |pipe|  
> > Eric W.  
> > both of those hang indefinitely. It doesn’t hang if I add the line:

pipe.puts “quit”

but that doesn’t give me all the output either. So maybe pymol  
behaves differently than many commandline programs?

Here is what I’ve worked out for the time being. It opens a thread  
for reading the output and asks if there is any output every 1/2  
second. Once we go 1/2 second without output we kill the thread.

my\_string = “”  
Open3.popen3(“pymol -cq -p”) do |si, so, se|  
si.puts “load file.pdb, mymodel\n”  
si.puts “run my\_script.py\n”  
si.puts “myscript mymodel\n”

forstdout = Thread.new do  
Thread.current[‘lines’] = []  
while line = so.gets  
Thread.current[‘lines’] \<\< line  
end  
end

past\_size = -1  
loop do  
sleep(0.5)  
current\_size = forstdout[‘lines’].size  
break if current\_size == past\_size  
past\_size = current\_size  
end  
my\_string = forstdout[‘lines’]  
forstdout.kill  
end

This is not the fastest method, but it does seem to work OK. I would  
still love to hear other ideas.

---

<div class="post-metadata">

**Author:** ![bwv549](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/ac8455/32.png) [@bwv549](https://www.ruby-forum.com/u/bwv549)\
**Post date:** [December 16, 2009, 9:06am UTC](https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264/5 "2009-12-16T09:06:18Z")

</div>

2009/12/16 bwv549 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> > > pipe.puts “my\_script mymodel”  
> > > work. Anybody know how to solve this? I’m happy to use a different
> > 
> > –  
> > Eric W.
> 
> both of those hang indefinitely. It doesn’t hang if I add the line:
> 
> pipe.puts “quit”
> 
> but that doesn’t give me all the output either. So maybe pymol  
> behaves differently than many commandline programs?

What does the documentation of “pymol” say?

> forstdout = Thread.new do  
> Thread.current[‘lines’] = []  
> while line = so.gets  
> Thread.current[‘lines’] \<\< line  
> end  
> end

You don’t need the thread local for this to work. Note there is also  
Thread#value!

How does this work:

output = IO.popen(“pymol -cq -p”, ‘w+’) do |pipe|  
reader = Thread.new { pipe.to\_a }

pipe.puts “load file.pdb, mymodel”  
pipe.puts “run my\_script.py”

# this command will generate a whole bunch of output to stdout

pipe.puts “my\_script mymodel”

pipe.close\_write  
reader.value  
end

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![bwv549](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/ac8455/32.png) [@bwv549](https://www.ruby-forum.com/u/bwv549)\
**Post date:** [December 17, 2009, 7:49am UTC](https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264/6 "2009-12-17T07:49:57Z")

</div>

On 17.12.2009 06:40, bwv549 wrote:

> @Robert: I’ve looked through a lot of pymol docs and haven’t found  
> anything on it yet, will post if I do.
> 
> The solution you post looks very intriguing. It runs fine, but closes  
> before I get back the output I need.

Frankly, I don’t see how this can be: the reader thread should read  
_everthing_ that comes out of the pipe and only stops if there is EOF.  
The only explanation I have so far is that maybe some CTRL-D or other  
EOF character is sent through the pipe making the reader think it has  
reached EOF.

> It seems like there must be some  
> kind of a pause to let pymol start spitting out output. I may play  
> around with it some to see if I can get it working.

Even if there is a pause the reader must wait. There must be something  
else happening. Maybe try to do it on the shell and use ‘od’ to see  
what is sent across the pipe.

> @Eric: this solution works nicely and returns all the expected  
> output. I will be using it unless something better comes along.

Good that you have a working solution. However, what you state worries  
me a bit. I believe we might not have identified the root cause of the  
issue.

> Thanks to both of you for your excellent input on this. --John

You’re welcome!

Kind regards

robert

---

<div class="post-metadata">

**Author:** ![bwv549](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/ac8455/32.png) [@bwv549](https://www.ruby-forum.com/u/bwv549)\
**Post date:** [December 17, 2009, 7:08am UTC](https://www.ruby-forum.com/t/working-with-a-slow-pipe-io-popen/180264/7 "2009-12-17T07:08:59Z")

</div>

@Robert: I’ve looked through a lot of pymol docs and haven’t found  
anything on it yet, will post if I do.

The solution you post looks very intriguing. It runs fine, but closes  
before I get back the output I need. It seems like there must be some  
kind of a pause to let pymol start spitting out output. I may play  
around with it some to see if I can get it working.

@Eric: this solution works nicely and returns all the expected  
output. I will be using it unless something better comes along.

Thanks to both of you for your excellent input on this. --John