Working with a slow pipe (IO.popen)

Ruby
1

I’m doing some work with pymol and having trouble getting all the
pymol output out:

I open it in quiet/commandline mode with a pipe:

IO.popen(“pymol -cq -p”, ‘w+’) do |pipe|
pipe.puts “load file.pdb, mymodel\n”
pipe.puts “run my_script.py”

this command will generate a whole bunch of output to stdout

pipe.puts “my_script mymodel”
sleep(5) # <— this is what I have to do in order to get the
output
pipe.close_write
output = pipe.read
end

What is annoying is that unless I sleep for 2-5 seconds I don’t get
output or (even worse) my output is cut off. There has to be a way
besides arbitrarily sleeping to ensure that the command is finished.
I’ve tried things like piping in “quit”, but that doesn’t seem to
work. Anybody know how to solve this? I’m happy to use a different
approach if it means I can be sure I get all of my output out. –
Thanks!

2

bwv549 [email protected] wrote:

output
Thanks!
Hi,

I don’t know about pymol, but some apps do not respond well to having
its stdin closed on it with close_write. pipe.read should just block
until you get output, otherwise put an IO.select([pipe) in front of it.

Does this work?

IO.popen(“pymol -cq -p”, ‘w+’) do |pipe|
pipe.puts “load file.pdb, mymodel\n”
pipe.puts “run my_script.py”
pipe.puts “my_script mymodel”

# IO.select([pipe]) # you shouldn't need to uncomment this...

output = pipe.read

end

3

bwv549 [email protected] wrote:

pipe.puts “my_script mymodel”
work. Anybody know how to solve this? I’m happy to use a different

both of those hang indefinitely. It doesn’t hang if I add the line:

pipe.puts “quit”

but that doesn’t give me all the output either. So maybe pymol
behaves differently than many commandline programs?

Even IO.select([pipe]) hangs indefinitely? Oh, I guess pymol
expects an explicit quit of some sort… What if you did:

IO.select([pipe])
pipe.readpartial(whatever_size_you_are_comfortable_with)

Here is what I’ve worked out for the time being. It opens a thread
for reading the output and asks if there is any output every 1/2
second. Once we go 1/2 second without output we kill the thread.

Here’s a version without threads, should work the same:

my_string = “”
Open3.popen3(“pymol -cq -p”) do |si, so, se|
si.puts “load file.pdb, mymodel\n”
si.puts “run my_script.py\n”
si.puts “myscript mymodel\n”

# await input for 0.5 seconds, will return nil and
# break the loop if there is nothing to read from so after 0.5s
while ready = IO.select([so], nil, nil, 0.5)
  # ready.first == so # in this case

  # read until the current pipe buffer is empty
  begin
    my_string << so.read_nonblock(4096)
  rescue Errno::EAGAIN
    break
  end while true
end

end

4

On Dec 15, 11:32 pm, Eric W. [email protected] wrote:

sleep(5) # <— this is what I have to do in order to get the
approach if it means I can be sure I get all of my output out. –
IO.popen(“pymol -cq -p”, ‘w+’) do |pipe|
Eric W.
both of those hang indefinitely. It doesn’t hang if I add the line:

pipe.puts “quit”

but that doesn’t give me all the output either. So maybe pymol
behaves differently than many commandline programs?

Here is what I’ve worked out for the time being. It opens a thread
for reading the output and asks if there is any output every 1/2
second. Once we go 1/2 second without output we kill the thread.

my_string = “”
Open3.popen3(“pymol -cq -p”) do |si, so, se|
si.puts “load file.pdb, mymodel\n”
si.puts “run my_script.py\n”
si.puts “myscript mymodel\n”

forstdout = Thread.new do
Thread.current[‘lines’] = []
while line = so.gets
Thread.current[‘lines’] << line
end
end

past_size = -1
loop do
sleep(0.5)
current_size = forstdout[‘lines’].size
break if current_size == past_size
past_size = current_size
end
my_string = forstdout[‘lines’]
forstdout.kill
end

This is not the fastest method, but it does seem to work OK. I would
still love to hear other ideas.

5

2009/12/16 bwv549 [email protected]:

pipe.puts “my_script mymodel”
work. Anybody know how to solve this? I’m happy to use a different

–
Eric W.

both of those hang indefinitely. It doesn’t hang if I add the line:

pipe.puts “quit”

but that doesn’t give me all the output either. So maybe pymol
behaves differently than many commandline programs?

What does the documentation of “pymol” say?

forstdout = Thread.new do
Thread.current[‘lines’] = []
while line = so.gets
Thread.current[‘lines’] << line
end
end

You don’t need the thread local for this to work. Note there is also
Thread#value!

How does this work:

output = IO.popen(“pymol -cq -p”, ‘w+’) do |pipe|
reader = Thread.new { pipe.to_a }

pipe.puts “load file.pdb, mymodel”
pipe.puts “run my_script.py”

this command will generate a whole bunch of output to stdout

pipe.puts “my_script mymodel”

pipe.close_write
reader.value
end

Kind regards

robert

6

On 17.12.2009 06:40, bwv549 wrote:

@Robert: I’ve looked through a lot of pymol docs and haven’t found
anything on it yet, will post if I do.

The solution you post looks very intriguing. It runs fine, but closes
before I get back the output I need.

Frankly, I don’t see how this can be: the reader thread should read
everthing that comes out of the pipe and only stops if there is EOF.
The only explanation I have so far is that maybe some CTRL-D or other
EOF character is sent through the pipe making the reader think it has
reached EOF.

It seems like there must be some
kind of a pause to let pymol start spitting out output. I may play
around with it some to see if I can get it working.

Even if there is a pause the reader must wait. There must be something
else happening. Maybe try to do it on the shell and use ‘od’ to see
what is sent across the pipe.

@Eric: this solution works nicely and returns all the expected
output. I will be using it unless something better comes along.

Good that you have a working solution. However, what you state worries
me a bit. I believe we might not have identified the root cause of the
issue.

Thanks to both of you for your excellent input on this. --John

You’re welcome!

Kind regards

robert

7

@Robert: I’ve looked through a lot of pymol docs and haven’t found
anything on it yet, will post if I do.

The solution you post looks very intriguing. It runs fine, but closes
before I get back the output I need. It seems like there must be some
kind of a pause to let pymol start spitting out output. I may play
around with it some to see if I can get it working.

@Eric: this solution works nicely and returns all the expected
output. I will be using it unless something better comes along.

Thanks to both of you for your excellent input on this. --John