Slow reading output from Open3.popen3?

I’ve never tried using this function before, but I am now using as part
of a code compilation build script (using ‘make’ on UNIX).

It looks something like this:

==================================

stdin, stdout, stderr = Open3.popen3( ‘make codename’ )
stdin.close

puts “Reading STDOUT”
outfile.puts stdout.read
stdout.close

==================================

I’m trying to get the output from the ‘make’ command (which is quite
large), so I can dump it to a file for later diagnostics.

However, when it gets to the stdout.read, it seems to hang (or at least
take so long I’ve never seen it finish).

Any thoughts on why this is happening? Is there a better approach for
me to try?

Thanks in advance.

On 05/14/2013 05:38 PM, Thomas L. wrote:

puts “Reading STDOUT”
outfile.puts stdout.read
stdout.close

==================================

I’m trying to get the output from the ‘make’ command (which is quite
large), so I can dump it to a file for later diagnostics.

There are a couple ways to tackle what you want to do, but one big
problem with what you’re doing here is that you’re trying to read all
of the output from stdout into memory at once. I’m not sure how big
“quite large” is, but you should probably do this line by line or in
limited block sizes instead.

stdout.each_line do |line|
outfile.write(line)
end

OR

while (data = stdout.read(1024)).size > 0
outfile.write(data)
end

However, when it gets to the stdout.read, it seems to hang (or at least
take so long I’ve never seen it finish).

Any thoughts on why this is happening? Is there a better approach for
me to try?

You never read from stderr in your example. I’m wondering if the
subprocess isn’t actually blocking while trying to write something to
stderr. Because you never read from it, the buffer for the pipe
connected to stderr may fill up. If that were to happen while the
subprocess was writing to stderr, the subprocess would block while
trying to write and never get to close its end of the stdout pipe and
exit. Thus your read from stdout will block forever.

You need to read from both stdout and stderr in your script to avoid
this problem; however, this can get complicated since you need to avoid
blocking while reading from either one of them. You could also skip
stderr entirely and just use IO.popen, in which case the stderr output
will go to the terminal or wherever your script’s stderr was wired to
go. Finally, you could use IO.popen while redirecting the subprocess’
stderr to its stdout so that you can read both streams at once:

IO.popen(‘make codename 2>&1’, ‘r’) do |pipe|
pipe.each_line do |line|
outfile.write(line)
end
end

Given your stated goal of sending the output of make to a file for later
diagnostics, you would probably be better off simply redirecting the
output directly to a file and skipping ruby entirely:

bash$ make codename >/path/to/make.log 2>&1

The above when run in the bash shell would send both the stdout and
stderr into the file /path/to/make.log. If you still wanted to see the
output on screen at the same time as logging to the file, you can use
the tee program:

bash$ make codename 2>&1 | tee /path/to/make.log

The same thing happens here as before but you’ll also see the output in
the terminal while make runs.

-Jeremy

On Wed, May 15, 2013 at 5:11 AM, Jeremy B. [email protected] wrote:

problem with what you’re doing here is that you’re trying to read all
of the output from stdout into memory at once. I’m not sure how big
“quite large” is, but you should probably do this line by line or in
limited block sizes instead.

stdout.each_line do |line|
outfile.write(line)
end

If you use #each_line then I would also use #puts for output because
both
are line oriented.

OR

while (data = stdout.read(1024)).size > 0
outfile.write(data)
end

That doesn’t work because #read returns nil at EOF. You just need

while data = stdout.read(1024)
outfile.write(data)
end

OR, a tad more efficient

data = “”

while stdout.read(1024, data)
outfile.write(data)
end

subprocess was writing to stderr, the subprocess would block while
trying to write and never get to close its end of the stdout pipe and
exit. Thus your read from stdout will block forever.

Right!

You need to read from both stdout and stderr in your script to avoid

end
end

I wouldn’t want to do that because then I cannot differenciate between
regular and error output.

Given your stated goal of sending the output of make to a file for later
diagnostics, you would probably be better off simply redirecting the
output directly to a file and skipping ruby entirely:

bash$ make codename >/path/to/make.log 2>&1

Agree.

The above when run in the bash shell would send both the stdout and
stderr into the file /path/to/make.log. If you still wanted to see the
output on screen at the same time as logging to the file, you can use
the tee program:

bash$ make codename 2>&1 | tee /path/to/make.log

The same thing happens here as before but you’ll also see the output in
the terminal while make runs.

And here’s a solution using open3:

require ‘open3’

Open3.popen3( %w{make codename} ) do |stdin, stdout, stderr, t|
stdin.close
err_thr = Thread.new { IO.copy_stream(stderr, outfile) }
puts “Reading STDOUT”
IO.copy_stream(stdout, outfile)
err_thr.join
end

Note: I also used an Array for the make command invocation because that
avoids parsing issues in the shell because it omits the shell altogether
(see Process.spawn).

You might also prefer a more line based approach

def copy_lines(str_in, str_out)
str_in.each_line {|line| str_out.puts line}
end

Open3.popen3( ‘make codename’ ) do |stdin, stdout, stderr, t|
stdin.close
err_thr = Thread.new { copy_lines(stderr, $stderr) }
puts “Reading STDOUT”
copy_lines(stdout, $stdout)
err_thr.join
end

Kind regards

robert

Thanks, guys. I found a workaround (i.e. redirect), but I think I’ll
play around to see if the suggestions give me what I want. It is true,
the amount of data I’m reading may be quite large, which would cause the
problem.

On Thu, May 23, 2013 at 4:24 AM, Thomas L.
[email protected]wrote:

Thanks, guys. I found a workaround (i.e. redirect), but I think I’ll
play around to see if the suggestions give me what I want. It is true,
the amount of data I’m reading may be quite large, which would cause the
problem.

The problem you will be facing most probably is a deadlock. Your “make”
or
any process started by it may write to stdout and to stderr and the
process
will be put to sleep if the buffer of stderr fills up (from which you do
not read) as Jeremy explained already. That will also block any other
activity of the process unless they use infinite buffering internally
and
asynchronous output (which there is no reason to do so it’s unlikely).

Cheers

robert