Capture output in real time

Ruby
1

I’m wondering how to capture data written to stdout (or stderr) in real
time using popen3. The problem occurs with the following sample
scripts:

sleep.rb

puts “before sleep”
sleep 3
puts “after sleep”

capture.rb

require ‘open3’

Open3.popen3(“ruby sleep.rb”) do |stdin, stdout, stderr|

Thread.new do
loop do
out = stdout.gets
puts out if out
end
end

sleep 5
end

capture.rb outputs both “before sleep” and “after sleep” after 3 seconds
have passed. When using stderr, however, output is seen in real time.

Note, I’m avoiding STDOUT.flush in sleep.rb since I won’t be able to
change the actual scripts I’ll be passing to popen3.

Thanks for any help.
Paul

2

On May 9, 2008, at 6:08 PM, Paul Winward wrote:

Note, I’m avoiding STDOUT.flush in sleep.rb since I won’t be able to
change the actual scripts I’ll be passing to popen3.

popen3 is going to be using fork/exec under the hood and should
inherit the sync setting of STDOUT, so try setting

STDOUT.sync = true

before the popen and see if that helps - untested, but i’m running out
the door…

cheers.

a @ http://codeforpeople.com/

3

Thanks for the suggestion but I’m still having the problem.

Paul

ara.t.howard wrote:

On May 9, 2008, at 6:08 PM, Paul Winward wrote:

Note, I’m avoiding STDOUT.flush in sleep.rb since I won’t be able to
change the actual scripts I’ll be passing to popen3.

popen3 is going to be using fork/exec under the hood and should
inherit the sync setting of STDOUT, so try setting

STDOUT.sync = true

before the popen and see if that helps - untested, but i’m running out
the door…

cheers.

a @ http://codeforpeople.com/

4

Another approach I had tried was to use IO.select, as shown in
http://whynotwiki.com/Ruby_/_Process_management ( attributed to Jamis
Buck). This too, produced the same delay in output, but is shown below:

Thank you for any help

def read_until(pipe, stop_at, verbose = true)
lines = []
line = “”
while result = IO.select([pipe]) #, nil, nil, 10)
next if result.empty?

c = pipe.read(1)
break if c.nil?

line << c
break if line =~ stop_at

# Start a new line?
if line[-1] == ?\n
  puts line if verbose
  lines << line
  line = ""
end

end
lines
end

Paul Winward wrote:

Thanks for the suggestion but I’m still having the problem.

Paul

ara.t.howard wrote:

On May 9, 2008, at 6:08 PM, Paul Winward wrote:

Note, I’m avoiding STDOUT.flush in sleep.rb since I won’t be able to
change the actual scripts I’ll be passing to popen3.

popen3 is going to be using fork/exec under the hood and should
inherit the sync setting of STDOUT, so try setting

STDOUT.sync = true

before the popen and see if that helps - untested, but i’m running out
the door…

cheers.

a @ http://codeforpeople.com/

5

I found a perl solution at http://www.perlmonks.org/?node_id=305812 that
makes use of pseudo tty to get line-buffered output. Unfortunately, I
can find very little documentation on how to use the ruby module PTY,
and it looks like it is no longer supported
(So pty is no longer supported? - Ruby - Ruby-Forum). Also, I wonder if a
pseudo tty can capture stderr as well
(http://unix.derkeiler.com/Newsgroups/comp.unix.programmer/2003-07/0544.html)

The perl solution is system dependent, but it worked for me on Linux
2.6.22. I did see a debian package libpty-ruby but I don’t see any
documentation and am not sure if this is still supported.

The notion of a pseudo tty is new to me and I’d appreciate any help. Is
there a ruby equivalent for the perl solution below?

Thanks

#!/usr/bin/perl -w
use strict;
use IO::Pty;

my $pty = new IO::Pty;
my $slave = $pty->slave();

my $pid = fork();
die “Couldn’t fork: $!” unless defined $pid;

if($pid){ # dup STDOUT to Pty and run external program:
$pty->close_slave();
open(STDOUT, “>&”,$pty)||die $!;
system “perl blackbox.pl”;
print “\cD”; # send ^d to end

} else { # this is your monitoring process
$pty->make_slave_controlling_terminal();
print “*$_” while <$slave>;
exit;
}
END

Paul Winward wrote:

Another approach I had tried was to use IO.select, as shown in
http://whynotwiki.com/Ruby_/_Process_management ( attributed to Jamis
Buck). This too, produced the same delay in output, but is shown below:

Thank you for any help

def read_until(pipe, stop_at, verbose = true)
lines = []
line = “”
while result = IO.select([pipe]) #, nil, nil, 10)
next if result.empty?

c = pipe.read(1)
break if c.nil?

line << c
break if line =~ stop_at

# Start a new line?
if line[-1] == ?\n
  puts line if verbose
  lines << line
  line = ""
end

end
lines
end

Paul Winward wrote:

Thanks for the suggestion but I’m still having the problem.

Paul

ara.t.howard wrote:

On May 9, 2008, at 6:08 PM, Paul Winward wrote:

Note, I’m avoiding STDOUT.flush in sleep.rb since I won’t be able to
change the actual scripts I’ll be passing to popen3.

popen3 is going to be using fork/exec under the hood and should
inherit the sync setting of STDOUT, so try setting

STDOUT.sync = true

before the popen and see if that helps - untested, but i’m running out
the door…

cheers.

a @ http://codeforpeople.com/

6

On May 10, 2008, at 12:08 AM, Paul Winward wrote:

Thanks for the suggestion but I’m still having the problem.

if the child program makes a call to setvbuf (as ruby does) there is
little you can do to alter this behaviour. otherwise the ‘normal’
approach will work:

cfp:~ > cat a.rb
r, w = IO.pipe

child = fork{ STDOUT.reopen w; w.close; system ‘echo 4 && sleep 1 &&
echo 2’ }

w.close

while(( char = r.read(1) ))
printf “%f : %s\n”, Time.now.to_f, char.inspect
end

cfp:~ > ruby a.rb
1210456103.163009 : “4”
1210456103.163153 : “\n”
1210456104.166967 : “2”
1210456104.167023 : “\n”

i might be wrong, but i really don’t think there is anything you can
do with a child program that alters the buffering mode itself.

a @ http://codeforpeople.com/

7

Thanks for the pointers. I wasn’t aware of setvbuf. I’ll look more into
pty.

ara.t.howard wrote:

On May 10, 2008, at 12:08 AM, Paul Winward wrote:

Thanks for the suggestion but I’m still having the problem.

if the child program makes a call to setvbuf (as ruby does) there is
little you can do to alter this behaviour. otherwise the ‘normal’
approach will work:

8

Hi folks.

After trying a lot of things, this worked for me on Linux:

%x{stty -icanon -echo}
key = STDIN.read(1)

I found this at

Instead of the %x{…} you could use Termios instead to reset the
‘ICANON’ flag:

http://devver.net/blog/2009/07/a-command-line-prompt-with-timeout-and-countdown/

9

On May 10, 2008, at 1:28 PM, Paul Winward wrote:

The perl solution is system dependent, but it worked for me on Linux
2.6.22. I did see a debian package libpty-ruby but I don’t see any
documentation and am not sure if this is still supported.

The notion of a pseudo tty is new to me and I’d appreciate any
help. Is
there a ruby equivalent for the perl solution below?

pty is supported and the dist has a bunch of demo code in it (ext/pty/
*) but you cannot capture stderr akaikt.

a @ http://codeforpeople.com/

10

2008/5/10 ara.t.howard [email protected]:

before the popen and see if that helps - untested, but i’m running out the
door…

IMHO that can only work if you use fork with a block. Since Paul
wants to start a new process that cannot possibly inherit buffering
settings that way. Since Ruby decides buffering mode of stdout based
on the type (i.e. if it is a terminal then sync = true otherwise not)
and the child process cannot be influenced by Paul it follows that the
only feasible way to use would be to present a tty as stdout to the
child (via some form of pseudo tty mechanism). But that may have
adverse effects because some programs change their output based on the
type of stdout as well (just think about including colorization on
ttys and omitting it on pipes).

This is the place to ask: Paul, what do you need that behavior for?
Maybe there are other mechanism to use.

Kind regards

robert