Interacting with a long-lived external process from ruby?

Ruby
#1

Hey Folks,

I have a long-lived process that I am trying to get ruby to interact
with over time.

Essentially, I’m working in an environment that requires a compiler
that was written in Java (and is naturally quite slow). There is an
additional Java-based, shell application that sits in front of the
compiler and allows for exponentially faster, incremental compilation.

I am building a set of project creation and build tools that hide
these aspects of development in order to make it much easier to get up
and running with a new project and to grow existing projects.

http://code.google.com/p/projectsprouts/

Everything - and I mean everything - is working adequately, except my
front end to this compiler-front-end.

Here is what I’m doing:

I’m using what Stephen Wong posted (http://blade.nagaokaut.ac.jp/cgi-
bin/scat.rb/ruby/ruby-talk/118672) as a foundation.

  1. Initiate an external process from Ruby using fork (I have also
    tried execute, IO.popen, Open3.popen3, system and ``, none of which
    seemed to work as easily as what Stephen provided).

  2. Redirect $stdin, $stdout and $stderr from this forked process to
    three new IO streams.

  3. write to one stream and read from the others so that the user can
    see what’s going on as Ruby works on this forked process.

  4. I am able to successfully call write_stream.write(‘msg’) and the
    process receives this message and responds as expected - if and only
    if - I do not call read_stream.read at any time.

  5. The only way I have been able to get the read_stream to not break
    the write operation, is if I call write_stream.close after writing. In
    this case, everything works perfectly except I cannot perform any
    additional write operations - thereby rendering the whole long-lived
    process worthless.

The questions:

  1. Does write_stream.close actually kill the entire forked process? I
    can’t really tell what’s going on in there. It looks like I should be
    able to simply reopen the write_stream against the existing process,
    but I can’t figure out how to do that.

  2. Is there some way to ‘reopen’ an IO stream that has been closed
    when the stream was originally attached inside of a fork statement?
    Please see the reopen calls in Stephen’s code:
    http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-talk/118672, I
    can’t seem to store references to those $std[in/out/err] streams
    outside of the fork closure scope. It seems like reopening the write
    stream after closing it would work - if the process is still around.

  3. If the forked process is a Java shell application, could it expect
    or transmit some non standard end of line character for read/write
    operations?

  4. Is it typical for a forked process to block all previous or
    subsequent write operations when read is called?

  5. What tools would you use to figure out what is going on here? Since
    I’m messing with my shell $std[in/out/err] streams, and working in
    forks and threads, I’m having trouble debugging - is there some tool
    that will tell me what is going on in my computer?

Any help is greatly appreciated!

Thanks,

Luke B.
http://www.asserttrue.com

#2

OK - I figured it out.

The process I am interacting with has similar behavior to the irb
shell application.

What I didn’t realize was that when the process writes to the stream
that I’m reading on, the prompt line does not include an EOL
character. This was blocking the read stream until either
write_stream.close or other output was forced.

Essentially, I just decorated my process and implemented a read method
as follows:

----------------8<-----------------

def read
line = ‘’
response = ‘’
while(!r.eof?)
char = r.getc.chr
line += char
if(line == ‘(fcsh)’)
response += “[PROMPT] #{line}”
break
end
if(char == “\n”)
response += “[FCSH] #{line}”
line = ‘’
end
end
return response
end

---------------->8-----------------

IO.getc being the key component as it returns every byte that is
written to the stream, without regard for EOL characters.

This read method behaves similarly to the IO.read method except it
will return when the prompt is encountered.

Also -

As it turns out Open3.popen3 works just fine and is much simpler than
what I was using before.

Thanks,

Luke B.
http://www.asserttrue.com

#3

One quick note about the Pick-axe book and ruby docs…

The example on page 148 for IO.popen:

pig = IO.popen("/usr/local/bin/pig", “w+”)
pig.puts “ice cream after they go to bed”
pig.close_write
puts pig.gets

Includes the following comment:

“… it turns out that the pig program doesn’t flush the output it
writes. Our original attempt at this example, which had a pig.puts
followed by a pig.gets, hung forever.”

I don’t have the pig latin program installed on my machine, so I can’t
be sure about this, but I suspect Dave may have been running into
exactly the same issue I found here. It looks like the pig program
doesn’t return an EOL character.