Forum: Ruby Thread reading from a pipe blocks other threads, why?

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
Anders Lindgren (Guest)
on 2007-03-06 10:17
(Received via mailing list)
Hi!

I'm implementing an application where the main Ruby program spawns off
a number of external programs using pipes. Each pipe is handled by its
own thread. Well, so far so good... Unfortunately, I'm having problems
since when one thread calls "pipe.each_line" the other (!) threads
hang. Well, if I add a "Process.waitpid" call before the call to
pipe.each_line then the external command will sooner or later hang
since noone consumes anything in the pipe.

To exemplify, below are two ruby programs. The first, lotsoflines.rb,
will print so much output that the pipe will be filles (in the real
world this is typically not written in Ruby). The second, test.rb,
without the "waitpid" will block the thread printing "BG Thread"; with
the "waitpid" line the background process will never terminate.

So, what I would need is a way to read from the pipe *without blocking
other threads*, or a wait until there is data to read, or similar. All
suggestions are welcome!

Oh, btw, I need this to work both under win32 and unix.

    -- Anders Lindgren

------------------------------------ lotsoflines.rb
ARGV[0].to_i.times do puts "A LINE" end
sleep 2
------------------------------------ test.rb
cmd = "ruby lotsoflines.rb 300"

Thread.new do
  while true
    sleep 0.1
    puts "BG Thread"
  end
end

puts "Before popen"

IO.popen(cmd) do |pipe|

  puts "Inside popen"

  # With this line, the background thread will be blocked, with this
  # line the process at the other end of the pipe will hang...

  Process.waitpid(pipe.pid)

  pipe.each_line do |x|
    puts "XXX:" + x
  end
end

puts "After popen"
unknown (Guest)
on 2007-03-06 16:36
(Received via mailing list)
On Tue, 6 Mar 2007, Anders Lindgren wrote:

> To exemplify, below are two ruby programs. The first, lotsoflines.rb, will
> print so much output that the pipe will be filles (in the real world this is
> typically not written in Ruby). The second, test.rb, without the "waitpid"
> will block the thread printing "BG Thread"; with the "waitpid" line the
> background process will never terminate.
>
> So, what I would need is a way to read from the pipe *without blocking other
> threads*, or a wait until there is data to read, or similar. All suggestions
> are welcome!
>
> Oh, btw, I need this to work both under win32 and unix.

it can't be done using popen and threads with current ruby.  search the
archives.

this may help

   http://codeforpeople.com/lib/ruby/systemu/systemu-...

gem install systemu


>  while true
>
> puts "After popen"
>
>

-a
Anders Lindgren (Guest)
on 2007-03-06 17:40
(Received via mailing list)
Thanks for the reply!

> it can't be done using popen and threads with current ruby.  search the
> archives.

Well, I did search the archive, but I didn't get a straight answer on
this. I couldn't imagine that it would be impossible to read data from
more than one source at the same time.

You said "current ruby" -- well, maybe there is hope for the future!

I just noticed that in the weekly newsletter it was announced that
Pragmatic Programmers will publish a book on Erlang. This is a
language I worked on many years ago and I haven't really missed it --
until today, that is. The model it uses for building concurrent
applications is really a winner. In that environment you simply
specify the start function (the "main" of the thread), communication
is typically performed by passing messages. The "fork" paradigm and
shared variables used by Ruby is something that really should be
declared as deprecated, since it really makes the program difficult to
write (not to mention impossible to read).

I cross my fingers that the next major version of Ruby uses a non-toy
concurrent solution.


> this may help
>
>    http://codeforpeople.com/lib/ruby/systemu/systemu-...
>
> gem install systemu

Thanks, I will take a look at it!

    -- Anders Lindgren
MenTaLguY (Guest)
on 2007-03-06 17:50
(Received via mailing list)
On Wed, 7 Mar 2007 00:40:06 +0900, "Anders Lindgren" 
<removed_email_address@domain.invalid>
wrote:

> I cross my fingers that the next major version of Ruby uses a non-toy
> concurrent solution.

I'm working on an implementation of actors for Ruby (as part of an
omnibus concurrency library).  While it doesn't eliminate shared state
from the language, in principle it makes it possible to avoid in many
cases.

-mental
This topic is locked and can not be replied to.