# Thread reading from a pipe blocks other threads, why? **URL:** \ **Category:** Ruby\ **Created:** [March 6, 2007, 9:17am UTC](https://www.ruby-forum.com/t/thread-reading-from-a-pipe-blocks-other-threads-why/87829 "2007-03-06T09:17:00Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Anders\_Lindgren](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/c37758/32.png) [@Anders\_Lindgren](https://www.ruby-forum.com/u/Anders_Lindgren)\ **Post date:** [March 6, 2007, 9:17am UTC](https://www.ruby-forum.com/t/thread-reading-from-a-pipe-blocks-other-threads-why/87829/1 "2007-03-06T09:17:00Z")
Hi! I’m implementing an application where the main Ruby program spawns off a number of external programs using pipes. Each pipe is handled by its own thread. Well, so far so good… Unfortunately, I’m having problems since when one thread calls “pipe.each\_line” the other (!) threads hang. Well, if I add a “Process.waitpid” call before the call to pipe.each\_line then the external command will sooner or later hang since noone consumes anything in the pipe. To exemplify, below are two ruby programs. The first, lotsoflines.rb, will print so much output that the pipe will be filles (in the real world this is typically not written in Ruby). The second, test.rb, without the “waitpid” will block the thread printing “BG Thread”; with the “waitpid” line the background process will never terminate. So, what I would need is a way to read from the pipe _without blocking other threads_, or a wait until there is data to read, or similar. All suggestions are welcome! Oh, btw, I need this to work both under win32 and unix. ``` -- Anders Lindgren ``` ------------------------------------ lotsoflines.rb ARGV[0].to\_i.times do puts “A LINE” end sleep 2 ------------------------------------ test.rb cmd = “ruby lotsoflines.rb 300” Thread.new do while true sleep 0.1 puts “BG Thread” end end puts “Before popen” IO.popen(cmd) do |pipe| puts “Inside popen” # With this line, the background thread will be blocked, with this # line the process at the other end of the pipe will hang… Process.waitpid(pipe.pid) pipe.each\_line do |x| puts “XXX:” + x end end puts “After popen” ---
**Author:** ![Anders\_Lindgren](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/c37758/32.png) [@Anders\_Lindgren](https://www.ruby-forum.com/u/Anders_Lindgren)\ **Post date:** [March 6, 2007, 3:36pm UTC](https://www.ruby-forum.com/t/thread-reading-from-a-pipe-blocks-other-threads-why/87829/2 "2007-03-06T15:36:15Z")
On Tue, 6 Mar 2007, Anders Lindgren wrote: > To exemplify, below are two ruby programs. The first, lotsoflines.rb, will > print so much output that the pipe will be filles (in the real world this is > typically not written in Ruby). The second, test.rb, without the “waitpid” > will block the thread printing “BG Thread”; with the “waitpid” line the > background process will never terminate. > > So, what I would need is a way to read from the pipe _without blocking other > threads_, or a wait until there is data to read, or similar. All suggestions > are welcome! > > Oh, btw, I need this to work both under win32 and unix. it can’t be done using popen and threads with current ruby. search the archives. this may help [http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README](http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README) gem install systemu > while true > > puts “After popen” -a ---
**Author:** ![Anders\_Lindgren](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/c37758/32.png) [@Anders\_Lindgren](https://www.ruby-forum.com/u/Anders_Lindgren)\ **Post date:** [March 6, 2007, 4:40pm UTC](https://www.ruby-forum.com/t/thread-reading-from-a-pipe-blocks-other-threads-why/87829/3 "2007-03-06T16:40:30Z")
Thanks for the reply! > it can’t be done using popen and threads with current ruby. search the > archives. Well, I did search the archive, but I didn’t get a straight answer on this. I couldn’t imagine that it would be impossible to read data from more than one source at the same time. You said “current ruby” – well, maybe there is hope for the future! I just noticed that in the weekly newsletter it was announced that Pragmatic Programmers will publish a book on Erlang. This is a language I worked on many years ago and I haven’t really missed it – until today, that is. The model it uses for building concurrent applications is really a winner. In that environment you simply specify the start function (the “main” of the thread), communication is typically performed by passing messages. The “fork” paradigm and shared variables used by Ruby is something that really should be declared as deprecated, since it really makes the program difficult to write (not to mention impossible to read). I cross my fingers that the next major version of Ruby uses a non-toy concurrent solution. > this may help > > [http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README](http://codeforpeople.com/lib/ruby/systemu/systemu-1.0.0/README) > > gem install systemu Thanks, I will take a look at it! ``` -- Anders Lindgren ``` ---
**Author:** ![Anders\_Lindgren](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/c37758/32.png) [@Anders\_Lindgren](https://www.ruby-forum.com/u/Anders_Lindgren)\ **Post date:** [March 6, 2007, 4:50pm UTC](https://www.ruby-forum.com/t/thread-reading-from-a-pipe-blocks-other-threads-why/87829/4 "2007-03-06T16:50:33Z")
On Wed, 7 Mar 2007 00:40:06 +0900, “Anders Lindgren” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > I cross my fingers that the next major version of Ruby uses a non-toy > concurrent solution. I’m working on an implementation of actors for Ruby (as part of an omnibus concurrency library). While it doesn’t eliminate shared state from the language, in principle it makes it possible to avoid in many cases. -mental