Some six month ago I was investigating how it was possible to emulate
UNIX pipes in Ruby with fork so every command was running on a core of
its own: Resolving pipe deadlock - Ruby - Ruby-Forum
After finding a solution that appeared to work well I proceeded with my
project, but now that I am starting to test on large data sets things
are turning ugly. The forked version is unstable (Ruby crash
experienced), prone to hanging, and slow (processes in top show CPU core
load ~90%, I was expecting 100%). The unforked version is stable and
apparently fast, however, it only uses a single core. I suspect the
issue is blocked pipes, but debugging this is a mess :o(
For anyone interested the code is here:
and in particular line 101 ff of this file holds the problem:
https://github.com/maasha/biopieces/blob/master/lib/biopieces/pipeline.rb
For a real life test do:
gem install biopieces
Get input data for testing here:
ftp://ftp_20140826_2135:[email protected]
and run this pipeline:
Cheers,
Martin