Forkoff-0.0.1

NAME

forkoff

SYNOPSIS

brain-dead simple parallel processing for ruby

URI

http://rubyforge.org/projects/codeforpeople

INSTALL

gem install forkoff

HISTORY

0.0.1

  • updated to use producer threds pushing onto a SizedQueue for each
    consumer
    channel. in this way the producers do not build up a massize
    parllel data
    structure but provide data to the consumers only as fast as they
    can fork
    and proccess it. basically for a 4 process run you’ll end up
    with 4
    channels of size 1 between 4 produces and 4 consumers, each
    consumer is a
    thread popping of jobs, forking, and yielding results.

  • removed use of Queue for capturing the output. now it’s simply
    an array
    of arrays which removed some sync overhead.

  • you can configure the number of processes globally with

    Forkoff.default[‘proccess’] = 4

  • you can now pass either an options hash

    forkoff( :processes => 2 ) …

    or plain vanilla number

    forkoff( 2 ) …

    to the forkoff call

  • default number of processes is 2, not 8

DESCRIPTION

forkoff works for any enumerable object, iterating a code block to
run in a
child process and collecting the results. forkoff can limit the
number of
child processes which is, by default, 2.

SAMPLES

<========< samples/a.rb >========>

~ > cat samples/a.rb

 #
 # forkoff makes it trivial to do parallel processing with ruby,

the following
# prints out each word in a separate process
#

   require 'forkoff'

   %w( hey you ).forkoff!{|word| puts "#{ word } from

#{ Process.pid }"}

~ > ruby samples/a.rb

 hey from 19511
 you from 19512

<========< samples/b.rb >========>

~ > cat samples/b.rb

 #
 # for example, this takes only 4 seconds or so to complete (8

iterations
# running in two processes = twice as fast)
#

   require 'forkoff'

   a = Time.now.to_f

   results =
     (0..7).forkoff do |i|
       sleep 1
       i ** 2
     end

   b = Time.now.to_f

   elapsed = b - a

   puts "elapsed: #{ elapsed }"
   puts "results: #{ results.inspect }"

~ > ruby samples/b.rb

 elapsed: 4.19796895980835
 results: [0, 1, 4, 9, 16, 25, 36, 49]

<========< samples/c.rb >========>

~ > cat samples/c.rb

 #
 # forkoff does *NOT* spawn processes in batches, waiting for each

batch to
# complete. rather, it keeps a certain number of processes busy
until all
# results have been gathered. in otherwords the following will
ensure that 3
# processes are running at all times, until the list is complete.
note that
# the following will take about 3 seconds to run (3 sets of 3 @ 1
second).
#

 require 'forkoff'

 pid = Process.pid

 a = Time.now.to_f

 pstrees =
   %w( a b c d e f g h i ).forkoff! :processes => 3 do |letter|
     sleep 1
     { letter => ` pstree -l 2 #{ pid } ` }
   end


 b = Time.now.to_f

 puts
 puts "pid: #{ pid }"
 puts "elapsed: #{ b - a }"
 puts

 require 'yaml'

 pstrees.each do |pstree|
   y pstree
 end

~ > ruby samples/c.rb

 pid: 19526
 elapsed: 3.39831399917603

 ---
 a: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |-+- 19527 ahoward ruby -Ilib samples/c.rb
    |-+- 19528 ahoward ruby -Ilib samples/c.rb
    \-+- 19529 ahoward ruby -Ilib samples/c.rb

 ---
 b: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |-+- 19527 ahoward ruby -Ilib samples/c.rb
    |-+- 19528 ahoward ruby -Ilib samples/c.rb
    \-+- 19529 ahoward ruby -Ilib samples/c.rb

 ---
 c: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |-+- 19527 ahoward (ruby)
    |-+- 19528 ahoward (ruby)
    \-+- 19529 ahoward ruby -Ilib samples/c.rb

 ---
 d: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |-+- 19536 ahoward ruby -Ilib samples/c.rb
    |-+- 19537 ahoward (ruby)
    \--- 19538 ahoward ruby -Ilib samples/c.rb

 ---
 e: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |-+- 19536 ahoward ruby -Ilib samples/c.rb
    |-+- 19537 ahoward ruby -Ilib samples/c.rb
    \--- 19538 ahoward ruby -Ilib samples/c.rb

 ---
 f: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |--- 19536 ahoward (ruby)
    |-+- 19538 ahoward ruby -Ilib samples/c.rb
    \--- 19543 ahoward ruby -Ilib samples/c.rb

 ---
 g: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |--- 19543 ahoward (ruby)
    |-+- 19546 ahoward ruby -Ilib samples/c.rb
    \--- 19547 ahoward ruby -Ilib samples/c.rb

 ---
 h: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |-+- 19543 ahoward ruby -Ilib samples/c.rb
    |--- 19546 ahoward ruby -Ilib samples/c.rb
    \--- 19547 ahoward ruby -Ilib samples/c.rb

 ---
 i: |
   -+- 19526 ahoward ruby -Ilib samples/c.rb
    |-+- 19543 ahoward ruby -Ilib samples/c.rb
    |-+- 19546 ahoward ruby -Ilib samples/c.rb
    \-+- 19547 ahoward ruby -Ilib samples/c.rb

a @ http://codeforpeople.com/

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs