Options for running code in parallel

Martin_DeMello · July 1, 2008, 8:35pm

Our testing department wants a small tool that can make a number of
http calls simultaneously to the same server, the basic logic being

-----------------------------------------------------------------

calls = [“call1”…“calln”].map {|x| “http://server/#{x}”}

calls.each do |x|
OnSignal(START) do
http_send call(x)
log_result
end
end

SendSignal(START)

-----------------------------------------------------------------

What are the options for doing this cleanly and robustly? I haven’t
been keeping up with the recent work people have been doing on
distributed/parallel code in ruby, but I get the general impression
there’s been a lot of activity in this area.

martin

Martin_DeMello · July 1, 2008, 9:06pm

Martin DeMello wrote:

Our testing department wants a small tool that can make a number of
http calls simultaneously to the same server

Before writing your own, you may want to take a look at existing tools
like:

If you still want to build your own, please review the last couple days
of ruby-talk mailing list archives because there’s been some spirited
debate about fork/thread that may be useful.

-igal

Martin_DeMello · July 1, 2008, 8:54pm

On Jul 1, 2008, at 11:52 AM, Martin DeMello wrote:

end
martin

give this a whack:

cfp:~> cat a.rb
module Threadify
require ‘thread’

@threads = 8
@abort_on_exception = true

class << self
attr_accessor :threads
attr_accessor :abort_on_exception
end
end

module Enumerable
def threadify opts = {}, &block
opts = {:threads => opts} if Numeric === opts
threads = Integer(opts[:threads] || opts[‘threads’] ||
Threadify.threads)
done = Object.new.freeze
jobs = Queue.new

 each_with_index{|elem, i| jobs.push [elem, i]}
 threads.times{ jobs.push done} # mark the end

 consumers = Array.new threads

 threads.times do |i|
   consumers[i] = Thread.new do
     this = Thread.current
     this.abort_on_exception = Threadify.abort_on_exception
     loop{
       job = jobs.pop
       this.exit if job == done
       args = job.first
       jobs << (job << block.call(*args))
     }
   end
 end

 consumers.map{|t| t.join}
 jobs.push done

 ret = []
 while((job = jobs.pop) != done)
   elem, i, value = job
   ret[i] = value
 end
 ret

end
end

class Thread
def Thread.ify enumerable, *args, &block
enumerable.send :threadify, *args, &block
end
end

if FILE == $0
require ‘open-uri’
require ‘yaml’

uris = %w( http://google.com http://yahoo.com http://rubyforge.org/
http://ruby-lang.org
)

Thread.ify uris, :threads => 3 do |uri|
body = open(uri){|pipe| pipe.read}
y uri => body.size
end
end