Worker pool - what to use?

luislavena · January 24, 2011, 8:16pm

Hey,

My requirement: A worker pool that basically renders templates to
html/pdf and sends emails.

The messages to process are currently stored in an OpenMQ queue
(accessible via stomp for ruby).

I am wondering whats the best way to create this worker pool. Initially
I thought about using resque, as most worker “frameworks” come with
their own queue… however a few concerns:

does putting msgs from one into another queue make sense?
how reliable is resque? I cannot lose mails
how to handle confirmation on the OpenMQ? if I insert the msgs into
resque

So I wondered how you would do this…
Maybe just create a lightweight pool of workers? How? I only found
existing ones having a queue… which is not really needed in this
scenario as I confirm messages on the OpenMQ…

I would appreciate any suggestions!
Thanks, Christoph

sol · January 25, 2011, 11:12am

On Mon, Jan 24, 2011 at 8:16 PM, Christoph B. [email protected]
wrote:

their own queue… however a few concerns:

I would appreciate any suggestions!

I don’t know stomp and I have only rudimentary knowledge
of OpenMQ,

It seems the simplest solution would be to create a number of Ruby
processes that each individually fetch data from your OpenMQ one after
another and process it. You could add a controlling master process
that restarts child processes if they die. Sample:

gist.github.com

https://gist.github.com/rklemme/794746

work.rb

require 'set'

LOCK = Mutex.new

num = Integer(ARGV.shift || 4)
processes = Set.new

Signal.trap "INT" do |s|
  $stderr.puts "Shutdown"

This file has been truncated. show original

Cheers

robert

sol · January 25, 2011, 6:46pm

On Jan 25, 2011, at 3:11 AM, Robert K. wrote:

I thought about using resque, as most worker “frameworks” come with
scenario as I confirm messages on the OpenMQ…

Sample work distribution across processes · GitHub

Here is an example that will spawn 7 worker process and connect to a
beanstalkd job queue. Replace beanstalkd with an OpenMQ connection if
you want. However beanstalkd is very simple, reliable, requeues jobs if
they fail, etc.

github.com

TwP/servolux/blob/master/examples/beanstalk.rb

# Preforking Beanstalkd job runner using Servolux.
#
# In this example, we prefork 7 processes each of which connect to our
# Beanstalkd queue and then wait for jobs to process. We are using a module so
# that we can connect to the beanstalk queue before executing and then
# disconnect from the beanstalk queue after exiting. These methods are called
# exactly once per child process.
#
# A variation on this is to load source code in the before_executing method
# and initialize an object that will process jobs. This is advantageous because
# now you can send SIGHUP to a child process and it will restart, loading your
# Ruby libraries before executing. Now you can do a rolling deploy of new
# code.
#
#   def before_executing
#     Kernel.load '/your/source/code.rb'
#     @job_runner = Your::Source::Code::JobRunner.new
#   end
# --------

This file has been truncated. show original

http://kr.github.com/beanstalkd/

Blessings,
TwP

sol · January 30, 2011, 10:12pm

Hey, thanks a lot for the input!

I will give those examples a try, looks pretty easy.
I also thought about just creating a thread pool.

One more thing:
How would you wrap this into a daemon, e.g. to have an executable with
start/stop etc commands?

I remember there was something like this, saw an example of a self
contained sinatra app, but I don’t remember it anymore…

I would need that for monitoring with monit

Thanks for your help,
Christoph