Worker pool - what to use?

Hey,

My requirement: A worker pool that basically renders templates to
html/pdf and sends emails.

The messages to process are currently stored in an OpenMQ queue
(accessible via stomp for ruby).

I am wondering whats the best way to create this worker pool. Initially
I thought about using resque, as most worker “frameworks” come with
their own queue… however a few concerns:

  • does putting msgs from one into another queue make sense?
  • how reliable is resque? I cannot lose mails
  • how to handle confirmation on the OpenMQ? if I insert the msgs into
    resque

So I wondered how you would do this…
Maybe just create a lightweight pool of workers? How? I only found
existing ones having a queue… which is not really needed in this
scenario as I confirm messages on the OpenMQ…

I would appreciate any suggestions!
Thanks, Christoph

On Mon, Jan 24, 2011 at 8:16 PM, Christoph B. [email protected]
wrote:

their own queue… however a few concerns:

I would appreciate any suggestions!

I don’t know stomp and I have only rudimentary knowledge
of OpenMQ,

It seems the simplest solution would be to create a number of Ruby
processes that each individually fetch data from your OpenMQ one after
another and process it. You could add a controlling master process
that restarts child processes if they die. Sample:

https://gist.github.com/794746

Cheers

robert

On Jan 25, 2011, at 3:11 AM, Robert K. wrote:

I thought about using resque, as most worker “frameworks” come with
scenario as I confirm messages on the OpenMQ…

https://gist.github.com/794746

Here is an example that will spawn 7 worker process and connect to a
beanstalkd job queue. Replace beanstalkd with an OpenMQ connection if
you want. However beanstalkd is very simple, reliable, requeues jobs if
they fail, etc.

https://github.com/TwP/servolux/blob/master/examples/beanstalk.rb

http://kr.github.com/beanstalkd/

Blessings,
TwP

Hey, thanks a lot for the input!

I will give those examples a try, looks pretty easy.
I also thought about just creating a thread pool.

One more thing:
How would you wrap this into a daemon, e.g. to have an executable with
start/stop etc commands?

I remember there was something like this, saw an example of a self
contained sinatra app, but I don’t remember it anymore…

I would need that for monitoring with monit

Thanks for your help,
Christoph

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs