I run Resque with x ‘general’ queues. With ‘general’ I mean a queue that
may run every type of Worker. Or in the Rake command: COUNT=x QUEUE=*
rake resque:work
When a worker runs a job of a specific class, it may not run another job
of the same class simultaniously. This job has to wait until the
previous job has finished before it may start.
BTW The reason jobs may not run simultaniously is that the jobs do
extrernal HTTP-requests. Too many external HTTP-requests for one class
may occur a too heavy load for the service this class is using.
The most easy solutaion is to do just this for every class I need a
worker:
QUEUE= rake resque:work
The problem with this solution is that it consumes a lot of memory. At
the moment I have 30 different classes and it is still growing.
Another solution is to let several classes share one queue like this:
QUEUE=,, rake resque:work
The problem with this is when a job of e.g. class1 hangs, jobs of class2
and class3 won’t run either.
Exiting the job when it runs too long isn’t an option either, because I
never know how long a normal job will run.
So that’s why I’m coming to the solution I described at the start of
this post. Create an x number of workers that may run every type of job,
BUT when a job of specific type is running, the workers may not run a
second job of that type.
My problem is that I don’t know how to implement this in Resque. Any
ideas?