Shared Queue / Exclusive Query Results

Rails
1

My application is going to have a work queue of scheduled tasks stored
in the DB. This work queue will be shared across multiple Rails app
servers. I need a way to ensure that no two servers get the same jobs
if multiple servers pop jobs off the top of the queue simultaneously.

Is there a fairly simple way to acheive this? Or do I need to come up
with some fancy secondary server to dispatch and marshall (serialize)
job assignments?

Thanks! - Don

2

On May 9, 2006, at 9:49 AM, Don S. wrote:

My application is going to have a work queue of scheduled tasks stored
in the DB. This work queue will be shared across multiple Rails app
servers. I need a way to ensure that no two servers get the same jobs
if multiple servers pop jobs off the top of the queue simultaneously.

Is there a fairly simple way to acheive this? Or do I need to come up
with some fancy secondary server to dispatch and marshall (serialize)
job assignments?

Attempt to lock the job after popping it from the queue:

class Job < AR::Base
#self.locking_column = ‘lock_version’
belongs_to :queue
end

class Queue < AR::Base
has_many :pending_jobs, :class_name => Job, :conditions =>
‘pending=true’

 def pop
   if target = pending_jobs.find(:first)
     target.pending = false
     target.save!
     target
   end
 rescue ActiveRecord::StaleObjectError
   # Grab another if someone else locked ours first.
   retry
 end

end

jeremy

3

Fantastic input Jeremy. Thanks! That really solves a lot of problems.
I’ll have to experiment and give it a test drive. If it works well in a
very busy environment then I’ll be set. Thanks a ton!

By the way, there may be a lot of jobs and many servers accessing the
queue. What are your thoughts on the most efficient way to fetch
multiple jobs at a time?

  • Don
4

Matt Bauer wrote:

I’m not sure about your application but maybe you should look into
Distributed Ruby and/or Rinda. It provides a lot of functionality for
working with queues and tasks in a distributed space.

Matt

I looked into and experimented with Rinda and DRb but I couldn’t figure
out how it would solve my original problem without introducing a single
point of failure. If all my Rails servers used the same intance of my
Queue via DRb, then I would need to engineer a contignecy for failover.
And that would probably mean multiple DRb servers, in which case I’d
need to do what Jeremy suggests (or something like it) anyway.
Additionaly, since my queue is comprised of fairly dynamic ActiveRecord
objects, they go out of sync with the locally cached objects published
via DRb. So the DRb server would need to continually refresh the
objects in its cache from the DB.

Although Jeremy’s solution is less sexy than a Distributed Ruby design,
it seems like it’s less complex.

I’m brand new to Ruby / Rails / DRb / Rinda. So I’m probably missing a
lot here. Any insight is always appreciated!

Thanks, Don

5

I’m not sure about your application but maybe you should look into
Distributed Ruby and/or Rinda. It provides a lot of functionality for
working with queues and tasks in a distributed space.

Matt

6

On May 9, 2006, at 12:09 PM, Don S. wrote:

By the way, there may be a lot of jobs and many servers accessing the
queue. What are your thoughts on the most efficient way to fetch
multiple jobs at a time?

I’d add a :lock option to AR::Base#find so you could:

class Job < AR::Base
belongs_to :owner

 def self.claim!(owner, n = 10)
   # select * from jobs where owner_id is null limit 10 for update
   jobs = find(:all, :conditions => 'owner_id is null', :limit =>

n, :lock => ‘for update’)

   # update jobs set owner_id=1 where id in (1,2,3,4,5)
   update_all(['owner_id=?', owner], ['id in (?)', jobs.map { |j|

j.id }])

   jobs
 end

end

select … for update gives you an exclusive lock on the selected
rows so others can’t set themselves as owner before you do.

Best,
jeremy