Multiple workling instances

Hi,
I’m running Starling and Workling to process background jobs. I’ve four
instances of starling running on the same server, and I’m trying to get
Workling to use them all. However, all of the jobs seem to be queued up
and carried out one by one, instead of up to four being process
simultaneously.

As a result, I’ve got a very long job queue which is taking a long time
to process, and I’d like to reduce it.

I’ve set :multiple to true in script/workling_client.rb, but this
doesn’t seem to help.

I’m very inexperienced with both Starling and Workling, so it’s quite
possible I’m either missing something simple, or approaching this the
wrong way entirely…

Any suggestions or insight much appreciated!

best

I actually just was in the same situation. I also had thought
multiple Starling instances would help things, but it didn’t. The
bottleneck was Workling, and I imagine it is the same for you.
Despite the lack of documentation, it turns out to be extremely easy
to run several instances:

  1. In script/workling_client.rb, change :multiple from false to true
    as you already have, so the Daemon knows to allow for multiple
    instances
  2. Just start more instances by calling “script/workling_client start”
    several times. It will create workling0.pid, workling1.pid, etc. I’m
    not sure how to properly setup god.rb to monitor the multiple
    instances so if you do please let me know :slight_smile:

FYI, you could also create multiple worker classes (my_worker.rb,
another_worker.rb etc), and each Workling instance would use a thread
for each class. So if you split up your methods into several classes,
it might help things even more.

PS. Here is an extremely helpful series of blog posts for starling
and workling:
http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/

I have modified my workling_client script to better handle multiple
worklings. This approach gives you full control over the workling
process names and pid files, which you can then use to cleanly manage
the worklings using God or Monit.

See below for my (admittedly hacky) solution:

#!/usr/bin/env ruby
require ‘rubygems’
require ‘daemons’

This is a bit hacky, but we don’t have a clean way to hook into the

parsed

options, as Daemons is managing these internally.

number = 0
for i in 0…ARGV.length
if ARGV[i] == ‘–number’
number = ARGV[i + 1]
2.times {ARGV.delete_at(i)}
end
end

puts “Workling Number: #{number}”

workling = File.join(File.dirname(FILE), ‘…’, ‘vendor’,
‘plugins’, ‘workling’, ‘script’, ‘listen.rb’)
options = {
:app_name => “workling-#{number}-client”,
:ARGV => ARGV,
:dir_mode => :normal,
:dir => File.join(File.dirname(FILE), ‘…’, ‘log’),
:log_output => true,
:multiple => false,
:backtrace => true,
:monitor => false
}

Daemons.run(workling, options)