Splitting up a long running task with delayed job

I have a method on my model that looks like this.

def self.itunes_update

don’t run it if it’s already running

if Setting.first.update_lock?
puts “updates locked”

it’s not running right now so run it

all_apps = self.all
all_apps.each do |app|
# do stuff with each app

job done, clear the lock


It works fine, but it processes 10,000 records and takes about an hour
to run on heroku. I originally just had a rake task calling it every 3
hours but heroku were SIGTERMing it and suggested that I move it to
delayed_job, which I have done like this…


it’s working fine.

They do however suggest that I process this in batches or smaller
chunks. How would I do that, and does that sound like a good idea?

Also - do I need to do the locking any more, I think delayed job handles
it for me - that said, guess it doesn’t do any harm?



anyone done anything like this before?

bingo bob wrote in post #989039:

anyone done anything like this before?

What if you select 100 records per pass in an oldest-update ==
first-selected order?

model.Find(:all, :order => “updated_at”, :limit => 100)

Update each record after you examine/process it.

I’m not sure this is what you mean, but I use a daemon for anything
that would make the browser respond slowly. I have a table
“delayed_tasks” and the controller just writes a row to that table and
finishes. Then the daemon looks for rows in the table and handles them
behind the scenes and deletes them when handled.

For instance, I do that for sending email because it takes a number of
seconds for email to get delivered.

If it is something the user is waiting for, the browser can
periodically do an ajax call to see if the results are ready.