Forum: Ruby on Rails Parallelizing Tasks in the Background

6e672922c21a5298f2a666ecb1c11d7c?d=identicon&s=25 Phil (Guest)
on 2013-10-05 06:18
(Received via mailing list)
I'm experimenting with optimizing some areas that involve long
processing
times on some SQL selects (or any background task, really).  This is in
Rails 4.0.0 with Ruby 2.0.0p247, PostgreSQL 9.2.4.

In this case, I can do intelligent dividing of selects to even the
workload
amongst N processes for the DB backend.

I currently am trying Spawnling, and this works for exactly one hit
before
I have to restart the web service:

  spawns = []
  ABC.each do # criteria for selects to divide load
   spawns << Spawnling.new do
     # portion of big SQL select here via ActiveRecord, and store in
memcache
   end
  end
  # wait for all N blocks of code to finish running
  Spawnling.wait(spawns)
  # assemble result from memcache and deliver
  render ...

Works for one hit and then future hits log correctly but fail to deliver
any content to the browser. There seems to be a number of issues with
Spawnling anyways as it seems to be in disrepair.

How are folks parallelizing tasks these days for lots of CPUs per hit?
 What's the Rails 4/Ruby 2 way to do it?

Thanks for any thoughts.


Phil
7829b7e1c842398d49af0e29c05bf97d?d=identicon&s=25 gabriel jaldon (gjaldon)
on 2013-10-06 16:52
(Received via mailing list)
Hi Phil,

Sidekiq sounds like the gem you need. It does background processing
using
threads so you get true parallelism. You would need an implementation of
Ruby though that can support true parallelism like Rubinius or JRuby.
Ruby
MRI has the GIL so it can't do true multithreaded processing.

Here are links for Sidekiq:
1. http://railscasts.com/episodes/366-sidekiq
2. https://github.com/mperham/sidekiq

Hope it helps!

Cheers,
Gabe
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.