Asynchronous jobs in Rails

Rails
1

Hi,

I have been researching about this for couple of days now. My rails
application needs to send email notifications to large number of
customers. Before sending the message, application customizes content
for each recipient. The requirement compels me to call ActionMailer’s
deliver method for each email address.

I would like to send notifications asynchronously. I looked into
backgroundRB. I have been playing around with it for almost two days
and I can’t get it working on cygwin. However, my main concern is the
poor performance of backgroundRB for moderate or high scalability
needs.

I also looked into Threads in ruby. Since rails is single threaded and
especially ActiveRecord is not thread safe, I dropped that idea too.

What are the options generally used for asynchronous jobs in rails ?
From google search I find only two options mentioned above. Please
tell me your experience with either of them. Is there any other
solution that I have overlooked ?

Thank you!

2

Thanks for you response. I was thinking about Amazon queue service and
if that might help me here. Our production servers are on linux but my
dev machine ( to my dismay) is a windows machine. BackgroundRB seems
promising as along as I am able to get it to run.

On Jun 10, 2:37 pm, “[email protected]”

3

We’ve used BackgroundRb for our light-to-this-point asynchronous
processing needs. It was a pain in the butt to get working initially
(and we can still only run it on our Linux boxes, not on Windows), but
I’ve perceived it to be probably the most frequently used asynchronous
solution for Rails, so we’re going to go with it until we can’t.
Though it’s a memory and process hog, I don’t think there would be
trouble sending many mails with it, as long as its running on a
machine with a lot of memory, and as long as those mails aren’t spam
sent to me inbox. :slight_smile:

Another popular asynchronous option is to combine a queuing system
with rake tasks that run jobs as they are put in the queue. Starling
is a fairly reliable and easy to setup queuing solution developed and
used by the folks at Twitter. You can use it with Workling to
automatically process your tasks, or you can write a rake task that
runs via crontab and checks for jobs to run on some timed interval.

There’s some documentation on setting up Starling on my blog.

Bill
http://www.williambharding.com/blog

4

Hey Procyon,

I’ve recently set up backgroundrb to work with my app and it performs
an emailing function as well.

Althought I haven’t put it through a rigorous stress test yet, one
test that I did conduct was setting my Worker method to run every 1
second and having it process 200 emails each time.

The server was a 512mb slice from slicehost. It has 2 rails sites on
it right now. At normal operations (with bgrb stopped), it has about
403mb free and 108mb used. During my test for about 5 minutes, the
server memory usage went to around 200mb free. I was using sendmail to
send off the emails… and I didn’t check how much of the memory usage
that was using. I should also note that the Worker code did not use
the thread_pool feature so that might help bring memory usage down as
well.

For my purpose, I think I won’t be sending more than 1000 emails per
hour and so it looks like it will be able to serve my needs easily
unless my traffic goes crazy like FaceStat.

Hope this helps.

~ mel

5

You should also check out Starling with the Workling plugin. I found
it to be a much easier, robust, and efficient method for doing
background processing. Check out
http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/
for a series I did after not being pleased with how backgroundrb was
working.

6

PS: If you need any help with getting things set up, feel free to IM
me on Gtalk at melvin @-AT=@ volcanicmarketing.com. I’m no expert with
it but I’ll try to help

7

On Wed, Jun 11, 2008 at 3:07 AM, [email protected]
[email protected] wrote:

We’ve used BackgroundRb for our light-to-this-point asynchronous
processing needs. It was a pain in the butt to get working initially
(and we can still only run it on our Linux boxes, not on Windows), but
I’ve perceived it to be probably the most frequently used asynchronous
solution for Rails, so we’re going to go with it until we can’t.
Though it’s a memory and process hog, I don’t think there would be
trouble sending many mails with it, as long as its running on a
machine with a lot of memory, and as long as those mails aren’t spam
sent to me inbox. :slight_smile:

I am afraid, that has changed unless you are running old svn version
of backgroundrb:

http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/

Many users have found git version to be quite stable and since now
bdrb uses fork and exec ( meaning it replaces old process image with
new one), there is no sharing of resources between workers, which used
to prevent garbage collection of objects, which were allocated before
fork. Its not a problem of bdrb per se, since Ruby interpretor is
known to be not COW(copy on write) friendly (for uninitiated, it means
that, in ruby, when you fork, all the objects in scope have a bit set
in them, so as they get copied in forked process. This is fine, but
this prevents, those objects from garbage collected. Ideally, it
should copy only when an object’s value is changed in child , hence
copy on write . But Ruby doesn’t do that. You will find plenty of
material about this on internet)

Github version of bdrb uses fork and exec, and hence it means that,
objects are no longer shared and hence garbage collection can work as
usual. In fact, if you want access to rails environment, this is as
much as lean and thin you can get. You can’t have your cake and eat it
too.

Bill

From google search I find only two options mentioned above. Please
tell me your experience with either of them. Is there any other
solution that I have overlooked ?

Thank you!

–
Let them talk of their oriental summer climes of everlasting
conservatories; give me the privilege of making my own summer with my
own coals.

http://gnufied.org