# Asynchronous jobs in Rails

**URL:** <https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710>\
**Category:** Rails\
**Created:** [June 10, 2008, 11:44pm UTC](https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710 "2008-06-10T23:44:27Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![procyon](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ac91a4/32.png) [@procyon](https://www.ruby-forum.com/u/procyon)\
**Post date:** [June 10, 2008, 11:44pm UTC](https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710/1 "2008-06-10T23:44:27Z")

</div>

Hi,

I have been researching about this for couple of days now. My rails  
application needs to send email notifications to large number of  
customers. Before sending the message, application customizes content  
for each recipient. The requirement compels me to call ActionMailer’s  
deliver method for each email address.

I would like to send notifications asynchronously. I looked into  
backgroundRB. I have been playing around with it for almost two days  
and I can’t get it working on cygwin. However, my main concern is the  
poor performance of backgroundRB for moderate or high scalability  
needs.

I also looked into Threads in ruby. Since rails is single threaded and  
especially ActiveRecord is not thread safe, I dropped that idea too.

What are the options generally used for asynchronous jobs in rails ?  
From google search I find only two options mentioned above. Please  
tell me your experience with either of them. Is there any other  
solution that I have overlooked ?

Thank you!

---

<div class="post-metadata">

**Author:** ![procyon](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ac91a4/32.png) [@procyon](https://www.ruby-forum.com/u/procyon)\
**Post date:** [June 10, 2008, 11:54pm UTC](https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710/2 "2008-06-10T23:54:25Z")

</div>

Thanks for you response. I was thinking about Amazon queue service and  
if that might help me here. Our production servers are on linux but my  
dev machine ( to my dismay) is a windows machine. BackgroundRB seems  
promising as along as I am able to get it to run.

On Jun 10, 2:37 pm, “removed\_email\_address@domain.invalid”

---

<div class="post-metadata">

**Author:** ![procyon](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ac91a4/32.png) [@procyon](https://www.ruby-forum.com/u/procyon)\
**Post date:** [June 10, 2008, 11:50pm UTC](https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710/3 "2008-06-10T23:50:19Z")

</div>

We’ve used BackgroundRb for our light-to-this-point asynchronous  
processing needs. It was a pain in the butt to get working initially  
(and we can still only run it on our Linux boxes, not on Windows), but  
I’ve perceived it to be probably the most frequently used asynchronous  
solution for Rails, so we’re going to go with it until we can’t.  
Though it’s a memory and process hog, I don’t think there would be  
trouble sending many mails with it, as long as its running on a  
machine with a lot of memory, and as long as those mails aren’t spam  
sent to me inbox. 🙂

Another popular asynchronous option is to combine a queuing system  
with rake tasks that run jobs as they are put in the queue. Starling  
is a fairly reliable and easy to setup queuing solution developed and  
used by the folks at Twitter. You can use it with Workling to  
automatically process your tasks, or you can write a rake task that  
runs via crontab and checks for jobs to run on some timed interval.

There’s some documentation on setting up Starling on my blog.

Bill  
[http://www.williambharding.com/blog](http://www.williambharding.com/blog)

---

<div class="post-metadata">

**Author:** ![procyon](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ac91a4/32.png) [@procyon](https://www.ruby-forum.com/u/procyon)\
**Post date:** [June 11, 2008, 2:10am UTC](https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710/4 "2008-06-11T02:10:21Z")

</div>

Hey Procyon,

I’ve recently set up backgroundrb to work with my app and it performs  
an emailing function as well.

Althought I haven’t put it through a rigorous stress test yet, one  
test that I did conduct was setting my Worker method to run every 1  
second and having it process 200 emails each time.

The server was a 512mb slice from slicehost. It has 2 rails sites on  
it right now. At normal operations (with bgrb stopped), it has about  
403mb free and 108mb used. During my test for about 5 minutes, the  
server memory usage went to around 200mb free. I was using sendmail to  
send off the emails… and I didn’t check how much of the memory usage  
that was using. I should also note that the Worker code did not use  
the thread\_pool feature so that might help bring memory usage down as  
well.

For my purpose, I think I won’t be sending more than 1000 emails per  
hour and so it looks like it will be able to serve my needs easily  
unless my traffic goes crazy like FaceStat.

Hope this helps.

~ mel

---

<div class="post-metadata">

**Author:** ![procyon](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ac91a4/32.png) [@procyon](https://www.ruby-forum.com/u/procyon)\
**Post date:** [June 11, 2008, 4:30am UTC](https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710/5 "2008-06-11T04:30:44Z")

</div>

You should also check out Starling with the Workling plugin. I found  
it to be a much easier, robust, and efficient method for doing  
background processing. Check out  
[http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/](http://davedupre.com/2008/03/25/ruby-background-tasks-with-starling/)  
for a series I did after not being pleased with how backgroundrb was  
working.

---

<div class="post-metadata">

**Author:** ![procyon](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ac91a4/32.png) [@procyon](https://www.ruby-forum.com/u/procyon)\
**Post date:** [June 11, 2008, 2:11am UTC](https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710/6 "2008-06-11T02:11:32Z")

</div>

PS: If you need any help with getting things set up, feel free to IM  
me on Gtalk at melvin @-AT=@ [volcanicmarketing.com](http://volcanicmarketing.com). I’m no expert with  
it but I’ll try to help

---

<div class="post-metadata">

**Author:** ![procyon](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/p/ac91a4/32.png) [@procyon](https://www.ruby-forum.com/u/procyon)\
**Post date:** [June 11, 2008, 5:25am UTC](https://www.ruby-forum.com/t/asynchronous-jobs-in-rails/196710/7 "2008-06-11T05:25:06Z")

</div>

On Wed, Jun 11, 2008 at 3:07 AM, removed\_email\_address@domain.invalid  
[removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> We’ve used BackgroundRb for our light-to-this-point asynchronous  
> processing needs. It was a pain in the butt to get working initially  
> (and we can still only run it on our Linux boxes, not on Windows), but  
> I’ve perceived it to be probably the most frequently used asynchronous  
> solution for Rails, so we’re going to go with it until we can’t.  
> Though it’s a memory and process hog, I don’t think there would be  
> trouble sending many mails with it, as long as its running on a  
> machine with a lot of memory, and as long as those mails aren’t spam  
> sent to me inbox. 🙂

I am afraid, that has changed unless you are running old svn version  
of backgroundrb:

[http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/](http://gnufied.org/2008/05/21/bleeding-edge-version-of-backgroundrb-for-better-memory-usage/)

Many users have found git version to be quite stable and since now  
bdrb uses fork and exec ( meaning it replaces old process image with  
new one), there is no sharing of resources between workers, which used  
to prevent garbage collection of objects, which were allocated before  
fork. Its not a problem of bdrb per se, since Ruby interpretor is  
known to be not COW(copy on write) friendly (for uninitiated, it means  
that, in ruby, when you fork, all the objects in scope have a bit set  
in them, so as they get copied in forked process. This is fine, but  
this prevents, those objects from garbage collected. Ideally, it  
should copy only when an object’s value is changed in child , hence  
copy on write . But Ruby doesn’t do that. You will find plenty of  
material about this on internet)

Github version of bdrb uses fork and exec, and hence it means that,  
objects are no longer shared and hence garbage collection can work as  
usual. In fact, if you want access to rails environment, this is as  
much as lean and thin you can get. You can’t have your cake and eat it  
too.

> Bill
> 
> > From google search I find only two options mentioned above. Please  
> > tell me your experience with either of them. Is there any other  
> > solution that I have overlooked ?
> > 
> > Thank you!

–  
Let them talk of their oriental summer climes of everlasting  
conservatories; give me the privilege of making my own summer with my  
own coals.

[http://gnufied.org](http://gnufied.org)