# Offloading Background Tasks

**URL:** <https://www.ruby-forum.com/t/offloading-background-tasks/130640>\
**Category:** Mongrel\
**Created:** [March 1, 2008, 2:40am UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640 "2008-03-01T02:40:44Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [March 1, 2008, 2:40am UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/1 "2008-03-01T02:40:44Z")

</div>

Hello everyone,

I am very interested in off-loading tasks from the request/response  
cycle in order to speed up my application. My application is heavily  
connected to using web services both external and from other servers we  
host.

My application does the following long-running tasks:

- accesses / pushes to the twitter api’s (with possible need for status  
responses)
- accesses / pushes the facebook api’s (with need for status responses)
- requesting data from another server we run (rails needs access data)
- pushes information to aim using ruburple

I am just curious and grateful for any help people can give for what  
would suit me best. I am familar with backgroundrb and bj along with  
starling, etc. I don’t know which one will suit me best based on the  
tasks I need to queue.

My main concern is how to properly request data and then show it to the  
user. For instance, when the user logs into the rails application, I  
need to show them certain information about them which comes from  
complex analyzing of data done by another server we run.

When a user logs in, I need to show them the current results of this  
analysis. To do this, I need to perform a GET request to another server  
and get the results and then show them to the user. What is the “best  
practice” way to achieve this in which I can offload this task to a  
background worker and then get the results and display them to the user.

For instance, say the user logs in and then the background worker kicks  
off to retrieve their current analyzed data from the server. How do I  
then wait for the data to be retrieved by the worker and when it’s ready  
display it to the user?

I feel like I am missing something because I don’t see how the worker  
can “tell” the main rails application that it now has the data and then  
somehow show it when its “ready”. Do I need to constantly poll the  
server or how is this usually done.

Also, I need to allow the user to make certain changes to data fields  
and then I need to push those changes to twitter and facebook via a  
background process.

What solution is best suited based on what I described and could someone  
please help me understand the pattern for how to do this. I will be  
eternally grateful.

---

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [March 2, 2008, 3:18am UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/2 "2008-03-02T03:18:24Z")

</div>

On Sat, Mar 1, 2008 at 7:10 AM, Nathan E. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> responses)  
> user. For instance, when the user logs into the rails application, I  
> off to retrieve their current analyzed data from the server. How do I  
> background process.
> 
> ## What solution is best suited based on what I described and could someone please help me understand the pattern for how to do this. I will be eternally grateful.

For what you are describing, I think BackgrounDRb will be best fit.  
Since, its the only solution ( that I know of ) provides interfacing  
with rails so as you can actually fetch results retrieved by a worker  
in rails.

Bj is a worker queue to offload tasks. AFAIK, you can’t really  
interact seamlessly from rails. I don’t know much about startling. But  
this is coming from guy who maintains bdrb to take it with a grain of  
salt.

---

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [March 2, 2008, 9:02pm UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/3 "2008-03-02T21:02:40Z")

</div>

At 06:17 PM 3/1/2008, removed\_email\_address@domain.invalid wrote:

> > Hello everyone,  
> > I feel like I am missing something because I don’t see how the  
> > background process.  
> > with rails so as you can actually fetch results retrieved by a worker  
> > in rails.
> 
> Bj is a worker queue to offload tasks. AFAIK, you can’t really  
> interact seamlessly from rails. I don’t know much about startling. But  
> this is coming from guy who maintains bdrb to take it with a grain of  
> salt.

Hi,

Firstly - apologies to Hemant for misinformation from me a while ago  
about the current status of BackgrounDrb. It’s clear now of course to  
me that this an active and healthy project.

I have been messing around with Ara Howards BackgroundJob (bj) and it’s  
got a way to spin out a bunch of jobs and then wait until they all  
return, which is similar to the way threads work:

> ```
> jobs = Bj.submit list_of_jobs, :tag => 'important'
> ...
> 
> jobs.each do |job|
> if job.finished?
> p job.exit_status
> p job.stdout
> p job.stderr
> end
> end
> 
> ```

You can see that if by the time you get down to the “if jobs.finished?”  
section, you could choose to ignore whatever jobs haven’t returned yet  
(or possibly even TERM them if that’s appropriate).

I wonder also if you’ve considered adding some architecture to your  
system so that you can collect whatever you can from your external  
services from within Rails but then whatever is left over you could  
pull down via Ajax after the page had mostly loaded (so some parts of  
the page might be left with placeholders and then get filled in with  
the correct info later as the Ajax calls retrieve it from Rails…

That’d be my thought on this: build something (with BackgrounDRb, Bj or  
whatever) that lets you spawn a bunch of a async queries. Give those  
queries some kind of unique handle. When you’re done building the whole  
page, have your view fill in data from whichever queries have returned.  
Then send the incomplete page to the browser and have ajax queries pull  
down the remaining query results (using a webservice interface passing  
in those unique handles you set up above to ID which job you want).

Hope this helps!

Steve

---

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [March 9, 2008, 5:58am UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/4 "2008-03-09T05:58:36Z")

</div>

Thanks for your responses. Just wanted to reply back that I installed  
backgroundrb (the new version) and I definitely appreciate how useful it  
is and how it is fairly easy to use.

Thanks for all your hard work Hermant.

---

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [March 4, 2008, 3:40pm UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/5 "2008-03-04T15:40:16Z")

</div>

We’ve been testing using Spawn. Has anyone had any success in a  
production environment with the spawn plugin? It’s very simple but  
seems to work well although there doesn’t seem to be any recent  
activity around it.

Eventually I’d like to use Amazons SQS for our application but we  
still have other architecture issues to sort out before going that  
route.

Mike

---

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [March 9, 2008, 8:23am UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/6 "2008-03-09T08:23:05Z")

</div>

On Sun, 2008-03-09 at 05:58 +0100, Nathan E. wrote:

> Thanks for your responses. Just wanted to reply back that I installed  
> backgroundrb (the new version) and I definitely appreciate how useful it  
> is and how it is fairly easy to use.
> 
> Thanks for all your hard work Hermant.

No problems!

If you are having any problems feel free to shoot your questions on  
BackgrounDRb mailing list.

---

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [April 23, 2008, 7:54am UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/7 "2008-04-23T07:54:43Z")

</div>

Hemant K. wrote:

> On Sun, 2008-03-09 at 05:58 +0100, Nathan E. wrote:
> 
> > Thanks for your responses. Just wanted to reply back that I installed  
> > backgroundrb (the new version) and I definitely appreciate how useful it  
> > is and how it is fairly easy to use.
> > 
> > Thanks for all your hard work Hermant.
> 
> No problems!
> 
> If you are having any problems feel free to shoot your questions on  
> BackgrounDRb mailing list.

Hi!

Im using this backgroundrb plugin! The steps i followed for using this  
plugin are as follows: Kindly correct me where i went wrong!

1)rails\_apps/app\_name/ruby script/plugin install  
[http://svn.devjavu.com/backgroundrb/trunk](http://svn.devjavu.com/backgroundrb/trunk)

1. rake backgroundrb:setup

2. ruby script/generate worker example

4)opened example\_worker.rb and just wrote :

1. class ExampleWorker \< BackgrounDRb::MetaWorker
2. set\_worker\_name :example\_worker
3. def create(args = nil)
4. 

```
# this method is called, when worker is loaded for the first 

```

time  
5. logger.info “here”  
6. end  
7. end

after that i tried to run #Ruby script/backgroundrb start  
and it gave me load error :bdrb\_config.rb not found! I tried copying it  
to some directory and then ultimately it popped up another error :  
fork function is umimplemented on this machine!  
I have also installed the required gems :packets and chronic…

Can someone help me where i went wrong! or give me a step by step  
procedure to start backgroundrb running.  
Thanks in advance!  
Sup

---

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [April 24, 2008, 6:52am UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/8 "2008-04-24T06:52:02Z")

</div>

Quoting Supriya A. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> after that i tried to run #Ruby script/backgroundrb start  
> and it gave me load error :bdrb\_config.rb not found! I tried copying it  
> to some directory and then ultimately it popped up another error :  
> fork function is umimplemented on this machine!  
> I have also installed the required gems :packets and chronic…

This isn’t really the right list for this but…

“fork function is umimplemented on this machine!”

## Are you running on Windows? I think the error above indicates that your OS fundamentally can’t do what backgroundRb tries to do. See Wikipedia for more info on forking [fork (system call) - Wikipedia](http://en.wikipedia.org/wiki/Fork_%28operating_system%29)

Cynthia K.

---

<div class="post-metadata">

**Author:** ![xgamerx](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/x/d2c977/32.png) [@xgamerx](https://www.ruby-forum.com/u/xgamerx)\
**Post date:** [September 10, 2009, 5:19pm UTC](https://www.ruby-forum.com/t/offloading-background-tasks/130640/9 "2009-09-10T17:19:35Z")

</div>

Supriya A. wrote:

> Hi!
> 
> Im using this backgroundrb plugin! The steps i followed for using this  
> plugin are as follows: Kindly correct me where i went wrong!
> 
> 1)rails\_apps/app\_name/ruby script/plugin install  
> [http://svn.devjavu.com/backgroundrb/trunk](http://svn.devjavu.com/backgroundrb/trunk)
> 
> 1. rake backgroundrb:setup

You need to make a db:migrate

# rake db:migrate

Regards