I am very interested in off-loading tasks from the request/response
cycle in order to speed up my application. My application is heavily
connected to using web services both external and from other servers we
host.
My application does the following long-running tasks:
accesses / pushes to the twitter api’s (with possible need for status
responses)
accesses / pushes the facebook api’s (with need for status responses)
requesting data from another server we run (rails needs access data)
pushes information to aim using ruburple
I am just curious and grateful for any help people can give for what
would suit me best. I am familar with backgroundrb and bj along with
starling, etc. I don’t know which one will suit me best based on the
tasks I need to queue.
My main concern is how to properly request data and then show it to the
user. For instance, when the user logs into the rails application, I
need to show them certain information about them which comes from
complex analyzing of data done by another server we run.
When a user logs in, I need to show them the current results of this
analysis. To do this, I need to perform a GET request to another server
and get the results and then show them to the user. What is the “best
practice” way to achieve this in which I can offload this task to a
background worker and then get the results and display them to the user.
For instance, say the user logs in and then the background worker kicks
off to retrieve their current analyzed data from the server. How do I
then wait for the data to be retrieved by the worker and when it’s ready
display it to the user?
I feel like I am missing something because I don’t see how the worker
can “tell” the main rails application that it now has the data and then
somehow show it when its “ready”. Do I need to constantly poll the
server or how is this usually done.
Also, I need to allow the user to make certain changes to data fields
and then I need to push those changes to twitter and facebook via a
background process.
What solution is best suited based on what I described and could someone
please help me understand the pattern for how to do this. I will be
eternally grateful.
responses)
user. For instance, when the user logs into the rails application, I
off to retrieve their current analyzed data from the server. How do I
background process.
What solution is best suited based on what I described and could someone
please help me understand the pattern for how to do this. I will be
eternally grateful.
For what you are describing, I think BackgrounDRb will be best fit.
Since, its the only solution ( that I know of ) provides interfacing
with rails so as you can actually fetch results retrieved by a worker
in rails.
Bj is a worker queue to offload tasks. AFAIK, you can’t really
interact seamlessly from rails. I don’t know much about startling. But
this is coming from guy who maintains bdrb to take it with a grain of
salt.
Hello everyone,
I feel like I am missing something because I don’t see how the
background process.
with rails so as you can actually fetch results retrieved by a worker
in rails.
Bj is a worker queue to offload tasks. AFAIK, you can’t really
interact seamlessly from rails. I don’t know much about startling. But
this is coming from guy who maintains bdrb to take it with a grain of
salt.
Hi,
Firstly - apologies to Hemant for misinformation from me a while ago
about the current status of BackgrounDrb. It’s clear now of course to
me that this an active and healthy project.
I have been messing around with Ara Howards BackgroundJob (bj) and it’s
got a way to spin out a bunch of jobs and then wait until they all
return, which is similar to the way threads work:
jobs = Bj.submit list_of_jobs, :tag => 'important'
...
jobs.each do |job|
if job.finished?
p job.exit_status
p job.stdout
p job.stderr
end
end
You can see that if by the time you get down to the “if jobs.finished?”
section, you could choose to ignore whatever jobs haven’t returned yet
(or possibly even TERM them if that’s appropriate).
I wonder also if you’ve considered adding some architecture to your
system so that you can collect whatever you can from your external
services from within Rails but then whatever is left over you could
pull down via Ajax after the page had mostly loaded (so some parts of
the page might be left with placeholders and then get filled in with
the correct info later as the Ajax calls retrieve it from Rails…
That’d be my thought on this: build something (with BackgrounDRb, Bj or
whatever) that lets you spawn a bunch of a async queries. Give those
queries some kind of unique handle. When you’re done building the whole
page, have your view fill in data from whichever queries have returned.
Then send the incomplete page to the browser and have ajax queries pull
down the remaining query results (using a webservice interface passing
in those unique handles you set up above to ID which job you want).
Thanks for your responses. Just wanted to reply back that I installed
backgroundrb (the new version) and I definitely appreciate how useful it
is and how it is fairly easy to use.
We’ve been testing using Spawn. Has anyone had any success in a
production environment with the spawn plugin? It’s very simple but
seems to work well although there doesn’t seem to be any recent
activity around it.
Eventually I’d like to use Amazons SQS for our application but we
still have other architecture issues to sort out before going that
route.
On Sun, 2008-03-09 at 05:58 +0100, Nathan E. wrote:
Thanks for your responses. Just wanted to reply back that I installed
backgroundrb (the new version) and I definitely appreciate how useful it
is and how it is fairly easy to use.
Thanks for all your hard work Hermant.
No problems!
If you are having any problems feel free to shoot your questions on
BackgrounDRb mailing list.
On Sun, 2008-03-09 at 05:58 +0100, Nathan E. wrote:
Thanks for your responses. Just wanted to reply back that I installed
backgroundrb (the new version) and I definitely appreciate how useful it
is and how it is fairly easy to use.
Thanks for all your hard work Hermant.
No problems!
If you are having any problems feel free to shoot your questions on
BackgrounDRb mailing list.
Hi!
Im using this backgroundrb plugin! The steps i followed for using this
plugin are as follows: Kindly correct me where i went wrong!
# this method is called, when worker is loaded for the first
time
5. logger.info “here”
6. end
7. end
after that i tried to run Ruby script/backgroundrb start
and it gave me load error :bdrb_config.rb not found! I tried copying it
to some directory and then ultimately it popped up another error :
fork function is umimplemented on this machine!
I have also installed the required gems :packets and chronic…
Can someone help me where i went wrong! or give me a step by step
procedure to start backgroundrb running.
Thanks in advance!
Sup
after that i tried to run Ruby script/backgroundrb start
and it gave me load error :bdrb_config.rb not found! I tried copying it
to some directory and then ultimately it popped up another error :
fork function is umimplemented on this machine!
I have also installed the required gems :packets and chronic…
This isn’t really the right list for this but…
“fork function is umimplemented on this machine!”
Are you running on Windows? I think the error above indicates that
your OS fundamentally can’t do what backgroundRb tries to do. See
Wikipedia for more info on forking fork (system call) - Wikipedia