Friends-
I'm happy to annouce the first alpa release of BackgrounDRb. This is
a small framework for managing long running background tasks that
allows for ajax progress bars and more. It also serves as an
Application wide cache and context store for when you need something
like sessions but shared between users and multiple backend processes
like fcgi’s or mongrels. This MiddleMan runs in a drb(distributed
ruby) process that is separate from the rails process. This allows
for all backends to have one place to store data or launch jobs from.
You can see the proof of concept screencasts here:
http://brainspl.at/drb_progress.mov
http://brainspl.at/drb_ajax_tail.mov
And you can download the proof of concept rails app and run it
yourself from here:
http://opensvn.csie.org/ezra/rails/plugins/backgroundrb/
I'm looking for some folks to play with this and give feedback so I
can improve on it. So I appreciate any feedback from folks who try
this out for me.
Cheers-
-Ezra
Here is the README:
BackgrounDRb is a small framework for divorcing long running tasks from
Rails request/response cycle. With HTTP it is usually not a very good
idea to keep a request waiting for a response for long running actions.
BackgrounDRb also allows for status updates that in combination with
ajax can render live progress bars in the browser while the background
worker task gets completed. The MiddleMan can also be used as a cache.
You can store rendered templates or compute intensive results in the
MiddleMan for use later.
The MiddleMan drb server is a front controller or factory/delegate type
of object. It takes instructions from you and instantiates your worker
objects with the args you send from rails. It uses a hash to keep a key
pointing to a running worker that is also put into the session in rails
so railscan find the same Worker object that is running its job on
subsequent requests. The MiddleMan front object has a method that takes
a class type as a symbol and instantiates a new instance of said
class type. Then it returns a job_key to the client(rails). Rails can
then call a method on that object through the MiddleMan class.
There are many possible use cases for this system. More will be
implemented
soon. This is an open request for comments and feature requests.
The great thing about this framework is the fact that it creates a
shared
resource that is accessible from multiple backends like fcgi’s or
mongrel
processes. They each get a connection to the same MiddleMan object so
this
can be used as an application wide context as well as background process
runner that is shared across all users and all backends.
Let’s look at how this system works in detail.
Look at INSTALL for instructions on how to get everything set up and
working.
Lets look at a simple worker class.
class FooWorker
include DRbUndumped
def initialize(options={})
@progress = 0
@results = []
@options = options
start_working
end
def start_working
# Work loop goes inside a new thread so it doesn’t block
# rails while it works. A neat way to do progress bars in
# the browser is to have a @progress instance var that is
# initialized to 0 and then gets bumped up by your long
# running task. This way you can poll for the progress
# of your job via ajax and update a client side progress bar.
Thread.new do
# main work loop goes here. do work and update the
# progress bar instance var.
while something
@results << foo(@options)
@progress += 1
break if @progress > 99
end
end
end
def results
@results
end
def progress
puts “Rails is fetching progress: #{@progress}”
@progress
end
end
Your worker classes go into the RAILS_ROOT/lib/workers/ directory.
You can then use your worker class in rails like this:
in a controller
start new worker and put the job_key into the session so you can
get the status of your job later.
def background_task
session[:job_key] = MiddleMan.new_worker(:class => :foo_worker,
:args => {:baz =>
‘hello!’, :qux => 'another arg!})
end
def task_progress
if request.xhr?
progress = MiddleMan.get_worker(session[:job_key]).progress
render :update do |page|
page.replace_html(‘progress’,
“
#{progress}% done
” +“
”)if progress == 100
page.redirect_to :action => ‘results’
end
end
else
redirect_to :action => ‘index’
end
end
def results
@results = MiddleMan.get_worker(session[:job_key]).results
MiddleMan.delete_worker(session[:job_key])
end
Please note that when you use new_worker it takes a hash as the
argument.
the :class part of the hash is required so MiddleMan knows which
worker class to instantiate. You can give it either an underscore
version like :foo_worker or normal like :FooWorker. Also the :args key
points to a value that will be given to your worker class when
initialized.
The following will start a FooWorker class with a text argument of “Bar”
session[:job_key] = MiddleMan.new_worker(:class => :foo_worker,
:args => “Bar”)
In the background_task view you can use periodically_call_remote
to ping the task_progress method to get the progress of your job and
update
the progress bar. Once progress is equal to 100(or whatever you want)
you
redirect to the results page to display the results of the worker task.
There are a few simple examples in the workers dir. These are the
worker classes
I show being used here for proof of concept:
http://brainspl.at/drb_progress.mov
http://brainspl.at/drb_ajax_tail.mov
If you want to play with the demo app that implements those two
movies then
you can check out the rails app here to play with:
http://opensvn.csie.org/ezra/rails/plugins/backgroundrb/
If you want to have a named key instead of generated key you can
specify the
key yourself. This is handy for creating shared resources that more
then one
user will access so that multiple users and backends can get the same
object
by name.
MiddleMan.new_worker(:class => :foo_worker,
:args => “Bar”
:job_key => ‘shared_resource’)
For caching text or simple hashes or arrays or even rendered views you
can use a hash like syntax on MiddleMan:
MiddleMan[:cached_view] = render_to_string(:action => ‘complex_view’)
Then you can retrieve the cached rendered view just like a hash with:
MiddleMan[:cached_view]
You could create this cache and then have an ActiveRecord observer
expire the cache and create a new one when the data changes. Delete
the cached view with:
MiddleMan.delete_worker(:cached_view)
Best practice is to delete your job from the MiddleMan when you are
done with
it so it can be garbage collected. But if you don’t want to do this
then you
can use another option to clean up old jobs. Using cron or maybe
RailsCron
for a time, you can call the gc! method to delete jobs older then a
certain time.
Here is a ruby script you could run from cron that will delete all
workers
older then 24 hours.
#!/usr/bin/env ruby
require “drb”
DRb.start_service
MiddleMan = DRbObject.new(nil, “druby://localhost:22222”)
MiddleMan.gc!(Time.now - 606024)
** ROADMAP **
- Add better ActiveRecord caching facilities. Right now you can
cache text, hashes, arrays and many
other object types. But I am still working on the best way to
cache ActiveRecord
objects. I will probably use Marshal or YAML to do the right
thing here - More examples. A chat room is forthcoming as well as an email queue.
- More documentation.
- Detail how to set this up to work across physical servers. DNS
must be good and have reverse
dns as well for drb to work properly across machines. - Profit… ?
