Farming off zipping task to backgroundrb - advice requested!


#1

We have a problem at the moment: users build up lessons, which are then
zipped up with the lesson player into a standalone executable. This has
always been ok for us but the lessons and the player just got a lot
bigger, and the zipping process is now hogging resources too much: we
have an nginx server with a cluster of 12 mongrels, and if 3 people zip
a lesson at once, between them the three mongrels use up 100% of the
cpu time and tend to zombify.

What i’d like to do is move the zipping out of the mongrels into
seperate process, using backgroundrb. (I’ve got backgroundrb working
fine with some other stuff and am happy with it generally). This will
mean that the zips are built in a queue instead of all at once, thus
avoiding meltdown. It will also stop mongrels getting out of control.

My plan is this - does it sound sensible?

Controller action 1 - ‘build_zip’.
Call backgroundrb and tell it to make a zip of this lesson. Fall
through to ‘build_zip’ view, which just says “building zip file”.

Backgroundrb - builds zip.
When it’s finished, puts an entry in a table called
‘built_lesson_zips’

Back on ‘build_zip’ view page:
javascript action triggered by page load: polls (via
built_lesson_zips_controller) the built_lesson_zips database table to
see if the relevant zip is there. If not, refresh the ‘building zip
file’ message (or do something so that the user can see it hasn’t just
crashed), wait 5 seconds and try again. If it is there, then call
controller action 2, ‘built_zip’.

Controller action 2 - ‘built_zip’.
Proceed as before - start downloading the zip to the user etc.

Any feedback, suggestions, etc? Polling the table every five seconds
using javascript feels a bit dirty but since there’s 12 mongrels and
only one backgroundrb, getting backgroundrb to make a callback to the
right mongrel seemed too difficult for my limited knowledge, so i wanted
to keep the communication purely one-way.