Offloading big uploads to front-end webserver

Rails
1

I’m doing a website which has quite a few big uploads / downloads to
manage.

Since rails isn’t multithreaded (yet), this means every big upload/
download will keep a mongrel/thin/whatever thread for itself doing
nothing but waiting for the transfer to finish.

Now, for public files, this of course gets handled by my frontend
nginx server.

For private files, I use the X-Accel-Redirect header, so the transfer
gets offloaded to nginx as well.

Only for uploads, I couldn’t find a nice way to handle them.
Is there a way to somehow tell nginx to first receive the whole file,
then transfer it to mongrel/thin so it doesn’t lock up a full
instance?
Any way to do this, while first consulting rails if this should be
allowed?

Any help is greatly appreciated.

Greetings,
Mathijs

2

On 25 Mar 2008, at 22:45, [email protected] wrote:

For private files, I use the X-Accel-Redirect header, so the transfer
gets offloaded to nginx as well.

Only for uploads, I couldn’t find a nice way to handle them.
Is there a way to somehow tell nginx to first receive the whole file,
then transfer it to mongrel/thin so it doesn’t lock up a full
instance?
Any way to do this, while first consulting rails if this should be
allowed?

First of all, Mongrel is multithreaded and Mongrel doesn’t hand off
the uploaded file to Rails until after the upload itself (i.e. the
upload itself doesn’t block the mongrel at all). That said, after the
upload the Mongrel will be locked for the duration of processing in
Rails.

Depending on what you are actually doing in Rails, you have several
possibilities:

  1. Just saving the file to disk: just let Rails handle it using a
    Tempfile, see the attachment_fu source code. It’s pretty fast and
    won’t lock your Mongrel for long since the Rails request, which IS
    single threaded, won’t take long
  2. Extensive processing: offload the upload to a background process,
    and here you have another choice:
    • BackgroundRB (and other background processors): has access to
      the Rails stack and runs behind Rails, i.e. Rails will hand over the
      file to the background worker
    • Merb: a Rails like framework that contrary to Rails is
      multithreaded, it can sit behind a single mongrel and process heaps of
      uploads without locking the mongrel. You can plugin ActiveRecord into
      merb for model access. You’ll also need some rewrite magic to direct
      uploads to the Merb application if you want the rest served through
      Rails.

Best regards

Peter De Berdt