Photo uploads and scalability

Server NGINX
1

Getting ready to work on a new section of the site which will require a
reader to upload a photo and write a caption.

It’s my first time writing anything to do with uploading and I’m
wondering what things in the nginx realm I need to consider so the
system doesn’t get bogged down. Nginx is a dream for us serving files,
but since I’ve never handled uploads, what pitfalls/traps/surprises do I
need to be prepared for?

I’d like to think of scalability from day one so I’m not trying to fix
something under load. Right now I have one server. Is there anyway to
prepare my nginx.conf for easily handling, say, new servers getting
added/removed in the cloud?

[Hey, we might get 1 upload a week or a million a day, so it’s good to
be prepared.]

From a programming end I already know I’ll queue any post-processing
and inform the reader when it’s done so that process can be handled by
my local server or cloud instances etc.

Thanks for any tips!

2

On Friday 31 August 2012 21:40:16 Ian E. wrote:
[…]

From a programming end I already know I’ll queue any post-processing
and inform the reader when it’s done so that process can be handled by
my local server or cloud instances etc.

TIP: You can avoid passing files over an upstream connection.

Example:

location = /upload {
proxy_pass http://backend;
client_body_in_file_only clean;
proxy_set_body $request_body_file;
}

Your application should “mv” uploaded file somewhere, then register a
task
for further background processing, and return a response.

Reference: Module ngx_http_core_module

wbr, Valentin V. Bartenev

3

On 31/08/2012 2:01 PM, Valentin V. Bartenev wrote:

location = /upload {
proxy_pass http://backend;
client_body_in_file_only clean;
proxy_set_body $request_body_file;
}

Your application should “mv” uploaded file somewhere, then register a task
for further background processing, and return a response.

As photos are several megabytes, should I be looking into nginx’s upload
module?

4

upload_module removes uploaded files from body and replaces them with
paths to temporary files where there are stored at end.

So your backend don’t have to parse body again to extract files from
it. Parsing costs memory & cpu.

If you upload photos, they will land on disk. Your backend can analyze
them briefly, make first thumb to be shown back to user and ask/
schedule other workers to make scalling/rotating and any image
processing you want to be done later in real background.

I’m happy user of upload_module :wink:
in tandem with uploadprogress_module

It is easy to setup, very.

imho: it’s worthwhile

Regards
Pawe Marzec

Wiadomo napisana w dniu 2012-09-01, o godz. 07:17, przez Ian E.:

5

On 01/09/2012 3:06 AM, “Pawe Marzec - cojestgrane.pl” wrote:

I’m happy user of upload_module :wink:
in tandem with uploadprogress_module

It is easy to setup, very.

imho: it’s worthwhile

Is there an article or tutorial that show examples of how to set the
module up? I’ve been looking around and see that some people are using
it with the a plugin called plupload. Any experience with that?

6

try this:

Pawe Marzec

Wiadomo napisana w dniu 2012-09-01, o godz. 10:42, przez Ian E.:

7

On 01/09/2012 5:12 AM, “Pawe Marzec - cojestgrane.pl” wrote:

try this:

Thanks. Does anyone know of an example integrating it with PHP? I’m
seeing examples for Passenger and Rails, but I don’t use them.

Thanks for the help. Just want to make sure I can scale from day one.