Handle user uploads in static content server (newbie question)

Hi guys.

Newbie question here, apologies in advance.

By using a static content server (nginx as reverse proxy), how does
one deal with user uploads? I mean, I can understand if I client
request our website’s /images/logo.png file, because we manually store
it on the static content server, but what about the photo that our
user just uploaded to his photo album?

My confusion is: if a request is to a .php file, I’ll pass through the
back-end server. So If a user wants to upload a file, it’ll be upload
to /user-uploads/file.jpg on the back-end server. How can we make
the static server serve the image? I’m imagining there has to be some
sort of synchronization, but then again there the problem that the
image might take longer to be replicated to the static content server
and when the user request’s it, he’ll get a 404… not acceptable for
online businesses.

Can anyone with more experience tell me what am I missing?

Cheers

Leonardo

To make it simpler, here’s a scenario:

Server A is static.domain.com ip 10.10.10.10 servers ALL static
content (jpegs, gifs, pngs, mp3s etc… including user uploaded
content like photo albums)
Server B is www.domain.com ip 10.10.10.20 servers DYNAMIC content, php
files.

If user submits a jpg to his photo album, it’ll get uploaded to B, as
page’s going to run on server B because it’s a .php file (upload.php).
How can the image possibly be served from server A without
synchronization issues?

There, much better. :slight_smile:

Thanks, Leonardo.

On Apr 12, Leonardo C. wrote:

synchronization issues?
You options are:

(1) Shared filesystem between the two hosts to hold the static content

(2) Use nginx with caching proxy on the static machine
In this setup, a cache miss will result in a request for the static
resource from the dynamic server (which I assume is also capable of
serving static content). This should work as long you expect to have a
high cache hit ratio

Just like you configure a proxy to your backend server, you can have a
separate directory for uploaded files served through a proxy. There
should
be no need to sync files between the two servers.

For example:
location /uploads/ {
proxy_pass http://10.1.1.2/uploads/ # or whatever
}

On Mon, Apr 12, 2010 at 12:17 PM, Arvind Jayaprakash
[email protected] wrote:

How can the image possibly be served from server A without
synchronization issues?

You options are:

(1) Shared filesystem between the two hosts to hold the static content

They’re different linodes so I don’t think this is an option.

(2) Use nginx with caching proxy on the static machine
In this setup, a cache miss will result in a request for the static
resource from the dynamic server (which I assume is also capable of
serving static content). This should work as long you expect to have a
high cache hit ratio

So user uploads a file to Dynamic Server (D). All request to static
files are served from Static Server (S). If a file is not found on
(S), cache miss, serve the file from (D). Correct? In this scenario,
how would the file ever get served from the Static Server (S)?
Something has to copy the files from server D to S…

location /uploads/ {
proxy_pass http://10.1.1.2/uploads/ # or whatever
}

So the code above is in the .conf file of the Dynamic server and
10.1.1.2 is the Static server’s IP, correct?

I don’t see how this can work still. If a file is Posted to
upload.php, it’ll end up on Dynamic server and not on Static.

On Mon, Apr 12, 2010 at 9:52 AM, Leonardo C. [email protected]
wrote:

Does that sound reasonable?

I believe that’s what originally suggested.

The alternative is beaming the file using ssh to static server’s
directory automatically on upload (using php functions, etc). Might
cause slight delay and more complex setup though.


O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

I believe that’s what originally suggested.

Indeed Edho, i just expanded upon Arvind’s idea to make sure I
understand it.

The alternative is beaming the file using ssh to static server’s
directory automatically on upload (using php functions, etc). Might
cause slight delay and more complex setup though.

Yeah, NFS seems to be the winner.

If anyone is interested, I’ll follow up the solution posted here:

Cheers.

Do you guys think have another instance of nginx handling static
content on the same server would be a better solution than a dedicated
static server? I’d use proxy_pass for both scenarios…

Nginx handles things so well that I might have difficulties
justifying another server…

Cheers

  1. Shared filesystem between the two hosts to hold the static content

I just found out that I can do that on linode. Thanks for bringing this
idea up!

Here’s a solution:
Using NFS, mount on the Dynamic server the Static server’s directory
for static files (/images /user-uploads etc). Have nginx on Dynamic
with proxy_pass to Static for all static content. All uploads go to
the mounted drive on Dynamic (which points to Static).

Does that sound reasonable?

Cheers

Leonardo

On Mon, Apr 12, 2010 at 12:17 PM, Arvind Jayaprakash

On Apr 12, Leonardo C. wrote:

I just found out that I can do that on linode. Thanks for bringing this
idea up!

Here’s a solution:
Using NFS, mount on the Dynamic server the Static server’s directory
for static files (/images /user-uploads etc). Have nginx on Dynamic
with proxy_pass to Static for all static content. All uploads go to
the mounted drive on Dynamic (which points to Static).

Either I am understanding this incorrectly or this sounds like you have
made things worse!

If the user requests for the static content is going to enter via D,
then what is the point? User download/access of static content should
directly hit nginx running on S which accesses it’s local storage.

The reason why I’d push for the caching proxy option over shared storage
is that in the setup you suggested, NFS is being configured for remote
writes (during file upload). NFS writes should be avoided for the
following reasons:

  • Security (why allow remote write access?)

  • NFS does not offer all POSIX guarantess which might become an issue
    (usually does during writes more than reads)

  • Unavailability of the remote system translates into an inability to
    accept new uploads

On Apr 12, Leonardo C. wrote:

how would the file ever get served from the Static Server (S)?
Something has to copy the files from server D to S…

  • D is also capable of serving the static files.
  • S has a proxy pass to D with the caching options turned on.
  • User traffic always refers to S and not D for the static content.

This way, S pulls the file from D on a cache miss over HTTP.

I have been using this setup in production for a few months now. And it
works even in cases where D & S are in different continents.

Either I am understanding this incorrectly or this sounds like you have
made things worse!

If the user requests for the static content is going to enter via D,
then what is the point? User download/access of static content should
directly hit nginx running on S which accesses it’s local storage.

Well in theory user uploads to D and all requests come from S, but if
the img reside on a different server, latency issues arise.
Truly static content like Logo’s, Background etc are safely in S, but
user uploaded content is the problem.

The reason why I’d push for the caching proxy option over shared storage
is that in the setup you suggested, NFS is being configured for remote
writes (during file upload). NFS writes should be avoided for the
following reasons:

Not sure if I got the caching proxy option and how it could handle
user uploads…can you elaborate?

  • Security (why allow remote write access?)

  • NFS does not offer all POSIX guarantess which might become an issue
    (usually does during writes more than reads)

  • Unavailability of the remote system translates into an inability to
    accept new uploads

This is 100% true, it’ll add another point of failure… not sure if
there are other options though.

Cheers

Leo

On Tue, Apr 13, 2010 at 3:16 AM, Arvind Jayaprakash
[email protected] wrote:

On Apr 12, Leonardo C. wrote:

I just found out that I can do that on linode. Thanks for bringing this
idea up!

Here’s a solution:
Using NFS, mount on the Dynamic server the Static server’s directory
for static files (/images /user-uploads etc). Have nginx on Dynamic
with proxy_pass to Static for all static content. All uploads go to
the mounted drive on Dynamic (which points to Static).