# Running nginx upload progress module in a cluster?

**URL:** <https://www.ruby-forum.com/t/running-nginx-upload-progress-module-in-a-cluster/193241>\
**Category:** NGINX\
**Created:** [August 13, 2010, 10:02pm UTC](https://www.ruby-forum.com/t/running-nginx-upload-progress-module-in-a-cluster/193241 "2010-08-13T22:02:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![sthorpe](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/e36b37/32.png) [@sthorpe](https://www.ruby-forum.com/u/sthorpe)\
**Post date:** [August 13, 2010, 10:02pm UTC](https://www.ruby-forum.com/t/running-nginx-upload-progress-module-in-a-cluster/193241/1 "2010-08-13T22:02:29Z")

</div>

Hi,

I’m experimenting with the nginx upload progress module for a system I’m  
building. At some point, I expect I’ll need a cluster of “upload” nodes  
(currently planning to use the upload and upload progress modules) in  
order  
to handle all incoming uploads. The problem I’m expecting to have w/  
the  
upload progress module is, if I’m running a cluster of nodes w/ a shared  
IP  
(and/or behind a load balancer), is how to ensure “progress” requests  
are  
routed to the same host as the upload POST, (assuming POSTs always go to  
the  
same domain, e.g., [uploads.example.com](http://uploads.example.com)).

I’m currently planning to run this system on EC2 and am considering the  
following options, but thought I’d ask here to see if anyone has solved  
this  
problem; has other, better ideas; or might correct my assumptions –  
since I  
don’t particularly like any of these options.

Option 1: use Elastic Load Balancer with session stickiness  
Problem: ELB is time-based so it is possible for the session to  
“unstick” in  
the middle of an upload; higher time expiration settings could mitigate  
(but  
not prevent) this but work against the load balancers ability to balance  
traffic

Option 2: Distribute requests amongst nodes using a hash on  
the X-Progress-ID value and subdomains (e.g., hash ids to a value a-z,  
map  
[a-z].uploads.example.com evenly across N nodes; always post to the  
subdomain as a hash of X-Progress-ID)  
Problem: Spreads posts evenly, but complicates client logic and can’t  
easily  
intelligently balance requests based on current load

Option 4: Make the app pick a host (client must ask where to POST before  
POSTing upload)  
Problem: pushes load balancing work into the server app & makes the  
client  
do more work

Option 3: RYO load balancer (build (or configure) a custom load balancer  
that can run on EC2 instances and route requests intelligently)  
Problem: Seems like overkill

Anyway – thanks for any insights/thoughts.

Scott

---

<div class="post-metadata">

**Author:** ![sthorpe](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/e36b37/32.png) [@sthorpe](https://www.ruby-forum.com/u/sthorpe)\
**Post date:** [August 14, 2010, 5:19am UTC](https://www.ruby-forum.com/t/running-nginx-upload-progress-module-in-a-cluster/193241/2 "2010-08-14T05:19:43Z")

</div>

2010/8/14 Scott T. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

> I’m currently planning to run this system on EC2 and am considering the  
> [a-z].uploads.example.com evenly across N nodes; always post to the  
> Anyway – thanks for any insights/thoughts.  
> Scott

Some time ago we faced the same problem and i decided to use the  
“Option 2” approach. We have 2 HAProxy frontends and 3 nginx backends  
in our upload cluster. So far so good. And about what complicated  
client logic are you talking about?

---

<div class="post-metadata">

**Author:** ![sthorpe](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/s/e36b37/32.png) [@sthorpe](https://www.ruby-forum.com/u/sthorpe)\
**Post date:** [August 15, 2010, 5:08pm UTC](https://www.ruby-forum.com/t/running-nginx-upload-progress-module-in-a-cluster/193241/3 "2010-08-15T17:08:13Z")

</div>

On 13/08/10 22:01, Scott T. wrote:

> always go to the same domain, e.g., [uploads.example.com](http://uploads.example.com)  
> could mitigate (but not prevent) this but work against the load  
> POSTing upload)  
> Problem: pushes load balancing work into the server app & makes the  
> client do more work
> 
> Option 3: RYO load balancer (build (or configure) a custom load balancer  
> that can run on EC2 instances and route requests intelligently)  
> Problem: Seems like overkill

## When being asked this very same question, I always answer either 1) if the load balancer can use sticky sessions on request arguments (ie follow X-Progress-ID value), or 2). Both should be simple to build, 1) as configuration and 2) as client code… choose your poison 🙂

Brice F.  
My blog: [http://www.masterzen.fr/](http://www.masterzen.fr/)