# Nginx behind load balancer

**URL:** <https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087>\
**Category:** NGINX\
**Created:** [November 21, 2011, 1:26pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087 "2011-11-21T13:26:12Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [November 21, 2011, 1:26pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/1 "2011-11-21T13:26:12Z")

</div>

Hi Guys,

This weekend for scalability we tried putting our nginx servers behind  
amazon’s elastic load balancers and came across a road block: it does  
not  
transparently pass the user IP and header information to nginx. This  
caused  
issues with several pieces of nginx we use including the IP allow / deny  
rules, the limit\_req module, and the limit\_con module. Has anyone  
successfully put nginx behind a load balancer? Any ideas on how to make  
this work?

Thanks!  
Rami

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 21, 2011, 1:38pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/2 "2011-11-21T13:38:37Z")

</div>

Hello!

On Mon, Nov 21, 2011 at 07:25:39AM -0500, Rami E. wrote:

> Hi Guys,
> 
> This weekend for scalability we tried putting our nginx servers behind  
> amazon’s elastic load balancers and came across a road block: it does not  
> transparently pass the user IP and header information to nginx. This caused  
> issues with several pieces of nginx we use including the IP allow / deny  
> rules, the limit\_req module, and the limit\_con module. Has anyone  
> successfully put nginx behind a load balancer? Any ideas on how to make  
> this work?

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

Maxim D.

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 21, 2011, 1:41pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/3 "2011-11-21T13:41:07Z")

</div>

Thanks Maxim,

This looks like exactly what we need. In your experience does this  
resolve  
most issues behind a load balancer?

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 25, 2011, 3:54pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/4 "2011-11-25T15:54:39Z")

</div>

Hi Maxim,

We implemented to module and still had some trouble. A lot of the  
connections would return " 503 Service Temporarily Unavailable". Our  
configuration works fine without the load balancer but then gives these  
503  
errors behind the load balancer.

Looking into the error logs I notice a lot of these errors both with and  
without the load balancer “connect() failed (111: Connection refused)  
while  
connecting to upstream”. Could this be the reason that we are having  
issues?

Thanks!  
Rami

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 21, 2011, 1:46pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/5 "2011-11-21T13:46:42Z")

</div>

Hello!

On Mon, Nov 21, 2011 at 07:40:49AM -0500, Rami E. wrote:

> Thanks Maxim,
> 
> This looks like exactly what we need. In your experience does this resolve  
> most issues behind a load balancer?

Yes.

Maxim D.

> > > transparently pass the user IP and header information to nginx. This
> > 
> > * * *
> > 
> > nginx mailing list  
> > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> > [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 25, 2011, 6:15pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/6 "2011-11-25T18:15:21Z")

</div>

Hello!

On Fri, Nov 25, 2011 at 09:54:14AM -0500, Rami E. wrote:

> Hi Maxim,
> 
> We implemented to module and still had some trouble. A lot of the  
> connections would return " 503 Service Temporarily Unavailable". Our  
> configuration works fine without the load balancer but then gives these 503  
> errors behind the load balancer.

nginx itself will only return 503 if it hits either limit\_conn or  
limit\_req.

If you see this returned by nginx, and it only happens with load  
balancer, this may indicate you’ve not configured realip module  
properly (or your load balancer doesn’t provide appropriate  
headers) and you are hitting per-ip limits configured due to all  
requests appear to be from load balancer.

Check if client’s ip logged is really client’s one, not an ip of  
your load balancer.

> Looking into the error logs I notice a lot of these errors both with and  
> without the load balancer “connect() failed (111: Connection refused) while  
> connecting to upstream”. Could this be the reason that we are having  
> issues?

Unlikely, but It’s a good idea to track and fix this in any case.

Maxim D.

> > > On Mon, Nov 21, 2011 at 7:38 AM, Maxim D. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)
> > > 
> > > > > amazon’s elastic load balancers and came across a road block: it  
> > > > > [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/HttpRealIpModule)  
> > > > > nginx mailing list  
> > > > > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> > > > > [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)
> > 
> > * * *
> > 
> > nginx mailing list  
> > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> > [mailman.nginx.org archive](http://mailman.nginx.org/mailman/listinfo/nginx)

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 25, 2011, 10:19pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/7 "2011-11-25T22:19:53Z")

</div>

On Fri, Nov 25, 2011 at 12:37 PM, Rami E. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Thanks for the help Maxim! We disabled our limit\_req and that seemed to  
> have fixed the problem. Looking at the logs it seems that only 1/3 of the  
> requests are correctly getting the new IP assigned via the realIP module,  
> the remainder are still logging the load balancer IP. This probably is more  
> of an issue with the amazon load balancer but do you have any idea on what  
> may be going on?  
> Also, where would you recommend as aplaceto start tracking and fixing the  
> other issue?

You have not established that load balancer is setting X-Forwarded-For  
or some other header to pass Real IP.

Does the amazon pseudo device have enough sockets available to connect  
to nginx?

What do you expect from adding this layer? Were you hitting  
performance limits? Has it improved or degraded performance?

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 25, 2011, 6:38pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/8 "2011-11-25T18:38:25Z")

</div>

Thanks for the help Maxim! We disabled our limit\_req and that seemed to  
have fixed the problem. Looking at the logs it seems that only 1/3 of  
the  
requests are correctly getting the new IP assigned via the realIP  
module,  
the remainder are still logging the load balancer IP. This probably is  
more of an issue with the amazon load balancer but do you have any idea  
on  
what may be going on?

Also, where would you recommend as a place to start tracking and fixing  
the  
other issue?

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 25, 2011, 11:26pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/9 "2011-11-25T23:26:15Z")

</div>

As a quick update, it looks like this has happened before. The load  
balancer bounces off of several internal IP’s sometimes and nginx picks  
the  
last one only. Does anyone know of a workaround to remove the last two  
trusted IPs from the x-forwarded-for header?

> **[Geo check behind load balancer](https://forum.nginx.org/read.php?11,26102,214069)**
>
> I have a number of nginx servers running on EC2 behind Amazon's "Elastic Load Balancer". I want to find the country of the original requester. My "geo" line originally looked like this: \[code\] geo $country { ... }

Rami

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 25, 2011, 10:55pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/10 "2011-11-25T22:55:13Z")

</div>

In looking at the $proxy\_add\_x\_forwarded\_for variable I believe that the  
Loadbalancer is in fact passing the variables but nginx is taking the  
wrong  
value?

Here is what i get from the variables.  
$proxy\_add\_x\_forwarded\_for: 217.27.244.18, 10.160.43.200, 10.160.43.200  
$remote\_addr: 10.160.43.200

Does this mean that nginx is taking the last value in the  
x-forwarded-for?

To answer your other question, yes, the Amazon plenty of sockets  
available  
to connect to nginx.

The reason we added this was to be able to utilize Amazon’s auto scaling  
feature for redundancy and scalability. We have not been able to test  
performance yet since we are still trying to make it work but as soon as  
we  
do I will be sure to report back.

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 29, 2011, 10:37am UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/11 "2011-11-29T10:37:59Z")

</div>

Thanks for the note and the clever workaround. We were able to tweak  
this  
to work with but then it still left a lot of the other functions we were  
using such as deny/allow, limit\_con, ect not working. Instead we went  
back  
to amazon and it turns out they were able to correct the behavior of  
their  
load balancer.

I wanted to report back on the performance of putting nginx behind an  
elb.  
We compared elb to haproxy and on amazon’s cloud we got better  
performance  
through the elb than we did an haproxy instance. There is minimal impact  
on  
end user performance for adding this extra step. We did this for  
redundancy to allow us to automatically fail over to another zone if the  
current zone or instance go down.

Happy to answer any questions about the setup.

Rami

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [October 3, 2012, 10:27am UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/12 "2012-10-03T10:27:35Z")

</div>

Rami E. wrote in post #1034227:

> Thanks for the note and the clever workaround. We were able to tweak  
> this  
> to work with but then it still left a lot of the other functions we were  
> using such as deny/allow, limit\_con, ect not working. Instead we went  
> back  
> to amazon and it turns out they were able to correct the behavior of  
> their  
> load balancer.
> 
> I wanted to report back on the performance of putting nginx behind an  
> elb.  
> We compared elb to haproxy and on amazon’s cloud we got better  
> performance  
> through the elb than we did an haproxy instance. There is minimal impact  
> on  
> end user performance for adding this extra step. We did this for  
> redundancy to allow us to automatically fail over to another zone if the  
> current zone or instance go down.
> 
> Happy to answer any questions about the setup.
> 
> Rami

Hi Rami,

We are looking to implement this on our web servers - did you have to  
change the default configuration of the ELB?

Would you be able to copy and paste the final configuration of nginx  
that works?

You help is much appreciated,

Brian

---

<div class="post-metadata">

**Author:** ![Rami\_E](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/f05b48/32.png) [@Rami\_E](https://www.ruby-forum.com/u/Rami_E)\
**Post date:** [November 26, 2011, 9:45pm UTC](https://www.ruby-forum.com/t/nginx-behind-load-balancer/214087/13 "2011-11-26T21:45:39Z")

</div>

$remote\_addr(OSI 3layer) and $http\_x\_forwarded\_for(OSI 7layer) of these  
two  
variables is different. behind LB server may be you can use config like  
this.  
http {  
map $proxy\_add\_x\_forwarded\_for $xff\_pass {  
default “xff403”;  
~^61.152.90.8 “xffpass”;  
~^61.172.241. “xffpass”;  
limit\_req\_zone $http\_x\_forwarded\_for zone=one:10m  
rate=200r/s;  
…  
server {  
if ($xff\_pass !~\* xffpass) {  
return 403;  
}  
limit\_req zone=one;  
…  
}  
}

2011/11/26 Rami E. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)