Forum: NGINX proxy_pass to backend (varnish): delivered ip?

Hey,
maybe the solution is really simple, but i can't find it.

nginx (1.2.1) handles ssl and proxies the traffic to the backend 
(varnish,
which also handles http):

location / {
        proxy_set_header  Host   $host;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://127.0.0.1:80;
}

The requests are sent to varnish, in the varnish logs i see them, they 
look
like (shortened):

127.0.0.1 - - [30/Jan/2013:16:06:54 +0100] .........

So varnish receives 127.0.0.1 from nginx, but i want varnish to receive 
the
external ip.

For testing reasons i switched the ip to the lan ip (which varnish also
listens to):

location / {
        proxy_set_header  Host   $host;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://192.168.0.1:80;
}

And now the requests in varnish log look like:

192.168.0.1 - - [30/Jan/2013:16:16:25 +0100] .........

Well, where's the error? How can it be done that varnish receives the
external ip (smth. like 84.156.23.145) from nginx? I suppose it's quite
simple but i can't see it... :-/

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,235737,235737#msg-235737

On Wed, Jan 30, 2013 at 10:33:25AM -0500, revirii wrote:

Hi there,

> The requests are sent to varnish, in the varnish logs i see them, they look
> like (shortened):
>
> 127.0.0.1 - - [30/Jan/2013:16:06:54 +0100] .........
>
> So varnish receives 127.0.0.1 from nginx, but i want varnish to receive the
> external ip.

The connection to varnish comes from the address 127.0.0.1. That's what
it logs here.

> And now the requests in varnish log look like:
>
> 192.168.0.1 - - [30/Jan/2013:16:16:25 +0100] .........

The connection to varnish comes from the address 192.168.0.1. That's
what it logs here.

> Well, where's the error? How can it be done that varnish receives the
> external ip (smth. like 84.156.23.145) from nginx? I suppose it's quite
> simple but i can't see it... :-/

varnish could see the "true" remote_addr in the X-Real-IP: http header
that you send, if it looked there.

If you want varnish to log the contents of that header, or to log that
instead of what it sees as the connecting ip address, you'll be better
off to ask in a place where people know varnish.

  f
--
Francis Daly        francis@daoine.org

Hey,

thx for the fast answer :-)

> The connection to varnish comes from the address 127.0.0.1. That's
> what it logs here.

> The connection to varnish comes from the address 192.168.0.1. That's
> what it logs here.

But why? The only difference is the proxy_pass statement:

proxy_pass http://127.0.0.1:80;
vs.
proxy_pass http://192.168.0.1:80;

No other changes were done, and no changes in varnish config.

> varnish could see the "true" remote_addr in the X-Real-IP: http header
> that you send, if it looked there.
>
> If you want varnish to log the contents of that header, or to log that
> instead of what it sees as the connecting ip address, you'll be better
> off to ask in a place where people know varnish.

Hm, would be interesting which param varnish checks. It can't be
$remote_addr, so it has to be the address nginx proxies to (127.0.01 or
192.168.0.1). Very strange. So it seems to be a varnish problem? :-/


revirii

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,235737,235739#msg-235739

On Wed, Jan 30, 2013 at 11:13:38AM -0500, revirii wrote:

Hi there,

> proxy_pass http://192.168.0.1:80;
>
> No other changes were done, and no changes in varnish config.

Look at the routing table on your nginx server.

If it connects *to* 127.0.0.1, it will connect *from* 127.0.0.1 (which
is one of the nginx server's addresses).

If it connects *to* 192.168.0.1, it will connect *from* 192.168.0.1
(which is one of the nginx server's addresses).

(Probably, if it connects *to* 192.168.0.2 (which is on a different
machine), it will connect *from* 192.168.0.1.)

> Hm, would be interesting which param varnish checks. It can't be
> $remote_addr, so it has to be the address nginx proxies to (127.0.01 or
> 192.168.0.1).

No, it is the address that the connection to varnish comes *from*. 
Because
of your specific setup, that happens to match the address that nginx
connects to. But try connecting to varnish from some other machine and
you'll see the difference.

> Very strange. So it seems to be a varnish problem? :-/

It's usually considered a feature that the source address of a 
connection
is logged. There is nothing nginx can do to hide its source address.

What you want is something non-standard. Possibly there's a varnish
configuration to allow it.

  f
--
Francis Daly        francis@daoine.org

Hello,

Am 30.01.2013 17:13, schrieb revirii:

>> varnish could see the "true" remote_addr in the X-Real-IP: http
>> header
>> that you send, if it looked there.
>
> Hm, would be interesting which param varnish checks. It can't be
> $remote_addr, so it has to be the address nginx proxies to (127.0.01
> or
> 192.168.0.1). Very strange. So it seems to be a varnish problem? :-/

You can see the right X-Forwarded-For header with tcpdump.

Take a look at the log format varnish uses. I don't know how to
configure varnish, but with Apache you have to change your LogFormat
from

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\"
\"%{User-Agent}i\""  combined

to something like

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b
\"%{Referer}i\"\"%{User-Agent}i\"" nginx

rgds, Axel

--
Never argue with an idiot; people watching may not tell the difference

Good Morning,

>
> LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b
> \"%{Referer}i\"\"%{User-Agent}i\"" nginx

Good idea. I'll check if varnish is able to do that.

thx
revirii

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,235737,235761#msg-235761

Good Morning,

> If it connects *to* 127.0.0.1, it will connect *from* 127.0.0.1 (which
> is one of the nginx server's addresses).
>
> If it connects *to* 192.168.0.1, it will connect *from* 192.168.0.1
> (which is one of the nginx server's addresses).

Hm, the said nginx vhost doesn't use 127.0.0.1 or a lan address in
combination with port 80 - varnish listens on *.80. the nginx ssl vhost
listens on an "real" ip and passes the requets to varnish on 
127.0.0.1:80.
Or is there a misunderstanding on my side?

> No, it is the address that the connection to varnish comes *from*.
> Because
> of your specific setup, that happens to match the address that nginx
> connects to. But try connecting to varnish from some other machine and
> you'll see the difference.

Well, it's not that special, i think.
http: request -> varnish on real ip -> nginx backend on 127.0.0.1:81
https: request -> nginx on real ip:443 -> proxy_pass to varnish on
127.0.01.80 -> nginx backend on 127.0.0.1:81
(If necessary i could draw a small picture)

> It's usually considered a feature that the source address of a
> connection
> is logged. There is nothing nginx can do to hide its source address.
>
> What you want is something non-standard. Possibly there's a varnish
> configuration to allow it.

Ok, so what your're trying to tell me is:
if i proxy_pass to varnish with "proxy_pass http://127.0.0.1:80;" nginx 
uses
some localhost address (although it can't be 127.0.0.1:80, which is used 
by
varnish) to connect to varnish, and varnish sees this localhost address

if i proxy_pass to varnish with "proxy_pass http://192.168.0.1:80;" 
nginx
uses some lan address (although it can't be 192.168.0.1:80, which is 
used by
varnish) to connect to varnish, and varnish sees this lan address

if i proxy_pass to varnish with "proxy_pass http://real_ip:80;" nginx 
uses
the real ip (although it can't be real_ip:80, which is used by varnish) 
to
connect to varnish, and varnish sees this real_ip address

If so, there's nothing i can do within nginx config, as it's simply not
possible. And varnish config or log is the (only) place where i can 
achieve
this.

thx a lot
revirii

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,235737,235762#msg-235762

On Thu, Jan 31, 2013 at 02:13:39AM -0500, revirii wrote:

Hi there,

> > If it connects *to* 127.0.0.1, it will connect *from* 127.0.0.1 (which
> > is one of the nginx server's addresses).
> >
> > If it connects *to* 192.168.0.1, it will connect *from* 192.168.0.1
> > (which is one of the nginx server's addresses).
>
> Hm, the said nginx vhost doesn't use 127.0.0.1 or a lan address in
> combination with port 80 - varnish listens on *.80. the nginx ssl vhost
> listens on an "real" ip and passes the requets to varnish on 127.0.0.1:80.
> Or is there a misunderstanding on my side?

Yes.

nginx as a *server* listens on whatever ip:port pairs are in the
configuration.

nginx as a *client* uses the operating system's facilities to set the
client connection information.

When nginx talks to varnish, nginx is the client and varnish is the
server.

> Ok, so what your're trying to tell me is:
> if i proxy_pass to varnish with "proxy_pass http://127.0.0.1:80;" nginx uses
> some localhost address (although it can't be 127.0.0.1:80, which is used by
> varnish) to connect to varnish, and varnish sees this localhost address

The proxy_pass connection comes from the address and port that nginx
chooses -- which is usually "whatever the OS chooses", because nginx
doesn't care to choose. In this case, the address will be 127.0.0.1,
and the port will be something unpredictable you usually don't care 
about.

> if i proxy_pass to varnish with "proxy_pass http://real_ip:80;" nginx uses
> the real ip (although it can't be real_ip:80, which is used by varnish) to
> connect to varnish, and varnish sees this real_ip address

If real_ip is configured on the machine, then yes. If not, then the
OS will choose the appropriate locally-configured address (plus an
unpredictable port) to make the connection.

> If so, there's nothing i can do within nginx config, as it's simply not
> possible. And varnish config or log is the (only) place where i can achieve
> this.

That part is correct.

  f
--
Francis Daly        francis@daoine.org