Nginx removes X-Client-IP header added by loadbalancer

aris · December 8, 2012, 7:20pm

Hi,

I have a HAProxy running in front of Nginx, and I add ‘X-Client-IP’
header
to reflect the actual IP of the client. If the request is routed
directly to
the application server (Tornado in my case), this header is present and
I
can retrieve it with no problem.

However, if the request is first routed to Nginx and then to the
application
server using:

proxy_pass http://127.0.0.1:8080;
proxy_redirect off;

then the ‘X-Client-IP’ header is missing from the request when it
reaches
Tornado. I guess I might need to explicitly add this header in Nginx as
well, so it can relay it on. What is the right solution here?

Posted at Nginx Forum:

mrtn · December 8, 2012, 8:15pm

On Sat, Dec 08, 2012 at 01:20:12PM -0500, mrtn wrote:

Hi there,

However, if the request is first routed to Nginx and then to the application
server using:

proxy_pass http://127.0.0.1:8080;
proxy_redirect off;

then the ‘X-Client-IP’ header is missing from the request when it reaches
Tornado. I guess I might need to explicitly add this header in Nginx as
well, so it can relay it on. What is the right solution here?

It seems to work for me.

Can you provide an nginx.conf and nginx -V output that shows the
problem?

When I use

===
server {
listen 8000;

    location / {
        proxy_pass http://127.0.0.1:8080;
        proxy_redirect off;
    }
}

===

and try

curl -i -A “” -H X-Client-IP:1.2.3.4 http://localhost:8000/

then I can see in tcpdump output going to port 8080:

===
GET / HTTP/1.0
Host: 127.0.0.1:8080
Connection: close
Accept: /
X-Client-IP: 1.2.3.4

===

This is with nginx/1.2.4.

What’s different in your setup?

f

Francis D. [email protected]

mrtn · December 10, 2012, 1:48am

Hello!

On Sat, Dec 08, 2012 at 01:20:12PM -0500, mrtn wrote:

proxy_pass http://127.0.0.1:8080;
proxy_redirect off;

then the ‘X-Client-IP’ header is missing from the request when it reaches
Tornado. I guess I might need to explicitly add this header in Nginx as
well, so it can relay it on. What is the right solution here?

Most likely your balancer adds something like “X-Client_IP” instead,
with “_” (underscore) instead of “-” (dash). Such headers are
removed by nginx by default unless explicitly allowed, see

http://nginx.org/r/underscores_in_headers

You may either enable such headers in nginx (see above, but not
recommended), or (better) change “_” to “-” in your balancer
configs.

–
Maxim D.

mrtn · December 10, 2012, 2:31am

Hello Maxim,

Thanks for the suggestion. I checked my Haproxy config again, and this
is
what I use for adding the ‘X-Client-IP’ header:

option forwardfor header X-Client-IP

In addition, if Haproxy is passing header things like ‘X-Client_IP’,
then
why is that the requests routed directly to Tornado (not via Nginx)
contain
the correct header ‘X-Client-IP’? It seem like something happens when
the
requests go through nginx and the header is dropped.

Below is part of my nginx config, and the last location block is the
relevant one here.

worker_processes 2;
pid /var/run/nginx.pid;
daemon off;

events {
worker_connections 1024;
}

http {
include /etc/nginx/mime.types;
default_type application/octet-stream;

    sendfile                on;
    tcp_nodelay             on;
    tcp_nopush              on;
    client_body_timeout     10;
    client_header_timeout   10;
    keepalive_timeout       15;
    send_timeout            15;
    server_tokens           off;
    gzip                    on;
    gzip_http_version       1.1;
    gzip_comp_level         3;
    gzip_types              text/plain text/css
                            application/x-javascript
                            application/xml application/xml+rss
                            text/javascript;
    log_format      mylog       '$remote_addr - $remote_user

[$time_local] “$request” $status $sent_http_content_type
$body_bytes_sent
“$http_referer” “$http_user_agent”';

    server {
        listen                      8484;
        server_name                 www.mysite.com mysite.com;
        access_log

/home/www-data/logs/nginx_www_access.log;
error_log
/home/www-data/logs/nginx_www_error.log debug;
error_page 404 /404.html;
error_page 502 503 504 /50x.html;
error_page 403 /forbidden.html;

        root

/home/www-data/tornado/mysite/static;

        if ( $http_referer ~*

(babes|click|diamond|forsale|girl|jewelry|love|nudit|organic|poker|porn|sex|teen|video|webcam)
) {
return 405;
}

        location ~* (\.jpg|\.png|\.css|\.js|\.html)$ {
            valid_referers none blocked www.mysite.com

static.mysite.com
mysite.com;
if ($invalid_referer) { return 405; }
}

       location ^~ /doc/read/ {
            if ($uri ~* (\.jpg|\.png|\.css|\.js|\.html)$) { return

404;
}
if ($uri !~* /doc/read/[a-zA-Z0-9_-]+/[0-9]+$) { return
404;
}
#if ($http_cookie !~* “subject=[.]+”) { return 404; }
proxy_pass http://127.0.0.1:8080;
proxy_redirect off;
}
}
}

Posted at Nginx Forum:

mrtn · December 10, 2012, 3:59am

Hello!

On Sun, Dec 09, 2012 at 08:30:39PM -0500, mrtn wrote:

requests go through nginx and the header is dropped.
Usually headers are accessed by a backend code via CGI-like
environment variables like HTTP_X_CLIENT_IP, which makes it
impossible to distinguish “" from “-”, that’s why I suggested the
"” as a most likely reason.

Below is part of my nginx config, and the last location block is the
relevant one here.

There is nothing which may cause such a behaviour in your config,
but the “part” part is what always leaves a room for speculations.

By default nginx doesn’t removes any X-* headers from proxied
requests. If the header was indeed removed (i.e. you see it on
the wire before nginx, but not between nginx and a backend) it may
indicate one of the following:

The header was ignored as invalid, e.g. due to underscore in
it’s name. Check nginx logs for “client sent invalid header line”
messages at info level, it might provide additional information.
The header was hidden by proxy configuration, either by
proxy_set_header with the exact name, or using
proxy_pass_request_headers.
Something really bad happened which prevented request from
being parsed correctly.

In either case debug log
(A debugging log) should provide
enough info to diagnose the problem. Actually looking on the data
on the wire (e.g. with “tcpdump -Xs0” might also help).

–
Maxim D.