Realip module broken?

We have been using (and relying) on the realip module for a while now to
extract ‘X-Real-IP’ headers from our loadbalancers so our backends get
REMOTE_ADDR variables. Everything seemed to work fine.

However today I was tracing some of my own debugging on a live server
(lots of rps) and was surprised by the output I got. Because I was doing
HEAD requests and all the other requests are GET’s, a simple grep was
enough to filter out my debugging requests from the access_log

www.mydomain.com 200.199.140.205 - - [12/Aug/2008:09:57:04 -0400] HEAD /
HTTP/1.1 “403” 0 “-” “curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS”
www.mydomain.com 200.199.140.205 - - [12/Aug/2008:09:57:06 -0400] HEAD /
HTTP/1.1 “403” 0 “-” “curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS”
www.mydomain.com 88.80.254.219 - - [12/Aug/2008:09:57:10 -0400] HEAD /
HTTP/1.1 “403” 0 “-” “curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS”
www.mydomain.com 88.80.254.219 - - [12/Aug/2008:09:57:12 -0400] HEAD /
HTTP/1.1 “403” 0 “-” “curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS”
www.mydomain.com 66.153.141.107 - - [12/Aug/2008:09:57:55 -0400] HEAD /
HTTP/1.1 “403” 0 “-” “curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS”
www.mydomain.com 72.187.80.90 - - [12/Aug/2008:09:58:16 -0400] HEAD /
HTTP/1.1 “403” 0 “-” “curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS”
www.mydomain.com 72.187.80.90 - - [12/Aug/2008:09:58:30 -0400] HEAD /
HTTP/1.1 “403” 0 “-” “curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
OpenSSL/0.9.8g zlib/1.2.3 libssh2/0.15-CVS”

But hold on… Why am I getting al those random source IP’s?

After confirming (using a strace) that the correct headers are being
sent to Nginx, the only conclusion I can draw is that the realip module
is mixing up IP adresses. Zipping throught the logs seems to confirm
this, when I do a request I either see my own IP address (correct) or
the IP address from the request right before me (incorrect).

Does this ring a bell for anyone? It seems to me like the realip module
is seriously broken (at least in 0.6.32). I had a look at the source
code, but cannot come with anything obvious.

Spil G. wrote:

Does this ring a bell for anyone? It seems to me like the realip module
is seriously broken (at least in 0.6.32). I had a look at the source
code, but cannot come with anything obvious.

I think I figured it out. We run multiple Nginx backends behind a Zeus
ZXTM loadbalancer. The loadbalancer is configured to use keepalives to
the backends and thus pipes requests from various source addresses
through one connection.

When the first request on a new lb->ws connection comes in, Nginx
matches the source address to the value set through ‘set_real_ip_from’.
Let’s assume the request originates from the loadbalancer (obviously :P)
and that we have set ‘set_real_ip_from’ appropriately. The realip module
will patch the following:

sin->sin_addr.s_addr = addr;
r->connection->addr_text.len = len;
r->connection->addr_text.data = p;

So the IP address is corrected for this connection. So far, so good.

Now to the second request: Nginx will try to match the source address
again, but because this is the same (already patched) connection, it
will not match ‘set_real_ip_from’ and the address will pass unmodified,
which is incorrect because this request originates from a different
client ip address.

I can work around this problem by setting ‘set_real_ip_from’ to
‘0.0.0.0/0’, but I think this is essentially a bug in the module. It
should not patch the source ip address for the whole (keepalive)
connection, but only for the current request.

On Tue, Aug 12, 2008 at 04:07:39PM +0200, Spil G. wrote:

HTTP/1.1 “403” 0 “-” "curl/7.16.3 (i686-pc-cygwin) libcurl/7.16.3
www.mydomain.com 66.153.141.107 - - [12/Aug/2008:09:57:55 -0400] HEAD /
But hold on… Why am I getting al those random source IP’s?

After confirming (using a strace) that the correct headers are being
sent to Nginx, the only conclusion I can draw is that the realip module
is mixing up IP adresses. Zipping throught the logs seems to confirm
this, when I do a request I either see my own IP address (correct) or
the IP address from the request right before me (incorrect).

Does this ring a bell for anyone? It seems to me like the realip module
is seriously broken (at least in 0.6.32). I had a look at the source
code, but cannot come with anything obvious.

What loadbalancer (that sets X-Real-IP) do you use ?

On Tue, Aug 12, 2008 at 04:57:10PM +0200, Spil G. wrote:

When the first request on a new lb->ws connection comes in, Nginx

Now to the second request: Nginx will try to match the source address
again, but because this is the same (already patched) connection, it
will not match ‘set_real_ip_from’ and the address will pass unmodified,
which is incorrect because this request originates from a different
client ip address.

I can work around this problem by setting ‘set_real_ip_from’ to
‘0.0.0.0/0’, but I think this is essentially a bug in the module. It
should not patch the source ip address for the whole (keepalive)
connection, but only for the current request.

I suspected those are loadbalancer keepalive connections.
I will look how to resolve the issue. Right now you may disable
keepalive
on nginx side.

Igor S. wrote:

What loadbalancer (that sets X-Real-IP) do you use ?

Our posts crossed eachother, I think. Please see my previous mail. We’re
running Zeus ZXTM. I have implemented the mentioned workaround now, and
it seems to work. I have no clue however how to repair the bug in the
module. It probably requires saving the original source address
somewhere.

Igor S. wrote:

I will look how to resolve the issue. Right now you may disable
keepalive on nginx side.

I’ll keep it at 0.0.0.0/0 for now. The ‘set_real_ip_from’ directive
doesn’t add much security anyway:

a) The loadbalancer overwrites any existing X-Real-IP headers.

b) Even if a) would not be done, the header would be accepted because
all requests come from the loadbalancer IP.

It would be nice if the realip module could be fixed though. It’s a
matter of semantics, but I believe an X-Real-IP (or X-Forwarded-For)
header should only influence the request, not the entire connection.

Thanks.

Igor S. wrote:

The attached patch should fix the issue.

Thanks!

The patch did not apply cleanly to 0.6.32. Attached patch does.

On Wed, Aug 13, 2008 at 10:23:27AM +0200, Spil G. wrote:

all requests come from the loadbalancer IP.

It would be nice if the realip module could be fixed though. It’s a
matter of semantics, but I believe an X-Real-IP (or X-Forwarded-For)
header should only influence the request, not the entire connection.

The attached patch should fix the issue.