Proxy passing and the URI

mike · September 23, 2008, 7:46am

I have a request for http://foo.com/sites/something

I have this location block inside of server{} with root
/home/foo/web/foo.com

location ^~ /sites {
proxy_pass http://10.122.47.82;
proxy_set_header Host foo.com;
}

Trying to pass it to a second server.

The problem is, the second server receives this as the full URI; I
have to define “root” to be the base URI (/home/foo/web/foo.com) so
the /sites/something maps to it properly. Is there any way to remove
parts of the URI when passing via proxy? So the /sites/something/
isn’t needed on the upstream server?

Thanks

mike · September 23, 2008, 11:12am

Hello!

On Mon, Sep 22, 2008 at 10:31:02PM -0700, mike wrote:

The problem is, the second server receives this as the full URI; I
have to define “root” to be the base URI (/home/foo/web/foo.com) so
the /sites/something maps to it properly. Is there any way to remove
parts of the URI when passing via proxy? So the /sites/something/
isn’t needed on the upstream server?

There are two basic options:

Use proxy_pass with uri, e.g.

location ^~ /sites {
proxy_pass http://10.122.47.82/new;
}

This will replace ‘/sites’ part (the part that matched location)
with the ‘/new’ in the uri passed to backend server. So if you
write

 location ^~ /sites {
     proxy_pass http://10.122.47.82/;
 }

the ‘/sites’ part will be removed.

See http://wiki.codemongers.com/NginxHttpProxyModule#proxy_pass
for details.

Modify uri as needed with rewrite before proxy_pass. This less
efficient but may be used where 1 can’t be (e.g. in regex
locations). E.g.

location ~ ^/sites..cgi$ {
rewrite ^/sites(.) $1 break;
proxy_pass http://backend;
}

Maxim D.

mike · September 23, 2008, 4:19pm

On Tue, Sep 23, 2008 at 1:52 AM, Maxim D. [email protected]
wrote:

location ^~ /sites {
proxy_pass http://10.122.47.82/;
}

the ‘/sites’ part will be removed.

Weird, so that missing “/” at the end would have fixed my issue? I
-think- on a quick check it has. I thought I tried that earlier and it
didn’t help!

See http://wiki.codemongers.com/NginxHttpProxyModule#proxy_pass for details.

Modify uri as needed with rewrite before proxy_pass. This less efficient
but may be used where 1 can’t be (e.g. in regex locations). E.g.

location ~ ^/sites..cgi$ {
rewrite ^/sites(.) $1 break;
proxy_pass http://backend;
}

Yeah, I’ve tried this before too, executing rewrites before the
location block and such. I can’t really do it here - I could do it
-inside- of the location block however, if needed. This current host
supports a ton of other rewrites and it could get confusing and/or
break something if I did any more rewrites outside of the location
block!

Thanks Maxim!

mike · September 23, 2008, 7:01pm

Hello,

I am receiving a strange segault in dmesg every few hours… and then
nginx orphans those connections, so after about a day there are
2000-3000 connections that are just “waiting”.

This isn’t that bad, as the server has received over 1,228,503
connections since a restart this morning, but still those orphaned
connections pile up, and I have to restart nginx to clear them.

Here is what I am receiving:

nginx[27715]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27679]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27690]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27727]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27728]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27677]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27706]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27680]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[5214]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 error
4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27686]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27688]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27678]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27717]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27682]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27726]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27724]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
__ratelimit: 34 callbacks suppressed
nginx[27720]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[22566]: segfault at c ip 00000030d106c942 sp 00007fffcd340798
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27685]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[22587]: segfault at c ip 00000030d106c942 sp 00007fffcd340798
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27722]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27696]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27691]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27701]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[23722]: segfault at c ip 00000030d106c942 sp 00007fffcd340798
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[22562]: segfault at c ip 00000030d106c942 sp 00007fffcd340798
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27719]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[22578]: segfault at c ip 00000030d106c942 sp 00007fffcd340798
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[22560]: segfault at c ip 00000030d106c942 sp 00007fffcd340798
error 4 in libkrb5.so.3.3[30d1000000+9f000]

I know that libkrb5 is for kerberos… I don’t think I am even using
that in my configuration. No DAV support, etc… So I don’t understand
exactly why nginx is faulting.

Please let me know what additional information I can send that would be
helpful.

Thanks,

John

mike · September 23, 2008, 7:14pm

On Tue, Sep 23, 2008 at 11:49:05AM -0500, Resicow wrote:

Here is what I am receiving:
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27688]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27719]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
Please let me know what additional information I can send that would be
helpful.

What does “ldd /path/to/nginx” show ?

mike · September 23, 2008, 9:11pm

Igor S. wrote:

connections since a restart this morning, but still those orphaned
nginx[27727]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27726]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
nginx[22587]: segfault at c ip 00000030d106c942 sp 00007fffcd340798
error 4 in libkrb5.so.3.3[30d1000000+9f000]
that in my configuration. No DAV support, etc… So I don’t understand
exactly why nginx is faulting.

Please let me know what additional information I can send that would be
helpful.

What does “ldd /path/to/nginx” show ?

Here is the output:

ldd /usr/local/nginx/sbin/nginx
linux-vdso.so.1 => (0x00007fff745fe000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00000030d5c00000)
libpcre.so.0 => /lib64/libpcre.so.0 (0x00000030d8800000)
libssl.so.7 => /lib64/libssl.so.7 (0x00000030d3c00000)
libcrypto.so.7 => /lib64/libcrypto.so.7 (0x00000030d0400000)
libz.so.1 => /lib64/libz.so.1 (0x00000030c4800000)
libc.so.6 => /lib64/libc.so.6 (0x00000030c3800000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
(0x00000030d2000000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00000030d1000000)
libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00000030d0000000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3
(0x00000030d0c00000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000030c4000000)
/lib64/ld-linux-x86-64.so.2 (0x00000030c2600000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0
(0x00000030d0800000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00000030d1400000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00000030cac00000)
libselinux.so.1 => /lib64/libselinux.so.1 (0x00000030c4c00000)

Also, I just want to thank you for creating and maintaining such a great
product!

Thanks,

John

mike · September 23, 2008, 9:19pm

On Tue, Sep 23, 2008 at 12:49:42PM -0500, Resicow wrote:

Igor S. wrote:

On Tue, Sep 23, 2008 at 11:49:05AM -0500, Resicow wrote:

Hello,

I am receiving a strange segault in dmesg every few hours… and then
nginx orphans those connections, so after about a day there are
2000-3000 connections that are just “waiting”.

These orphan connections leave from segfaulted workers.
This is wrong counter statistics only. It does not actually leak
OS resources.

nginx[27690]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[27682]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
nginx[27685]: segfault at c ip 00000030d106c942 sp 00007fffcd340728
error 4 in libkrb5.so.3.3[30d1000000+9f000]
   libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 
   libselinux.so.1 => /lib64/libselinux.so.1 (0x00000030c4c00000)

It seems that librkb5 was linked via libssl/etc. You may try to do two
things:

build nginx without SSL,
build nginx without stripping debug info and allow to create
coredump.
Then I can invetsigate the bug.

mike · September 24, 2008, 1:14am

Igor S. wrote:

This isn’t that bad, as the server has received over 1,228,503
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[5214]: segfault at c ip 00000030d106c942 sp 00007fffcd340798 error
error 4 in libkrb5.so.3.3[30d1000000+9f000]
error 4 in libkrb5.so.3.3[30d1000000+9f000]
nginx[23722]: segfault at c ip 00000030d106c942 sp 00007fffcd340798
I know that libkrb5 is for kerberos… I don’t think I am even using
   libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 
   libselinux.so.1 => /lib64/libselinux.so.1 (0x00000030c4c00000)
It seems that librkb5 was linked via libssl/etc. You may try to do two things:

build nginx without SSL,

build nginx without stripping debug info and allow to create coredump.
Then I can invetsigate the bug.

I’m glad that the orphaned connections do not use system resources… at
least the system won’t be taken down and run out of available
connections.

The faults are very strange, since I have a replica setup on replica
hardware, and it doesn’t produce faults. I’ll have some time in a few
days to take the system down and create a coredump, but SSL support is
required and used in my setup. Also the faults come in waves, which is
also strange.

Both servers have the same version of libkrb5 as well… so maybe there
is some client (or attacker) trying to create SSL sessions in a bad way.

Is there anyway to find out what “error 4” is in libkrb5.so.3? Should I
upgrade openssl and then rebuild?

Thanks Again,

John