Weird issues with nginx

Server NGINX
1

Hello,

I was trying to add rate limiting to our setup (mp4 file serving) and
got into some very strange issues.

First, with version 1.2.0, I was seeing the following:

2013/06/14 13:25:02 [emerg] 12470#0: *147816
malloc(18446744073709551608) failed (12: Cannot allocate memory),
client: 92.247.59.175, server: localhost, request: “GET
/vid3/QkpnNF3f5RY0_D9j0L8ovw/1371217700/12/46812/46812_720x406_500k.mp4
HTTP/1.1”, host: “vm0.xxxxxxx”

The same happened with 1.2.7. I tried doing the same with 1.4.1, and
what I got was a crash, with stuff like this:
2013/06/14 20:50:58 [alert] 1690#0: worker process 26026 exited on
signal 6
2013/06/14 20:50:58 [alert] 1690#0: worker process 25973 exited on
signal 11
2013/06/14 20:50:58 [alert] 1690#0: worker process 25976 exited on
signal 11

and something like this, a bit cut:

*** glibc detected *** nginx: worker process: free(): invalid next size
(normal): 0x000000000199b900 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x76d76)[0x7f4ea69fed76]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x6c)[0x7f4ea6a03aac]
nginx: worker process[0x4059eb]
nginx: worker process[0x43067b]

Removing the ifs in the config fixes the problem, but that doesn’t help
me :slight_smile:

This is a compiled nginx from source, on debian stable (wheezy), with a
few small additions available at
https://github.com/krokodilerian/trafstat/tree/master/nginx .

Any ideas how to proceed with this, maybe adding a stack trace to find
who does the offending malloc? I’ve looked at some google results, and
some people have seen such insane mallocs (this one is -6 on a 64bit
unsigned value), when trying to use a variable that wasn’t there, but
that doesn’t seem to be my case. I have a debug log and other stuff I
can send, if that would be helpful.

Here’s my config, the file names look something
like /data/v/63/27763/27763_1920x1080_4000k.mp4 :

location /vid3/ {
error_log /usr/local/nginx/logs/debug.log debug;
location ~
^/vid3/(?[\w-=]+)/(?\w+)(?/[0-9/]+.[0-9]+x[0-9]+)(?[0-9]+00)(?k.mp4)?(?.*)$
{
mp4;
mp4_buffer_size 1m;
mp4_max_buffer_size 5m;

            if ($invalid_referer) {
                    return   403;
            }

            secure_link $rcvmd5,$linkexp;
            secure_link_md5

“XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX$remote_net$fnm$spd$kmp$linkexp” ;

            if ($secure_link = "") { # invalid
                    return 403;
            }

            if ($secure_link = "0") { # expired
                    return 410;
            }


            if ($spd = 400 ) {
                    limit_rate_after 400k;
                    limit_rate 50k;
            }

            if ($spd = 500 ) {
                    limit_rate_after 500k;
                    limit_rate 50k;
            }

            if ($spd = 1000 ) {
                    limit_rate_after 1600k;
                    limit_rate 200k;
            }

            if ($spd = 4000 ) {
                    limit_rate_after 3200k;
                    limit_rate 400k;
            }

            alias /data/v$fnm$spd$kmp;
    }
    return 404;

}

2

On Sat, Jun 15, 2013 at 04:32:04PM +0300, Vasil Kolev wrote:

Hi there,

Removing the ifs in the config fixes the problem, but that doesn’t help
me :slight_smile:

This is a compiled nginx from source, on debian stable (wheezy), with a
few small additions available at
trafstat/nginx at master · krokodilerian/trafstat · GitHub .

Any ideas how to proceed with this,

The usual first step would be to see whether the problem persists when
you just use the distributed nginx, with no external modules or patches.

But in this case, that’s probably unnecessary, since you already know
that without the “if” statements, no problems appear.

You use “if” inside “location”, and you do something other than “return
…” or “rewrite … last”. Under those circumstances, pretty much all
bets are off (unless you can explain why your use is safe).

See If is Evil… when used in location context | NGINX for some more details.

Perhaps using a “map” to set values to be used in limit_rate and
limit_rate_after directives would avoid the “if in location” problem?

f

Francis D. [email protected]

3

В 22:06 +0100 на 15.06.2013 (сб), Francis D. написа:

See If is Evil… when used in location context | NGINX for some more details.

Perhaps using a “map” to set values to be used in limit_rate and
limit_rate_after directives would avoid the “if in location” problem?

The limit_rate directive doesn’t accept a variable as a parameter,
otherwise that would’ve been easier. I might just copy the location 4
times with the speed hardcoded for each one, to get rid of the ifs,
shouldn’t be too ugly.

I’ll test this tomorrow.

4

On Sun, Jun 16, 2013 at 01:01:37AM +0300, Vasil Kolev wrote:

В 22:06 +0100 на 15.06.2013 (сб), Francis D. написа:

But in this case, that’s probably unnecessary, since you already know
that without the “if” statements, no problems appear.

Perhaps using a “map” to set values to be used in limit_rate and
limit_rate_after directives would avoid the “if in location” problem?

The limit_rate directive doesn’t accept a variable as a parameter,
otherwise that would’ve been easier.

Ah, yes, you’re correct.

$limit_rate could help there, but there doesn’t seem to be an equivalent
$limit_rate_after variable, so you’re still stuck.

I might just copy the location 4
times with the speed hardcoded for each one, to get rid of the ifs,
shouldn’t be too ugly.

That should work.

A series of server-level ifs to set the directive values might also work
– that may end up more maintainable, perhaps.

Good luck with it,

f

Francis D. [email protected]

5

В 01:01 +0300 на 16.06.2013 (нд), Vasil Kolev написа:

…" or “rewrite … last”. Under those circumstances, pretty much all
otherwise that would’ve been easier. I might just copy the location 4
times with the speed hardcoded for each one, to get rid of the ifs,
shouldn’t be too ugly.

I’ll test this tomorrow.

It definitely works this way. Is there anything that can be done to
detect whenever the “if” is used in a dangerous way? Seems like a bad
idea to be able to crash the webserver with a simple config.