Nginx 1.6 under load gives a ton of error 403

Server NGINX
1

Hello,

we have used Nginx 1.4.x (the “extras”, full featured package) for a
year
with no major troubles. Only some relatively rare error 403 we can’t
find
the source of, it only happens when our XEN VPS (Ubuntu 12.04 LTS) is
under
(other sharing customers) load and our website is itself under high
load.

I have upgraded to 1.6, the (Ondrej PPA). I had to downgrade to 1.4.7
VERY
fast because error 403 (access forbidden) were literally spammed every
other
page just with a smidge of load.

The same content shows not a single error is used under light load so
it’s
not a basic “wrong permissions” or similar issue.

I’d love if somebody could give me pointers about how to find out the
real
source of those errors 403. I can’t replicate the issue on a basic
“mule”
computer, our setup uses a lot of features including varnish 3, GeoIP
both
at Nginx and PHP-fpm levels and so on.

The odd thing is, version 1.4.7 very seldom shows that spurious error,
whereas to 1.6 it is a showstopper. To worsen things, sometimes varnish
caches those 403 pages so unrelated users start seeing those errors as
well.

I think the issue might be caused by too small buffers or something. The
config file is massively huge so I don’t know if it’s a good idea to
post it
(nobody would care to read it all).

So I am just looking for pointers and ideas about how to pinpoint the
problem source.

Best regards,
dfumagalli

Posted at Nginx Forum:

2

Domain anonymized sample error log: see the first entry returns 200, the
next 403 on the same page.

2.139.79.51 - - [06/May/2014:08:13:23 +0000] “GET /flowers/love
HTTP/1.1”
200 62
55 “http://dev.domain.com/flowers/friendship” “Mozilla/5.0 (Windows NT
6.3;
WOW64; rv:28.0) Gecko/20100101 Firefox/28.0”
2.139.79.51 - - [06/May/2014:08:13:28 +0000] “GET / HTTP/1.1” 403 134
“http://de
v.domain.com/flowers/love” “Mozilla/5.0 (Windows NT 6.3; WOW64; rv:28.0)
Gec
ko/20100101 Firefox/28.0”
2.139.79.51 - - [06/May/2014:08:15:01 +0000] “GET / HTTP/1.1” 403 134
“http://de
v.domain.com/flowers/love” “Mozilla/5.0 (Windows NT 6.3; WOW64; rv:28.0)
Gec
ko/20100101 Firefox/28.0”
2.139.79.51 - - [06/May/2014:08:17:31 +0000] “GET / HTTP/1.1” 403 134
“http://de
v.domain.com/flowers/love” “Mozilla/5.0 (Windows NT 6.3; WOW64; rv:28.0)
Gec
ko/20100101 Firefox/28.0”
2.139.79.51 - - [06/May/2014:08:17:52 +0000] “GET / HTTP/1.1” 403 134
“http://de
v.domain.com/flowers/love” “Mozilla/5.0 (Windows NT 6.3; WOW64; rv:28.0)
Gec
ko/20100101 Firefox/28.0”

Posted at Nginx Forum:

3

It is probably your application / backend that is generating the 403,
it’s
unlikely nginx is responsible for this. I guess rate / connection
limiting
with a custom error code may cause this, but you should know if you
configured this. Please show us your config and describe your backend in
more detail.

4

Your config is returning a 403 from any referrer containing “love” any
you
have such URLs on your own site according to your log excerpt. I would
not
recommend such referrer matching, it’s unlikely to help in any case.

5

This is also what I thought. I have searched the whole nginx etc
directory
for 403 and deny

/etc/nginx# grep -r ‘403’ .

and the results I got are these snippets:

    # Deny bad Referers
    if ($http_referer ~*

(babes|forsale|girl|jewelry|love|nudit|organic|poker|porn|sex|teen)) {
return 403;
}

…

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    location ~ /\. { access_log off; log_not_found off; deny all; }

    # Wordpress uses the robots.txt
    location = /robots.txt  { access_log off; log_not_found off; }
    location = /favicon.ico { access_log off; log_not_found off; }
    location ~ ~$           { access_log off; log_not_found off;

deny
all; }

…

The apps are several, they all follow the “index.php is the controller”
paradygm.

    # Make sure files with the following extensions do not get

loaded by
nginx because nginx would display the source code, and these files can
contain PASSWORDS!
location ~*
.(engine|inc|ini|info|install|make|module|profile|test|po|sh|.sql|theme|tpl(.php)?|xtmpl)$|^(..|Entries.*|Repository|Root|Tag|Template)$|.php_
{
deny all;
}

    location ~ /config.php {
            deny all;
    }

It’s not the UFW firewall as well, because the error shows up even with
UFW
disabled. So the potential culprit may be php-fpm or some weird nginx
option. Here’s the master conf file, the others don’t specify anything
but
location

Posted at Nginx Forum:

6

Thank you so much. I am pretty sure the 403 comes from another source as
well (as it does not happen under light http server load) but the ‘love’
regex was really a good show of having a nice hawk eye from yours :smiley:

Posted at Nginx Forum:

7

For future reference:
empyrical observation seems to show that 1.6 is a bit heavier than
1.4.7.
I have raised some php-fpm.conf daemon settings and timeouts and the
situation improved:

emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 10s

Posted at Nginx Forum: