NGINX 500 http error

Server NGINX
1

Hi,

I have a lot of 500 http error in nginx access.log.
I think it’s due to the number of requests that nginx receives. Because
when
I was testing the configuration with one ip adress I don’t have this
error
but when I redirect all client traffic (sometimes more than 2000
simultaneous connexions) to nginx I’m getting 500 http error.
I don’t have any error log in error.log.

I tried to modify this parameters : worker_connections,
worker_rlimit_nofile, keepalive_timeout, proxy_connect_timeout …

And then I start to get this logs in error.log :
worker process XXXXX exited on signal 11
an upstream response is buffered to a temporary file
/var/lib/nginx/proxy/2/00/0000000002
failed (104: Connection reset by peer) while sending response to
client

Here is my nginx.log conf :

user www-data;
worker_processes 4;
worker_rlimit_nofile 50000;
pid /var/run/nginx.pid;

events {
worker_connections 50000;
}

http {

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 20;

    types_hash_max_size 2048;
    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    gzip on;
    gzip_disable "msie6";

proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size 10m;
client_body_buffer_size 50M;
proxy_connect_timeout 600;
proxy_send_timeout 600;
proxy_read_timeout 600;
proxy_buffer_size 128k;
proxy_buffers 4 254k;
proxy_busy_buffers_size 256k;

    include /etc/nginx/sites-enabled/*;

}

When I execute top command, cpu and RAM usage are not hignt (max cpu 7%
and
ram 4%).

Any help would be appreciated.
Thanks.

Posted at Nginx Forum:

2

Hello!

On Fri, Nov 15, 2013 at 05:51:16AM -0500, fatine,al wrote:

Hi,

I have a lot of 500 http error in nginx access.log.
I think it’s due to the number of requests that nginx receives. Because when
I was testing the configuration with one ip adress I don’t have this error
but when I redirect all client traffic (sometimes more than 2000
simultaneous connexions) to nginx I’m getting 500 http error.
I don’t have any error log in error.log.

The 500 errors usually indicate either configuration problem (e.g,
an infinite rewrite loop or something like this) or some resource
shortage (memory, file descriptors, space for temporary files, and
so on).

I tried to modify this parameters : worker_connections,
worker_rlimit_nofile, keepalive_timeout, proxy_connect_timeout …

And then I start to get this logs in error.log :
worker process XXXXX exited on signal 11

Here nginx exits due to memory access violation. This is a
serious problem and usually indicate a bug somewhere (well, it
might be also hardware problem).

First of all, it might be a good idea to show “nginx -V” output
(and basically make sure you are using latest version and no 3rd
party modules), and obtain core dump and show a backtrace.

See here for basic instructions and more details:

  an upstream response is buffered to a temporary file

/var/lib/nginx/proxy/2/00/0000000002
failed (104: Connection reset by peer) while sending response to
client

These are not problems per se.

Here is my nginx.log conf :

user www-data;
worker_processes 4;
worker_rlimit_nofile 50000;
pid /var/run/nginx.pid;

[…]

    include /etc/nginx/sites-enabled/*;

Note that configuration you’ve provided isn’t nearly a full one.
It lacks an unsepcified number of included files.

–
Maxim D.
http://nginx.org/en/donation.html

3

Hi,

Thank you for your quick response.

Here is the output of nginx -V :

nginx version: nginx/1.2.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-client-body-temp-path=/var/lib/nginx/body
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi
–http-log-path=/var/log/nginx/access.log
–http-proxy-temp-path=/var/lib/nginx/proxy
–http-scgi-temp-path=/var/lib/nginx/scgi
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi
–lock-path=/var/lock/nginx.lock
–pid-path=/var/run/nginx.pid --with-pcre-jit --with-http_ssl_module
–without-mail_pop3_module --without-mail_smtp_module
–without-mail_imap_module --without-http_uwsgi_module
–without-http_scgi_module --with-ipv6 --with-http_stub_status_module
–add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/nginx-upstream-fair
–add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/nginx-cache-purge
–add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/naxsi/naxsi_src

I added this two lines for core dump in nginx.conf :
worker_rlimit_core 500M;
working_directory /path/to/cores/;

But I think I have to compile nginx with backtrace module to dump
backtrace.

What other includes do I need in my nginx.conf ?

Thanks.

Posted at Nginx Forum:

4

Hello!

On Fri, Nov 15, 2013 at 07:00:02AM -0500, fatine,al wrote:

–http-client-body-temp-path=/var/lib/nginx/body

–add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/nginx-cache-purge

–add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/naxsi/naxsi_src

The 1.2.1 is rather old and not supported. Even in 1.2.x branch
there are at least several fixes of various segmentation faults.
It doesn’t really make any sense to try to debug anything further,
even traditional “please recompile without 3rd party modules to
see if it helps” doesn’t really apply here.

Upgrade to at least 1.4.3 (1.5.6 preferred) and come back if
you’ll see any problems there.

See here for recent versions of nginx available for download,
including precompiled linux packages:

http://nginx.org/en/download.html

–
Maxim D.
http://nginx.org/en/donation.html

5

OK.

I will upgrade nginx and tell you if the problem is solved.

Thank you.

Posted at Nginx Forum:

6

Note: --add-module=…/naxsi… should be the first one.

Posted at Nginx Forum:

7

Hi,

The problem is solved.
I installed nginx-1.5.6 and naxsi-core-0.50 from sources, and compile
nginx
with naxsi module and some options :

./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf --lock-path=/var/lock/nginx.lock
–pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log
–http-client-body-temp-path=/var/lib/nginx/body
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi
–http-proxy-temp-path=/var/lib/nginx/proxy
–http-scgi-temp-path=/var/lib/nginx/scgi
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-pcre-jit
–with-http_ssl_module --with-debug --with-ipv6
–with-http_stub_status_module
–add-module=…/gnosek-nginx-upstream-fair-a18b409/
–add-module=…/ngx_cache_purge-2.1
–add-module=…/naxsi-core-0.50/naxsi_src/

It works fine now.

Thank you for your help.

Posted at Nginx Forum:

8

OK.

:slight_smile:

Posted at Nginx Forum: