Forum: NGINX NGINX 500 http error

2974d09ac2541e892966b762aad84943?d=identicon&s=25 fatine,al (Guest)
on 2013-11-15 11:52
(Received via mailing list)
Hi,

I have a lot of 500 http error in nginx access.log.
I think it's due to the number of requests that nginx receives. Because
when
I was testing the configuration with one ip adress I don't have this
error
but when I redirect all client traffic (sometimes more than 2000
simultaneous connexions) to nginx I'm getting 500 http error.
I don't have any error log in error.log.

I tried to modify this parameters : worker_connections,
worker_rlimit_nofile, keepalive_timeout, proxy_connect_timeout ...

And then I start to get this logs in error.log :
      worker process XXXXX exited on signal 11
      an upstream response is buffered to a temporary file
/var/lib/nginx/proxy/2/00/0000000002
      failed (104: Connection reset by peer) while sending response to
client


Here is my nginx.log conf :

user www-data;
worker_processes 4;
worker_rlimit_nofile 50000;
pid /var/run/nginx.pid;

events {
        worker_connections 50000;
}

http {

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 20;

        types_hash_max_size 2048;
        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        gzip on;
        gzip_disable "msie6";

proxy_redirect          off;
proxy_set_header        Host            $host;
proxy_set_header        X-Real-IP       $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
client_max_body_size    10m;
client_body_buffer_size 50M;
proxy_connect_timeout   600;
proxy_send_timeout      600;
proxy_read_timeout      600;
proxy_buffer_size   128k;
proxy_buffers       4   254k;
proxy_busy_buffers_size 256k;

        include /etc/nginx/sites-enabled/*;
}

When I execute top command, cpu and RAM usage are not hignt (max cpu 7%
and
ram 4%).

Any help would be appreciated.
Thanks.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244693,244693#msg-244693
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2013-11-15 12:02
(Received via mailing list)
Hello!

On Fri, Nov 15, 2013 at 05:51:16AM -0500, fatine,al wrote:

> Hi,
>
> I have a lot of 500 http error in nginx access.log.
> I think it's due to the number of requests that nginx receives. Because when
> I was testing the configuration with one ip adress I don't have this error
> but when I redirect all client traffic (sometimes more than 2000
> simultaneous connexions) to nginx I'm getting 500 http error.
> I don't have any error log in error.log.

The 500 errors usually indicate either configuration problem (e.g,
an infinite rewrite loop or something like this) or some resource
shortage (memory, file descriptors, space for temporary files, and
so on).

> I tried to modify this parameters : worker_connections,
> worker_rlimit_nofile, keepalive_timeout, proxy_connect_timeout ...
>
> And then I start to get this logs in error.log :
>       worker process XXXXX exited on signal 11

Here nginx exits due to memory access violation.  This is a
serious problem and usually indicate a bug somewhere (well, it
might be also hardware problem).

First of all, it might be a good idea to show "nginx -V" output
(and basically make sure you are using latest version and no 3rd
party modules), and obtain core dump and show a backtrace.

See here for basic instructions and more details:

http://wiki.nginx.org/Debugging

>       an upstream response is buffered to a temporary file
> /var/lib/nginx/proxy/2/00/0000000002
>       failed (104: Connection reset by peer) while sending response to
> client

These are not problems per se.

> Here is my nginx.log conf :
>
> user www-data;
> worker_processes 4;
> worker_rlimit_nofile 50000;
> pid /var/run/nginx.pid;

[...]

>         include /etc/nginx/sites-enabled/*;

Note that configuration you've provided isn't nearly a full one.
It lacks an unsepcified number of included files.

--
Maxim Dounin
http://nginx.org/en/donation.html
2974d09ac2541e892966b762aad84943?d=identicon&s=25 fatine,al (Guest)
on 2013-11-15 13:00
(Received via mailing list)
Hi,

Thank you for your quick response.

Here is the output of nginx -V :

nginx version: nginx/1.2.1
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx
--conf-path=/etc/nginx/nginx.conf
--error-log-path=/var/log/nginx/error.log
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-log-path=/var/log/nginx/access.log
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi
--lock-path=/var/lock/nginx.lock
--pid-path=/var/run/nginx.pid --with-pcre-jit --with-http_ssl_module
--without-mail_pop3_module --without-mail_smtp_module
--without-mail_imap_module --without-http_uwsgi_module
--without-http_scgi_module --with-ipv6 --with-http_stub_status_module
--add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/nginx-upstream-fair
--add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/nginx-cache-purge
--add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/naxsi/naxsi_src


I added this two lines for core dump in nginx.conf :
worker_rlimit_core  500M;
working_directory   /path/to/cores/;

But I think I have to compile nginx with backtrace module to dump
backtrace.

What other includes do I need in my nginx.conf ?

Thanks.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244693,244700#msg-244700
A8108a0961c6087c43cda32c8616dcba?d=identicon&s=25 Maxim Dounin (Guest)
on 2013-11-15 14:22
(Received via mailing list)
Hello!

On Fri, Nov 15, 2013 at 07:00:02AM -0500, fatine,al wrote:

> --http-client-body-temp-path=/var/lib/nginx/body
>
--add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/nginx-cache-purge
>
--add-module=/build/buildd-nginx_1.2.1-2.2~bpo60+2-i386-EyCkdD/nginx-1.2.1/debian/modules/naxsi/naxsi_src

The 1.2.1 is rather old and not supported.  Even in 1.2.x branch
there are at least several fixes of various segmentation faults.
It doesn't really make any sense to try to debug anything further,
even traditional "please recompile without 3rd party modules to
see if it helps" doesn't really apply here.

Upgrade to at least 1.4.3 (1.5.6 preferred) and come back if
you'll see any problems there.

See here for recent versions of nginx available for download,
including precompiled linux packages:

http://nginx.org/en/download.html

--
Maxim Dounin
http://nginx.org/en/donation.html
2974d09ac2541e892966b762aad84943?d=identicon&s=25 fatine,al (Guest)
on 2013-11-15 15:17
(Received via mailing list)
OK.

I will upgrade nginx and tell you if the problem is solved.

Thank you.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244693,244709#msg-244709
2974d09ac2541e892966b762aad84943?d=identicon&s=25 fatine,al (Guest)
on 2013-11-29 05:24
(Received via mailing list)
Hi,

The problem is solved.
I installed nginx-1.5.6 and naxsi-core-0.50 from sources, and compile
nginx
with naxsi module and some options :

./configure --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx
--conf-path=/etc/nginx/nginx.conf  --lock-path=/var/lock/nginx.lock
--pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log
--http-log-path=/var/log/nginx/access.log
--http-client-body-temp-path=/var/lib/nginx/body
--http-fastcgi-temp-path=/var/lib/nginx/fastcgi
--http-proxy-temp-path=/var/lib/nginx/proxy
--http-scgi-temp-path=/var/lib/nginx/scgi
--http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-pcre-jit
--with-http_ssl_module --with-debug --with-ipv6
--with-http_stub_status_module
--add-module=../gnosek-nginx-upstream-fair-a18b409/
--add-module=../ngx_cache_purge-2.1
--add-module=../naxsi-core-0.50/naxsi_src/

It works fine now.

Thank you for your help.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244693,245052#msg-245052
2974d09ac2541e892966b762aad84943?d=identicon&s=25 itpp2012 (Guest)
on 2013-11-29 05:49
(Received via mailing list)
Note: --add-module=../naxsi... should be the first one.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244693,245053#msg-245053
2974d09ac2541e892966b762aad84943?d=identicon&s=25 fatine,al (Guest)
on 2013-12-03 10:21
(Received via mailing list)
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.