Client closed connection before nginx was able to accept() it.
This may be normal (i.e. user just closed a browser page while images
was still loading) and may be not (i.e. nginx wasn’t able to
accept() for a long time due to some problems and client bored
waiting and closed a page).
Some number of such messages are expected to appear. High number
of such errors may indicate problems, try looking at listen queues
at first (netstat -Lan).
2011/07/13 10:26:56 [error] 38059#0: *73523 writev() failed (54:
Connection reset by peer) while sending request to upstream …
Your upstream reset connection. This is a problem with your
backend (most likely it just died). Inspect backend logs.
2011/07/13 10:31:33 [error] 38060#0: *113640 writev() failed (32: Broken
pipe) while sending request to upstream …
The same as above, but in slightly different moment.
What should I tuning? What should I set up?
From here it looks like you have problems with backend(s).
It has 100,000,000 request per day and this number is increasing.
When this error occurs, the 50x page is shown OR totally disconnects
CPU: Intel(R) Xeon(R) CPU @ 2.40GHz x 6 ( x 2 ) = 24 CPU
This may be normal (i.e. user just closed a browser page while images
Your upstream reset connection. This is a problem with your
backend (most likely it just died). Inspect backend logs.
2011/07/13 10:31:33 [error] 38060#0: *113640 writev() failed (32: Broken pipe)
while sending request to upstream …
The same as above, but in slightly different moment.
What should I tuning? What should I set up?
From here it looks like you have problems with backend(s).
netstat -Lan shows:
Current listen queue sizes (qlen/incqlen/maxqlen)
Proto Listen Local Address
tcp4 0/0/4096 *.80
tcp4 27/0/128 127.0.0.1.9002
tcp4 32/0/128 127.0.0.1.9001
tcp4 26/0/128 127.0.0.1.9000
tcp4 0/0/128 *.2818
tcp6 0/0/128 *.2818
tcp4 0/0/10 127.0.0.1.25
Some tcp sockets may have been created.
unix 0/0/4 /var/run/devd.pipe
but sometimes the qlen over 128 for the 3 php backend. It should be the
problem I guess.. hmmm.
Client closed connection before nginx was able to accept() it.
This may be normal (i.e. user just closed a browser page while images
was still loading) and may be not (i.e. nginx wasn’t able to
accept() for a long time due to some problems and client bored
waiting and closed a page).
Some number of such messages are expected to appear. High number
of such errors may indicate problems, try looking at listen queues
at first (netstat -Lan).
On Wed, Jul 13, 2011 at 01:41:59PM +0200, Adam PAPAI wrote:
Some number of such messages are expected to appear. High number
tcp4 181/0/128 127.0.0.1.9000
It starts to throw the writev() failed (54: Connection reset by
peer) while sending request errors.
What should I increase? Any ideas? What is the real meaning of the
maxqlen? If it the qlen is greater than maxqlen it means…?
Maxqlen is maximum listen socket queue length (the queue of
connections completed handshake but not yet accept()'ed by
application), and FreeBSD will reset new connections if it’s
exhausted (or will just drop them if you have
net.inet.tcp.syncache.rst_on_sock_fail=0).
If you see qlen greater than maxqlen it means that your app can’t
cope with load. If this happens for a fraction of a second due to
connection bursts - this may be ok and you just need to increase
queue length to compensate bursts.
But if you’ve been able to see it in netstat output - it certainly
means there is something wrong with backends. You either have to
add more backends or find out/optimize bottlenecks in existing ones.
On Wed, Jul 13, 2011 at 04:10:52PM +0200, Adam PAPAI wrote:
If you see qlen greater than maxqlen it means that your app can’t
cope with load. If this happens for a fraction of a second due to
connection bursts - this may be ok and you just need to increase
queue length to compensate bursts.
The question is now: how to increase the queue length under FreeBSD.
I cannot find the config value in sysctl.
System limit is kern.ipc.somaxconn. But as far as I remember you
have 4096 queue for nginx, so it looks like you’ve already tuned
it.
The question is how to tune it in your backend, but there is no
simple answer: this depends on backend. E.g. php-fpm should use
kern.ipc.somaxconn as seen on process startup by default and have
an option (“backlog”) to tune it.
But again: as long as you’ve been able to see overflows, just
increasing listen queue isn’t likely to help. You have to add
more backends.
Maxqlen is maximum listen socket queue length (the queue of
connections completed handshake but not yet accept()'ed by
application), and FreeBSD will reset new connections if it’s
exhausted (or will just drop them if you have
net.inet.tcp.syncache.rst_on_sock_fail=0).
If you see qlen greater than maxqlen it means that your app can’t
cope with load. If this happens for a fraction of a second due to
connection bursts - this may be ok and you just need to increase
queue length to compensate bursts.
The question is now: how to increase the queue length under FreeBSD. I
cannot find the config value in sysctl.
