Nginx Active & Writing Connections exponential growth

Server NGINX
1

Hi there!
First of all, I am a new user of the nginx mailing list so there is a
short
introduction of myself. My name is Antonio, and I have been working for
a
long time as Web Operations Engineer at Uptodown.com, VisualizeUs.com
and
other minor projects.

We are running nginx in Uptodown.com for static content, without
problems,
and in VisualizeUs.com, since 2 years ago, for static and dynamic
content
without problems . But it is in visualizeUs where we are having some
problems, since we updated nginx to nginx/0.7.67, under debian Lenny 32
bits
(aws medium instance), with php-fcgi

There is the munin output for nginx connections.

We noticed, an exponential growth for connections, better apreciated
through
the graphic.

There is our nginx configuration:

http://dpaste.com/hold/223096/

And our ulimit for www-data:

maxfds …fs.file-max = 200000
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 20
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 20480
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
real-time priority (-r) 1
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited

As active connections grow, response time sligthly increases, so a nginx
reboot is needed to get back to optimal response time.Nginx reboots are
apreciated in munin graphic when it returns to normal values, in other
case,
exponential grows continues.

We have been tweaking system limits without noticiable diferences. Also
we
have been working around php-cgi configuration, finding out that the
exponential growth of connections still exists but in a slower rate.

Anything that make us point in the right direction will be appreciated.

Thanks in advanced.

Best Regards.
Antonio.

2

Hello!

On Thu, Jul 29, 2010 at 02:10:23PM +0200, Antonio Jerez Guillén wrote:

(aws medium instance), with php-fcgi

There is the munin output for nginx connections.
http://i29.tinypic.com/21y781.png

  1. Are you sure you are measuring connections here? Numbers looks
    odd, one can’t have 280.57 active connections (which is shown on
    the first graph as “max”), it should be integer.

  2. Are these connections correspond to real client and/or backend
    connections (i.e. compare with netstat data)? Most likely no, but
    numbers are suspiciously low.

  3. What does nginx -V show? Do you have any third party modules /
    patches? If yes - are you able to reproduce the problem without
    them?

  4. Are you able to reproduce the problem with 0.8.*?

We noticed, an exponential growth for connections, better apreciated through
the graphic.

Looks like linear one, not exponential, but anyway it looks like
socket leak.

I don’t remember any socket leaks relevant to 0.7.* branch which
aren’t fixed in 0.7.67. But things changed a lot in 0.8.* branch
and this particular problem may not even appear with new code.

There is our nginx configuration:

http://dpaste.com/hold/223096/

This doesn’t contain full nginx config so it’s mostly useless.

[…]

As active connections grow, response time sligthly increases, so a nginx
reboot is needed to get back to optimal response time.Nginx reboots are
apreciated in munin graphic when it returns to normal values, in other case,
exponential grows continues.

Could you please clarify what do you mean by “slightly increases”?
With epoll extra 1k idle connections shouldn’t be noticeable.

Maxim D.

3

Hi, Antonio.

I’ve met a very similar situation which you did about 2 years ago.
Did you solve this situation?
If so, how did you do that?

Thanks.