Nginx slowing down

gennix · September 23, 2008, 9:53pm

Hi everybody,

I have a problem…

When Nginx is reaching more than 120 requests/seconds, http connections
start to slow down (from 2 secondes to get all the home page to 20
secondes), and I don’t understand why.
I though it was PHP or MySQL but even a HTTP GET on a simple image is
tragically slowing down.

Any idea?

I use with a Pentium 4 with 1GB RAM:
. Linux 2.6.18
. Nginx 0.6.32
. PHP 5.2.6
. PHP-FPM 0.5.8
. XCache 1.2.2
. MySQL 5.0.45

The nginx.conf:

user nginx nginx;
worker_processes 1;

events
{
worker_connections 2048;
}

http
{
include mime.types;
default_type application/octet-stream;

access_log off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 60;

gzip on;

gzip_comp_level 2;
gzip_types text/plain text/html text/css text/javascript;

server
{
listen 80;
server_name www.domain.com domain.com www.domain.fr domain.fr;

     root         /home/www;

     location /
        {
           index  index.php;
           rewrite ^/?$            /index.php            last;
           rewrite ^/?index\.php$  /index.php            last;
           rewrite ^/?(.*)$        /search.php?query=$1  last;
        }

     # redirect server error pages to the static page /50x.html
     #
     error_page   500 502 503 504  /50x.html;
     location = /50x.html
        {
           root   /usr/local/nginx/html;
        }

     # PHP configuration
     location ~ .*\.php$
     {
        fastcgi_pass   127.0.0.1:9000;
        fastcgi_index  index.php;
        fastcgi_param  SCRIPT_FILENAME

/home/www/$fastcgi_script_name;
include fastcgi_params;
}

     # Static files configuration
     location ~* ^.+.(jpg|jpeg|gif|css|png|js|ico)$
     {
        access_log        off;
        expires           30d;
     }


     location /nginx_status
     {
        stub_status  on;
        access_log   off;
        allow        my.ip.addr.ess;
        deny         all;
     }

}
}

gennix · September 23, 2008, 11:39pm

your keep alives are way to high for 120req/sec.

Lower it down to something more reasonable such as 30-15 seconds.
Possibly your system is choking with many concurrent connections open.

Check that your not swapping, vmstat -p 1 … (on solaris, not sure of
the equivalent on linux) Make sure that you don’t see anonymous paging.

Third, do you have enough ram for system file cache? Disks are very
slow, and if your system does not have enough ram to cache frequent
files into ram, things will be slow.

In my production environment, nginx with solaris’s zfs ARC cache trumped
varnish. However, it might be in your benefit where you want a narrow
memory cache for static files (such as varnish’s mmaped cache file).

Good luck

fyi, we easily see 500-1000req/sec no problem here.

gennix · September 23, 2008, 11:40pm

oh yes based on looking at your config, up the worker count to 4. You
might be hitting a maxfile limit.

gennix · September 24, 2008, 1:01am

Victor I. wrote:

oh yes based on looking at your config, up the worker count to 4. You
might be hitting a maxfile limit.

It seems to work (i.e “worker_processes 4;”).
Many thanks, Victor!!

PS: what do you mean by hitting a maxfile limit?
I looked at the file descriptor usage, but it seems ok :

cat /proc/sys/fs/file-nr
896 0 99199

which means that since the kernel has been running the max number of
fd’s in use was 896, the current number is 0, and the max available is
99199.

gennix · September 23, 2008, 11:47pm

This picture
http://img34.picoodle.com/data/img34/3/9/23/f_slowdownm_e851af5.png
illustrates my problem.

The same jpeg image took 0.4 seconde to load versus 2.5 secondes when
the web trafic is high (i.e. > 100 requests/secondes) on Nginx.

I tried to find the source of this problem in the vmstat, but I found
nothing:

vmstat 1
procs -----memory-------- -----io---- --system-- -----cpu—
r b free buff cache bi bo in cs us sy id wa
0 1 29160 90508 156600 227 249 10 29 29 3 42 26
2 1 27808 90700 157100 592 4 1652 797 34 3 31 32
3 0 17648 90868 157532 628 0 1589 596 42 4 32 22
0 1 27276 91000 158052 616 996 1640 533 13 2 51 34
0 1 26916 91112 158440 536 0 1731 652 12 3 56 30
0 1 30556 89932 155080 404 0 1591 679 21 3 57 18
5 2 23276 90008 155456 392 1424 1742 608 31 4 33 32
0 2 29940 90108 155760 460 604 1806 759 31 5 26 39
0 3 30232 90200 156152 452 896 1765 433 4 2 45 49
0 2 29624 90276 156408 412 1244 1631 451 18 2 37 43
0 2 28704 90344 156840 340 720 1731 526 18 4 31 48
0 2 28656 90476 157256 596 492 1921 1078 14 3 32 52
1 2 28232 90588 157544 384 904 1621 469 26 4 32 38
6 1 27560 90696 157760 420 1428 1745 690 31 4 26 39
0 1 27632 90792 158164 384 832 1715 401 19 2 39 40
4 1 14440 90944 158464 572 0 1611 770 55 7 12 27
0 1 32004 89648 154496 704 0 1793 848 54 5 15 26
0 1 31036 89840 154996 704 0 1841 987 76 6 8 11
1 1 22476 90068 155396 696 940 1751 669 41 4 24 32
1 1 22820 89668 153400 624 0 1867 814 64 6 9 22
0 1 31696 89804 153928 668 0 1930 592 28 4 39 29
0 1 31832 89980 154320 588 0 1856 976 41 5 26 29
3 1 30732 90244 154756 748 0 1811 696 34 5 25 37
0 1 30248 90384 155172 488 1212 1755 662 40 5 27 28
4 1 28776 90664 155692 780 0 1812 699 43 4 24 29
1 1 30432 90596 155076 732 0 1904 1136 70 7 10 13
0 1 29520 90752 155504 600 0 1764 781 45 3 25 27
0 1 29164 90968 155968 640 0 1731 526 9 3 53 35
0 1 27816 91168 156468 704 1108 1700 769 31 4 35 30
1 1 27276 91364 156956 720 0 1850 731 32 4 32 32
0 1 26360 91464 157456 508 0 1738 494 11 3 60 27
0 0 26744 91568 157888 536 0 1695 1006 53 4 21 23
1 1 19576 91768 158488 868 0 2027 534 11 3 43 43
2 1 25668 91864 158900 560 1124 1855 977 40 5 27 28
1 1 15332 91988 159576 668 0 1952 901 51 6 19 25
6 1 17576 92064 159912 504 0 1830 712 61 4 17 19
3 1 28848 90368 153092 464 0 1716 646 71 6 12 12
0 1 31828 90604 153644 788 0 1854 574 25 4 34 38
0 1 31160 90728 154320 716 1008 2046 795 48 4 17 31
2 1 19896 91012 154940 892 0 1898 762 21 4 36 40
0 1 28908 91248 155456 816 0 2018 805 47 4 21 28
1 0 28420 91400 156040 736 0 1974 823 33 5 27 34
1 1 27568 91616 156556 720 0 1923 833 22 3 37 39
2 1 23432 91804 157100 736 1320 1820 529 13 4 42 41
5 1 24824 90748 153128 580 0 1696 957 80 7 7 7
1 1 31736 90900 153452 572 0 1746 780 50 5 25 21
0 1 31216 91100 153976 688 0 1732 663 16 2 45 36
1 1 20752 91340 154400 700 0 1863 973 59 7 13 22
6 1 28820 91444 154796 384 1036 1773 823 75 5 7 12
0 2 29380 91496 154968 268 1976 1748 478 40 3 28 29
1 2 28772 91612 155348 504 660 1831 535 9 2 42 46
0 2 28972 91796 155680 528 664 1717 596 20 3 17 60
0 2 31292 91048 152988 452 548 1647 571 36 5 26 33
1 2 31132 91144 153180 364 824 1588 689 49 5 21 26
0 1 31012 91236 153524 404 1300 1660 451 15 2 42 41
0 1 30408 91468 154044 764 0 1700 558 22 2 37 38
0 1 29560 91672 154660 808 0 1859 939 49 5 18 28
0 1 29600 91676 154656 780 0 1805 755 44 5 21 29
0 1 29340 91928 155180 808 1068 1844 648 10 3 40 48
1 1 28428 92148 155816 812 0 1894 832 28 5 24 44
7 1 11204 91772 154416 532 16 1741 823 61 5 16 18
0 1 32020 91608 153156 860 0 1909 659 22 3 34 41
3 1 26860 91780 153676 632 0 1813 791 48 4 16 33
0 1 30600 92044 154244 824 1244 1789 565 16 3 39 42
0 1 29320 92228 154900 820 0 1806 547 23 4 36 38
2 1 36628 87552 152040 508 0 1776 964 90 8 0 1
0 1 36368 87748 152484 700 0 1877 1376 46 5 21 28
0 1 35500 87976 153072 740 0 1805 555 32 3 34 31
0 1 35324 88032 153524 452 1116 1612 651 32 5 37 27
0 1 34720 88164 153980 620 0 1761 484 9 2 55 35
1 0 34240 88232 154404 544 0 1670 643 17 3 50 30
3 1 32948 88276 154808 392 0 1533 927 26 3 50 21
0 1 32840 88428 155228 572 0 1559 564 18 3 49 31
2 1 32168 88552 155564 440 740 1529 803 45 4 33 18
7 0 17492 88780 156092 740 0 1837 901 37 6 25 32
2 1 43828 86392 145784 504 0 1714 1059 74 7 12 8
1 1 41152 86572 146216 688 0 1927 768 41 4 24 31
0 1 42752 86760 146652 608 0 1730 727 45 4 25 25
3 1 30180 86988 147156 772 1184 1824 740 27 4 30 40
0 1 41912 87016 147628 440 0 1720 752 56 5 26 13
1 1 35168 87020 148024 400 0 1590 594 37 3 39 21
1 0 41260 87200 148584 644 0 1681 619 16 3 47 34
7 0 30300 87304 148856 500 0 1575 1135 52 6 23 19
0 1 39492 87436 149372 632 824 1671 713 53 6 21 21
0 1 39016 87640 149964 808 0 1810 757 18 3 28 51
0 2 38656 87800 150348 468 1832 1812 320 0 1 50 49
0 2 37492 87872 150756 528 628 1664 464 4 2 41 53
0 3 37684 87932 151012 228 960 1585 518 27 3 23 46
5 3 23996 88004 151416 528 500 1731 881 42 6 22 31
0 1 36468 88128 151728 444 1640 1664 484 16 3 30 51
7 1 34932 88252 152060 488 4 1545 515 52 3 20 25
11 1 34576 88376 152608 608 0 1724 746 69 3 10 17
3 1 17540 88564 153032 604 0 1825 837 67 8 2 24
0 1 33984 88792 153456 636 1008 1721 701 48 3 20 29
0 1 33316 88968 153836 572 0 1586 643 50 5 21 25
3 1 33016 89148 154180 560 0 1664 623 47 4 21 28
1 1 32604 89484 154828 976 0 1993 1010 29 6 24 43
3 1 31812 89644 155300 596 0 1773 598 14 1 52 33
1 0 31144 89724 155660 444 1136 1645 705 46 4 32 19
0 1 31032 89848 156056 496 0 1638 673 17 2 55 26

with:
swap.si=0
swap.so=0
memory.swpd=196960

top
top - 16:41:10 up 7 days, 1:22, 1 user, load average: 3.27, 2.88, 2.52
Tasks: 91 total, 1 running, 90 sleeping, 0 stopped, 0 zombie
Cpu: 37.2%us, 3.0%sy, 0.0%ni, 28.0%id, 30.8%wa, 0.2%hi, 0.8%si, 0.0%st
Mem: 1002732k total, 967336k used, 35396k free, 74648k buffers
Swap: 2024180k total, 196960k used, 1827220k free, 136376k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31031 apache 23 0 28120 13m 2608 S 6 1.4 0:10.99 php-cgi
31022 apache 15 0 18128 4032 2608 S 6 0.4 0:12.88 php-cgi
31029 apache 16 0 28120 13m 2608 S 6 1.4 0:13.38 php-cgi
11344 mysql 15 0 726m 431m 2904 S 6 44.1 568:50.21 mysqld
31045 apache 15 0 28120 13m 2608 S 6 1.4 0:08.48 php-cgi
31037 apache 15 0 28120 13m 2604 S 5 1.4 0:10.48 php-cgi
31027 apache 15 0 28120 13m 2604 S 5 1.4 0:11.72 php-cgi
31038 apache 16 0 18128 4032 2608 S 4 0.4 0:11.79 php-cgi
31025 apache 15 0 28120 13m 2632 S 3 1.4 0:13.96 php-cgi
31048 apache 16 0 18128 4032 2608 S 3 0.4 0:09.63 php-cgi
31049 apache 15 0 28120 13m 2608 S 3 1.4 0:08.05 php-cgi
31050 apache 17 0 28120 13m 2608 S 3 1.4 0:07.15 php-cgi
30167 apache 16 0 3900 2432 736 D 3 0.2 3:01.59 nginx
31020 apache 25 0 28120 13m 2608 S 3 1.4 0:15.37 php-cgi
31026 apache 16 0 28120 13m 2608 S 3 1.4 0:11.85 php-cgi
31034 apache 15 0 28120 13m 2604 S 3 1.4 0:11.83 php-cgi
31042 apache 18 0 28120 13m 2604 S 3 1.4 0:09.84 php-cgi
31017 apache 18 0 28120 13m 2608 S 3 1.4 0:16.80 php-cgi
31024 apache 15 0 28120 13m 2608 S 3 1.4 0:14.22 php-cgi
31040 apache 20 0 28120 13m 2640 S 3 1.4 0:10.87 php-cgi
11182 nobody 15 0 10848 3436 680 S 2 0.3 84:58.71 memcached
31033 apache 18 0 28120 13m 2608 S 1 1.4 0:11.93 php-cgi
1 root 15 0 2064 556 528 S 0 0.1 0:00.70 init
2 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
3 root 39 19 0 0 0 S 0 0.0 0:00.07 ksoftirqd/0
4 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0

gennix · September 24, 2008, 3:18am

max open file limit per process. Defaults to 255, which extremely
paltry.

These are the settings I use, minus the event port stuff, seems to be
buggy in nginx, Igor claims it is a solaris bug.

http://letsgetdugg.com/view/Nginx_hidden_performance_option

gennix · September 23, 2008, 11:48pm

Victor I. wrote:

oh yes based on looking at your config, up the worker count to 4. You
might be hitting a maxfile limit.

Thank you for your response Victor.
I will try your solution right now.