Nginx Slowdown on Solaris

Hello,

I use nginx 0.8.53 on Solaris 10 and I feel like I reach a kind of limit
but I’m not abble to find it.
I use Nginx to serve static image and sometimes to get one image I need
to wait 10 or more seconds.

nginx is launched with a ulimit -n 65536, and here is my config:

worker_processes 40;
error_log logs/error.log;
events {
worker_connections 8192;
devpoll_events 1;
}
http {
include mime.types;
default_type application/octet-stream;

sendfile        on;
keepalive_timeout  15;

gzip  on;
gzip_buffers 16 8k;
gzip_types text/plain text/html text/css text/javascript

application/x-javascript;
server {
listen 80;

}

I’ve got stub_status activated and here is a graph with the stats I
have:

http://img546.imageshack.us/img546/5369/nginx.png

I modify log level in order to check if I’ve got some warning about
wrong configuration but nothing.

Do you have any idea to solve this problem ?

Thanks for your help,

Julien

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,194233,194233#msg-194233

Hi,

On Thu, Apr 28, 2011 at 08:12:59AM -0400, darckos wrote:

Hello,

I use nginx 0.8.53 on Solaris 10 and I feel like I reach a kind of limit
but I’m not abble to find it.
I use Nginx to serve static image and sometimes to get one image I need
to wait 10 or more seconds.

nginx is launched with a ulimit -n 65536, and here is my config:

worker_processes 40;

[skip config]

why are you using such a large number of workers?

Could you show output for the following command:

$ ps ax -o %cpu,vsz,wchan,command | grep “nginx|PID”


Sergey A. Osokin
[email protected]

Hi,

At first, the configuration was 24 (number of cpu core), but I tryed to
increase it and I feel like it was better but it’s maybe a mistake.

Here is the output of ps -fe -o pcpu,vsz,wchan,comm | grep “nginx” (ps
-x is not available under Solaris)

0.0 5856 ffffffffabe334d0 /opt/nginx/sbin/nginx
0.0 5880 fffffe84dcf17638 /opt/nginx/sbin/nginx
0.0 6168 fffffe84e1c637c8 /opt/nginx/sbin/nginx
0.0 5880 fffffe84e1c68ca0 /opt/nginx/sbin/nginx
0.1 6032 ffffffffab58aae8 /opt/nginx/sbin/nginx
0.0 5984 ffffffffab5a74b0 /opt/nginx/sbin/nginx
0.0 5920 fffffe84d34bc660 /opt/nginx/sbin/nginx
0.0 5872 ffffffffaa576c88 /opt/nginx/sbin/nginx
0.1 5880 ffffffff9e03ef98 /opt/nginx/sbin/nginx
0.0 6008 fffffe84d61f74c8 /opt/nginx/sbin/nginx
0.0 5936 ffffffffaa13b4b0 /opt/nginx/sbin/nginx
0.0 6208 ffffffffab3e3e08 /opt/nginx/sbin/nginx
0.0 5888 ffffffffa3e3f330 /opt/nginx/sbin/nginx
0.0 5920 ffffffffa614c7e8 /opt/nginx/sbin/nginx
0.1 5904 ffffffffabc04fc0 /opt/nginx/sbin/nginx
0.0 5832 ffffffffaa080f98 /opt/nginx/sbin/nginx
0.1 5992 fffffe84e06dbca0 /opt/nginx/sbin/nginx
0.0 5888 ffffffffab7c1c98 /opt/nginx/sbin/nginx
0.0 3120 ffffffffa82ec5d6 /opt/nginx/sbin/nginx
0.0 6128 ffffffffab4ecaf8 /opt/nginx/sbin/nginx
0.0 5952 ffffffffabf64e10 /opt/nginx/sbin/nginx
0.0 5968 ffffffffab975c98 /opt/nginx/sbin/nginx
0.1 5992 ffffffffabbcf7e8 /opt/nginx/sbin/nginx
0.0 5968 ffffffffa9b01c98 /opt/nginx/sbin/nginx
0.1 6256 fffffe84e06e54c0 /opt/nginx/sbin/nginx
0.0 5888 fffffe84d28eae20 /opt/nginx/sbin/nginx
0.0 5744 fffffe84d5a8e960 /opt/nginx/sbin/nginx
0.0 5936 fffffe84e1c63c78 /opt/nginx/sbin/nginx
0.1 6184 ffffffffa9fb94b0 /opt/nginx/sbin/nginx
0.0 5880 fffffe84d28dcc88 /opt/nginx/sbin/nginx
0.1 5968 ffffffffab5404d0 /opt/nginx/sbin/nginx
0.0 5744 ffffffffa8e35f98 /opt/nginx/sbin/nginx
0.1 5976 ffffffffa9fa67d8 /opt/nginx/sbin/nginx
0.0 6144 fffffe84d5a50c98 /opt/nginx/sbin/nginx
0.0 6200 ffffffffa7839970 /opt/nginx/sbin/nginx
0.1 5984 fffffe84de37d188 /opt/nginx/sbin/nginx
0.1 5992 ffffffffab5a7190 /opt/nginx/sbin/nginx
0.1 6160 ffffffffab7edfc0 /opt/nginx/sbin/nginx
0.0 5984 fffffe84d5a55b00 /opt/nginx/sbin/nginx
0.0 5896 fffffe84d343f970 /opt/nginx/sbin/nginx
0.1 5856 fffffe84d28f8e30 /opt/nginx/sbin/nginx

I don’t know if you’ve what you need on this output. I’m not familliar
with format option of ps.

Thanks for your help

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,194233,194237#msg-194237

On Thu, Apr 28, 2011 at 08:48:18AM -0400, darckos wrote:

At first, the configuration was 24 (number of cpu core), but I tryed to
increase it and I feel like it was better but it’s maybe a mistake.

Here is the output of ps -fe -o pcpu,vsz,wchan,comm | grep “nginx” (ps
-x is not available under Solaris)

Oops, its Solaris.

I don’t know if you’ve what you need on this output. I’m not familliar
with format option of ps.

For Solaris the command should looks like:

% ps -ef -o pcpu,vsz,wchan,comm,args | grep nginx

(also -A or -a flags)

In output your should look at wchan parameter, the event (an address in
the system)
on which a process waits.


Sergey A. Osokin
[email protected]

Yes, like you said… it is Solaris :slight_smile:

ps -ef -o pcpu,vsz,wchan,comm,args | grep nginx
0.0 6032 ffffffffabc04660 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5880 ffffffffa9982188 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6168 ffffffffa81f9e20 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5880 ffffffffab3ba660 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6032 ffffffffa9fb97d0 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5984 ffffffffaa017338 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5920 ffffffffab7edb10 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5872 ffffffffa9bd9e30 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.1 5888 fffffe84e742b7f0 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6008 ffffffffab4ec968 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.1 5936 ffffffffa6422e10 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6208 fffffe84d5a61fa0 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5888 ffffffffab55d978 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5920 ffffffffab5a7320 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.1 5904 fffffe84d28fdae8 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5880 ffffffffab708188 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5992 ffffffffab975e28 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.1 5888 ffffffffa81f9fb0 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 3120 ffffffffa82ec5d6 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6128 ffffffffaa080318 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5952 fffffe84d077e640 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5968 ffffffffa9fa6fa8 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5992 fffffe84d2ec3650 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.1 5968 fffffe84d5a8b328 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.1 6256 fffffe84d5a4cca0 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5888 ffffffffaa080638 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5744 ffffffffa9a687d8 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.1 6008 fffffe84d28eae20 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6184 fffffe84d12127c8 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
1.0 5880 fffffe84d34bcb10 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5968 fffffe84e06d5188 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5744 ffffffffa9fa6198 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5976 fffffe84e7205fa8 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6144 ffffffffaa4f0c78 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6200 fffffe84d5a4cfc0 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5984 fffffe84e72007e0 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5992 ffffffffab603648 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 6160 ffffffffabfee330 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.1 5984 fffffe84d344c648 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5896 ffffffffab761fb0 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf
0.0 5960 fffffe84d1212f98 /opt/nginx/sbin/nginx
/opt/nginx/sbin/nginx -c
/opt/nginx/conf/nginx.conf

For the moment my nginx is soft:

http://img192.imageshack.us/img192/4152/nginxnow.png

I will try to launch the command when Nginx is more used (maybe this
night)

Thanks for your help,

Julien

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,194233,194246#msg-194246

Hello!

On Thu, Apr 28, 2011 at 08:12:59AM -0400, darckos wrote:

Hello,

I use nginx 0.8.53 on Solaris 10 and I feel like I reach a kind of limit
but I’m not abble to find it.
I use Nginx to serve static image and sometimes to get one image I need
to wait 10 or more seconds.

Most likely nginx actually disk bound and worker processes are
blocked on disk, and that’s what causes such delays.

You may want to examine iostat output to see if it really happens.

default_type  application/octet-stream;

sendfile        on;

[…]

Do you have any idea to solve this problem ?

Try the following (in no particular order, you may want actually
to start with (3)):

  1. Tune worker_processes to add more (or less) concurency to make
    sure disks are reasonably loaded but not overloaded (i.e. disk r/w
    times are small enough).

  2. Try switching accept_mutex off to get better request
    distribution between worker processes.

    events {
    accept_mutex off;

    }

  3. Switch off sendfile and tune output_buffers to use 1-2 big
    enough buffers to make sure requests to disks will be big (and
    disks aren’t trashed with large number of small requests).
    Something like

    output_buffers 2 256k;

will do the trick (using bigger buffers may help better if you
have enough memory and your images are big enough).

Ideally aio should be used in such situation, but it’s not
available with nginx under Solaris.

Maxim D.

Thanks for your help. I will try all these solutions.

Regards,

Julien

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,194233,194297#msg-194297

On Thu, Apr 28, 2011 at 09:38:18AM -0400, darckos wrote:

Yes, like you said… it is Solaris :slight_smile:

0.1 5984 fffffe84d344c648 /opt/nginx/sbin/nginx /opt/nginx/sbin/nginx -c >
/opt/nginx/conf/nginx.conf

[skipped other nginx procs]

Better, but on FreeBSD I have the name of process state:

0.0 9264 pause nginx: master process /usr/local/sbin/nginx
0.0 9264 kqread nginx: worker process (nginx)

It looks like it doesn’t exist in Solaris.


Sergey A. Osokin
[email protected]

Server is not busy at all. 16G of memory and 24 core just for static
delivery … :slight_smile:

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,194233,194456#msg-194456

darckos Wrote:

I use Nginx to serve static image and sometimes to
get one image I need to wait 10 or more seconds.

Strangely I was having the exact same problem on a very busy freebsd 8.0
i386 machine that serves images as well. Raising worker_priority to -10
helped a lot.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,194233,194370#msg-194370

darckos Wrote:

Server is not busy at all. 16G of memory and 24
core just for static delivery … :slight_smile:

Well sometimes it happens even when machine is idle and serving about 10
or 20 images per second. But seems to happen more often with normal
priority. So it might not be related to how busy machine is.
Anyway I still can’t get rid of this problem completely and can’t find a
way to reproduce it.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,194233,194501#msg-194501

Hello,

I just changed my config with option 3 and I feel like my nignx is
better now.

http://img232.imageshack.us/img232/5369/nginx.png

My new config is :

worker_processes 24;

events {
worker_connections 2048;
devpoll_events 1;
}

http {
include mime.types;
default_type application/octet-stream;

output_buffers 2 256k;
keepalive_timeout  15;


}

Thanks for your help

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,194233,195256#msg-195256

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs