Excessive RAM consumption - memory leak

Server NGINX
1

I am using Nginx 0.5.35 on a server that has a Xeon 5130 dual core, 4 GB
of RAM, and 10,000 RPM HD. Nginx serves millions of large images per
day. Sometimes tens of millions in a day. Over the past week the server
has been experiencing record traffic. I am looking for a way to reduce
the amount of RAM consumed by Nginx, but still deliver images at the
same rate. The CPU cycles are at an acceptable level, but yesterday
Nginx nearly consumed all 4 GB of RAM before I had to reboot the server
under very heavy traffic. All this server does is serve images files
like jpeg, etc. Some of my configuration is below:

Since this is a dual core CPU I am using:

worker_processes 2;
worker_connections 12000;
use epoll; # This is a RedHat Enterprise Server 4

I also have:

gzip on;
sendfile       on;
tcp_nopush     on;
tcp_nodelay    on;
keepalive_timeout  75 20;
server_names_hash_bucket_size 128;

I have tried reducing the keepalive_timeout to close the connection
sooner, so that resources might be freed sooner, but it has no noticable
effect.

Can someone make some suggestions how I could handle the same traffic,
but manage the RAM usage better.

2

On 2/20/08, Todd HG [email protected] wrote:

I have tried reducing the keepalive_timeout to close the connection
sooner, so that resources might be freed sooner, but it has no noticable
effect.

Can someone make some suggestions how I could handle the same traffic,
but manage the RAM usage better.

Wow. I would set keepalive to maybe 5 or 10, not 75.
If you are serving that much traffic, I might even try turning
keepalives off altogether.

But if you already modified those values, and didn’t see a change,
then I don’t know.
You may try setting expires headers for your images, if they don’t
change very often (or at all).

3

Hi Todd,

I’m supprised you are finding memory usage a problem, we handle 2mm
hits / day with nginx and memory usage is about 3.5 megabytes total.

What revision of nginx are you running?

Try pointing yslow at your host, and then follow its recommendations
to improve response times (mainly to reduce origin hits).

Cheers

Dave

4

On Wed, Feb 20, 2008 at 08:49:12PM +0100, Todd HG wrote:

Since this is a dual core CPU I am using:
tcp_nodelay on;
keepalive_timeout 75 20;
server_names_hash_bucket_size 128;

I have tried reducing the keepalive_timeout to close the connection
sooner, so that resources might be freed sooner, but it has no noticable
effect.

Can someone make some suggestions how I could handle the same traffic,
but manage the RAM usage better.

Do you use the standard nginx without any external modules ?
Does nginx serve static files only without any proxy, fastcgi, perl,
etc.
processing ?

What does

ps ax -o pid,ppid,%cpu,vsz,wchan,command|egrep ‘(nginx|PID)’

show ?

5

Do you use the standard nginx without any external modules ?
Does nginx serve static files only without any proxy, fastcgi, perl,
etc.
processing ?

What does

ps ax -o pid,ppid,%cpu,vsz,wchan,command|egrep ‘(nginx|PID)’

show ?

Yes, I use standard Nginx without proxy, fastcgi, perl, ect. It is
compiled and installed without any added modules. Nginx is only serving
the static image files.

ps ax -o pid,ppid,%cpu,vsz,wchan,command|egrep ‘(nginx|PID)’

shows:

PID PPID %CPU VSZ WCHAN COMMAND
9327 1 0.0 2376 rt_sig nginx: master process
/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
9328 9327 8.8 11192 - nginx: worker process
9329 9327 8.7 13200 - nginx: worker process
23660 23641 0.0 5136 pipe_w egrep (nginx|PID)

I also have configured:

client_header_timeout  3m;
client_body_timeout    3m;
send_timeout           3m;
6

On Wed, Feb 20, 2008 at 10:22:52PM +0100, Todd HG wrote:

/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
9328 9327 8.8 11192 - nginx: worker process
9329 9327 8.7 13200 - nginx: worker process
23660 23641 0.0 5136 pipe_w egrep (nginx|PID)

I also have configured:

client_header_timeout  3m;
client_body_timeout    3m;
send_timeout           3m;

What gzip settings do you use ?

7

Dave C. wrote:

Hi Todd,

I’m supprised you are finding memory usage a problem, we handle 2mm
hits / day with nginx and memory usage is about 3.5 megabytes total.

What revision of nginx are you running?

Try pointing yslow at your host, and then follow its recommendations
to improve response times (mainly to reduce origin hits).

Cheers

Dave

Could you post your config file, so that I might compare to my own?

8

You should probably use the defaults for a server that only serves
static images. If they client can’t talk to you fast enough to send a
small GET request, they probably won’t be able to receive the response
in a timely manner. Best to drop the quickly.

Dave

9

Sure, please forward me your contact details, the config is spread out
over many files so isn’t appropriate to post here.

Cheers

Dave

10

On Thu, Feb 21, 2008 at 08:36:31AM +1100, Dave C. wrote:

You should probably use the defaults for a server that only serves
static images. If they client can’t talk to you fast enough to send a
small GET request, they probably won’t be able to receive the response
in a timely manner. Best to drop the quickly.

If nginx uses sendfile, it eats kernel memory, but not its own.
So these timeouts should not affect on nginx memory usage.

11

Try pointing yslow at your host, and then follow its recommendations
to improve response times (mainly to reduce origin hits).

Thanks for letting me know about Yslow. It’s a great tool. For anyone
who reads this later, Yslow requires Firebug to run in Firefox.

http://www.getfirebug.com/

12

Igor S. wrote:

On Wed, Feb 20, 2008 at 10:22:52PM +0100, Todd HG wrote:

/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
9328 9327 8.8 11192 - nginx: worker process
9329 9327 8.7 13200 - nginx: worker process
23660 23641 0.0 5136 pipe_w egrep (nginx|PID)

I also have configured:

client_header_timeout  3m;
client_body_timeout    3m;
send_timeout           3m;

What gzip settings do you use ?

gzip on;
gzip_min_length  1100;
gzip_buffers     4 8k;
gzip_types       image/jpg image/jpeg image/gif image/png text/plain

text/xml application/xhtml+xml text/css application/xml image/svg+xml
application/rss+xml application/atom_xml application/x-javascript
application/x-httpd-php application/x-httpd-fastphp
application/x-httpd-eruby text/html;
gzip_comp_level 9;

13

Dave C. wrote:

Sure, please forward me your contact details, the config is spread out
over many files so isn’t appropriate to post here.

Cheers

Dave

I believe you can attach files when replying to these messages. I’m a
bit wary of posting my email address on a bbs or chat forum. However, I
would really like to see how you’ve configured your Nginx install.

14

On 20.02.2008, at 22:59, Todd HG wrote:

plain
text/xml application/xhtml+xml text/css application/xml image/svg+xml
application/rss+xml application/atom_xml application/x-javascript
application/x-httpd-php application/x-httpd-fastphp
application/x-httpd-eruby text/html;
gzip_comp_level 9;

sorry, why do you compress jpeg, gif and png files? they are already
compressed… double compression just uses cpu power and causes global
warming :slight_smile:

jodok

–
Posted via http://www.ruby-forum.com/.

–
“Beautiful is better than ugly.”
– The Zen of Python, by Tim Peters

Jodok B., Lovely Systems GmbH
Schmelzhütterstraße 26a, 6850 Dornbirn, Austria
mobile: +43 676 5683591, phone: +43 5572 908060

15

Jodok B. wrote:

sorry, why do you compress jpeg, gif and png files? they are already
compressed… double compression just uses cpu power and causes global
warming :slight_smile:

jodok

You may a great point. I had created a one-size-fits-all config, but I
will comment out those files that don’t need to be compressed. Thank you
for pointing that out.

16

dave AT cheney DOT net

17

On Wed, Feb 20, 2008 at 10:59:54PM +0100, Todd HG wrote:

text/xml application/xhtml+xml text/css application/xml image/svg+xml
application/rss+xml application/atom_xml application/x-javascript
application/x-httpd-php application/x-httpd-fastphp
application/x-httpd-eruby text/html;
gzip_comp_level 9;

This is cause of memory and CPU consumption. You do not need to compress
already compressed jpegs/etc. If you serve images only, you should turn
gzip off at all (it’s default).

As to other MIME types:

  1. there are no such types as
    application/x-httpd-php
    application/x-httpd-fastphp
    application/x-httpd-eruby
    they probably exist as internal MIME-types inside Apache, but they are
    never showed to a client.

  2. the following types as
    application/xhtml+xml
    application/rss+xml
    application/atom_xml

probably do not exist too.

Keep the list as small as possible, because nginx iterates it
sequenctally.

18

On Thu, Feb 21, 2008 at 01:22:08AM +0300, Igor S. wrote:

gzip_buffers     4 8k;

application/xhtml+xml
application/rss+xml
application/atom_xml

probably do not exist too.

Keep the list as small as possible, because nginx iterates it sequenctally.

image/jpg does not exist too.

By default nginx in conf/mime.type uses text/xml for xml and rss, so
application/xml is duplicate.

19

eliott wrote:

You may try setting expires headers for your images, if they don’t
change very often (or at all).

Do you have an example for setting the expire header. In the code
example it shows:

expires       24h;
expires       0;
expires       -1;
expires       epoch;
add_header    Cache-Control  private;
server_tokens off;

I’m not sure if I should be using only:

expires       24h;
add_header    Cache-Control  private;
server_tokens off;

or if I also need:

expires       0;
expires       -1;
expires       epoch;
20

You might want to consider dropping your gzip ratio, local testing
here showed little benefit past about 4. At level 9 you’ll be using 4x
the CPU for a tiny gain in compression, which is more than mitigated
by the extra delay in overcompressing the pages.

Also, as Jodok has just pointed out, there is little observable gain
in compressing image/* mime types.

[dave@crimson nginx]$ cat gzip.conf
gzip on;
gzip_proxied any;
gzip_http_version 1.1;
gzip_min_length 1100;
gzip_comp_level 5;
gzip_buffers 4 8k;
gzip_types text/plain text/html text/xml text/css application/x-
javascript text/xml application/xml application/xml+rss text/
javascript application/atom+xml;

You could try the gzip_static module in the development branch to