Excessive RAM consumption - memory leak

Server NGINX
21

Igor S. wrote:

On Thu, Feb 21, 2008 at 01:22:08AM +0300, Igor S. wrote:

gzip_buffers     4 8k;

application/xhtml+xml
application/rss+xml
application/atom_xml

probably do not exist too.

Keep the list as small as possible, because nginx iterates it sequenctally.

image/jpg does not exist too.

By default nginx in conf/mime.type uses text/xml for xml and rss, so
application/xml is duplicate.

I’ve trimmed down gzip to files I serve off the image server such as css
and js, but no longer include the image entries. My list is now:

gzip_types   text/css application/x-javascript;

and

gzip_comp_level 5;
22

eliott wrote:

You may try setting expires headers for your images, if they don’t
change very often (or at all).

I have added, but I’m not altogether sure if this is optimum:

http {
expires 24h;
add_header Cache-Control private;
server_tokens off;

23

Dave C. wrote:

You could try the gzip_static module in the development branch to

How do I compilet the gzip_static into Nginx?

24

On Wed, Feb 20, 2008 at 11:38:50PM +0100, Todd HG wrote:

Dave C. wrote:

You could try the gzip_static module in the development branch to

How do I compilet the gzip_static into Nginx?

http://wiki.codemongers.com/NginxHttpGzipStaticModule

25

On Wed, Feb 20, 2008 at 11:42:22PM +0100, Todd HG wrote:

gzip_types   text/css application/x-javascript;

OK.

and

gzip_comp_level 5;

Actually “gzip_comp_level 1” is enough (this is default).
But if you will use gzip_static_module, you should compress the files
with
“gzip -9”.

26

Why are you setting the cache control to private on a public asset
image ?

27

On Wed, Feb 20, 2008 at 11:47:29PM +0100, Todd HG wrote:

eliott wrote:

You may try setting expires headers for your images, if they don’t
change very often (or at all).

I have added, but I’m not altogether sure if this is optimum:

http {
expires 24h;
add_header Cache-Control private;
server_tokens off;

If the images are public (not per client) you should set

  expires   1M;

only to allow long caching in transit cache proxies.

“server_tokens off” simply turn off nginx version from “Server” header:
“Server: nginx” vs “Server: nginx/0.6.26”.

28

Igor S. wrote:

On Wed, Feb 20, 2008 at 11:47:29PM +0100, Todd HG wrote:

I’ve set the cache to:

expires       1M;
add_header    Cache-Control  public;

I’ve also added the following to the config:

connection_pool_size 256;
client_header_buffer_size 1k;
large_client_header_buffers 4 4k;
request_pool_size 4k;
output_buffers 4 32k;
postpone_output 1460;

ignore_invalid_headers on;

It appears the:

connection_pool_size 256;

… caused a big drop in my RAM usage. Without this set, does the server
just keep pooling connections?

29

Igor S. wrote:

On Wed, Feb 20, 2008 at 11:47:29PM +0100,

Is there anywhere I could read more about how Nginx uses
connection_pool_size, stores connections, if it is in a cache in RAM or
hard drive, and what else might be stored in RAM by Nginx?

30

On 2/20/08, Todd HG [email protected] wrote:

expires       -1;

or if I also need:

expires       0;
expires       -1;
expires       epoch;

The expires stanza tells how long the client can cache the object, or
tells a proxy how long it can cache it. So you only need one of them
(or one per location match stanzas).

Cache control private tells a proxy, for instance, that it should not
cache the object, but that a browser on an endpoint workstation can
cache it.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1

If you wanted to let everyone cache an image forever (like maybe an
image you never expect to change), you can set expires to max.

This may reduce some traffic to you, as it will allow for greater
cacheability downstream.

Take into consideration that clients may not fetch new objects if they
change though… I would put it into a stanza based on filetype, like
specific to images that will never change.

Hope some of that info helps.

31

On Thu, Feb 21, 2008 at 03:49:41AM +0100, Todd HG wrote:

Igor S. wrote:

On Wed, Feb 20, 2008 at 11:47:29PM +0100,

Is there anywhere I could read more about how Nginx uses
connection_pool_size, stores connections, if it is in a cache in RAM or
hard drive, and what else might be stored in RAM by Nginx?

All that you need is to disable gzipping images. It’s enough.
Other default settings do not allow workers to grow up.

32

Igor S. wrote:

On Wed, Feb 20, 2008 at 11:47:29PM +0100,

Is there anywhere I could read more about how Nginx uses
connection_pool_size, stores connections, if it is in a cache in RAM or
hard drive, and what else might be stored in RAM by Nginx?

All that you need is to disable gzipping images. It’s enough.
Other default settings do not allow workers to grow up.

Of course disabling gzip defeats the purpose of having gzip decrease
bandwidth and increase site speed for readers. Right now I only have
gzip handling a few million js and css files a day, in addition to tens
of millions of images which are not gzipped, but the RAM usage just
grows until it is completely consumed. By setting the gzip compression
level to 1 the RAM consumption grows more slowly, but eventually eats
all the RAM.

It appears what might be needed is a setting to allow the total number
of connections for gzip to be set before Nginx automatically kills and
restarts a worker. This would be similar to the Apache
MaxRequestsPerChild limit setting. There should be a way to set Nginx to
kill and restart the worker process to free the RAM, and start again at
zero for situations like mine.

Without a solution I need to restart my server about every 24 hours, and
this is a very robust server.

33

I do not know how you restart nginx but you can send the control process
“kill -HUP” and it will do exactly what you want, which is gracefully
restart each worker process. You can have a script check for the memory
usage and do that when you see it is getting high, or simply do that
every
24 hours.

Kiril

34

On Thu, Feb 21, 2008 at 01:22:10AM +0100, Todd HG wrote:

Igor S. wrote:

On Wed, Feb 20, 2008 at 11:47:29PM +0100, Todd HG wrote:

I’ve set the cache to:

expires       1M;
add_header    Cache-Control  public;

You may omit “add_header Cache-Control public” at all.

It appears the:

connection_pool_size 256;

… caused a big drop in my RAM usage.

This is default setting.

Without this set, does the server just keep pooling connections?

The connection_pool_size set initial size of memory pool per connection.
If a connection needs more memory nginx allocates it by chunk of this
size.

You may omit and use default values of

connection_pool_size
client_header_buffer_size
large_client_header_buffers
request_pool_size
output_buffers
postpone_output
ignore_invalid_headers

35

Kiril A. wrote:

I do not know how you restart nginx but you can send the control process
“kill -HUP” and it will do exactly what you want, which is gracefully
restart each worker process. You can have a script check for the memory
usage and do that when you see it is getting high, or simply do that
every
24 hours.

Kiril

I do have a script that restarts Nginx already, however, only rebooting
the machine actually clears the RAM. This is why I was curious to know
how and what Nginx stores in the RAM.

36

On Thu, Feb 21, 2008 at 08:17:01PM +0100, Todd HG wrote:

Of course disabling gzip defeats the purpose of having gzip decrease
MaxRequestsPerChild limit setting. There should be a way to set Nginx to
kill and restart the worker process to free the RAM, and start again at
zero for situations like mine.

Could you show what does

ps ax -o pid,ppid,%cpu,vsz,rss,wchan,command|egrep ‘(nginx|PID)’

show when nginx grows up ?

What OS do you use ?

Without a solution I need to restart my server about every 24 hours, and
this is a very robust server.

It’s really strange. I run all my sites unattended. The workers are
restarted
only for reconfiguration or online upgrade. For example, this nginx
runs more than 2 days (static, SSI, gzipping, proxying) without any
leaks (60-120M is stable state):

ps ax -o pid,ppid,%cpu,vsz,lstart,wchan,command|egrep ‘(nginx|PID)’
PID PPID %CPU VSZ STARTED WCHAN COMMAND
1645 1 0.0 16520 Mon Feb 18 02:16:37 2008 pause nginx: master
proces
66458 1645 24.9 78984 Tue Feb 19 18:10:20 2008 kqread nginx: worker
proces

Now it handles 22000 simultaneous connections:

fstat | grep ‘nginx.*tcp’ | awk ‘{print $3}’ | sort | uniq -c
8 1645
22013 66458

16 hours per day it handles 1000-2000 requests per seconds.
This is 60-100 millions per day.

37

I do have a script that restarts Nginx already, however, only rebooting
the machine actually clears the RAM. This is why I was curious to know
how and what Nginx stores in the RAM.

Wait… are you talking about overall system ram, or ram used by nginx
processes?
If you mean overall system ram, you are probably just noticing that
the OS (esp linux) has cached many of the files in the filesystem
(IOcache) to speed up read operations.

38

eliott wrote:

I do have a script that restarts Nginx already, however, only rebooting
the machine actually clears the RAM. This is why I was curious to know
how and what Nginx stores in the RAM.

Wait… are you talking about overall system ram, or ram used by nginx
processes?
If you mean overall system ram, you are probably just noticing that
the OS (esp linux) has cached many of the files in the filesystem
(IOcache) to speed up read operations.

How would I stop Linux from filling up RAM with IOcache? How would I
check to see if that is the case? I’m willing to consider all input.

39

Igor S. wrote:

On Thu, Feb 21, 2008 at 08:17:01PM +0100, Todd HG wrote:

Of course disabling gzip defeats the purpose of having gzip decrease
MaxRequestsPerChild limit setting. There should be a way to set Nginx to
kill and restart the worker process to free the RAM, and start again at
zero for situations like mine.

Could you show what does

ps ax -o pid,ppid,%cpu,vsz,rss,wchan,command|egrep ‘(nginx|PID)’

show when nginx grows up ?

What OS do you use ?

Without a solution I need to restart my server about every 24 hours, and
this is a very robust server.

It’s really strange. I run all my sites unattended. The workers are
restarted
only for reconfiguration or online upgrade. For example, this nginx
runs more than 2 days (static, SSI, gzipping, proxying) without any
leaks (60-120M is stable state):

ps ax -o pid,ppid,%cpu,vsz,lstart,wchan,command|egrep ‘(nginx|PID)’
PID PPID %CPU VSZ STARTED WCHAN COMMAND
1645 1 0.0 16520 Mon Feb 18 02:16:37 2008 pause nginx: master
proces
66458 1645 24.9 78984 Tue Feb 19 18:10:20 2008 kqread nginx: worker
proces

Now it handles 22000 simultaneous connections:

fstat | grep ‘nginx.*tcp’ | awk ‘{print $3}’ | sort | uniq -c
8 1645
22013 66458

16 hours per day it handles 1000-2000 requests per seconds.
This is 60-100 millions per day.

I agree this is a strange problem. I am running Nginx on Redhat
Enterprise Server 4. I will post the output of the Niginx master and
worker processes once the memory reaches it’s max again. That should be
in about 24 hours. Just for reference I am posting my exact current
configuration below, but of course I’ve replaced some values to keep
them private:

user nobody;
worker_processes 2;

The worker_processes and worker_connections from the event sections

allows you to calculate maxclients value:

max_clients = worker_processes * worker_connections

pid /usr/local/nginx/logs/nginx.pid;

events {
worker_connections 12000;
use epoll;
}

http {
include /usr/local/nginx/conf/mime.types;
default_type application/octet-stream;

expires       1M;
add_header    Cache-Control  must-revalidate;
add_header    Cache-Control  public;
server_tokens off;

client_header_timeout  3m;
client_body_timeout    3m;
send_timeout           3m;

gzip off;

sendfile       on;
tcp_nopush     on;
tcp_nodelay    on;

keepalive_timeout  5 5;

server_names_hash_bucket_size 128;

server {
    listen       ip-address:9000;
    server_name  images.mydomain.com;

error_page  404              http://www.mydomain.com/e404.php;

location / {
root /var/www/mydomain;
expires 30d;
valid_referers blocked mydomain.com;
if ($invalid_referer) {
# return 404;
rewrite ^(.*)$ http://www.mydomain.com/;
}
deny ip-address;
allow all;
}

}

}

40

On 2/21/08, Todd HG [email protected] wrote:

How would I stop Linux from filling up RAM with IOcache? How would I
check to see if that is the case? I’m willing to consider all input.

iocache is a good thing!
Linux uses available ram… no point having unused ram laying around
when you can speed up things with it. If an application needs ram, the
iocache is reduced to make room for it.

Not sure if that is what you are experiencing though.
Post the output of the ps command Igor listed, once you feel your
system has been up long enough to show us what you mean by ‘using lots
of ram’.

also do a ‘free -m’