Hi,
On a production server, I found that nginx appears to leak when using
ssl. With some investigation, it seems that this is actually memory
fragmentation due to the session cache. I made a very simple
configuration for the server:
daemon off;
master_process off;
pid /tmp/x.pid;
error_log /tmp/x.log;
events { use epoll; }
http {
client_body_temp_path /tmp;
proxy_temp_path /tmp;
fastcgi_temp_path /tmp;
access_log /tmp/access.log;
server {
listen localhost:8666;
ssl on;
ssl_certificate /home/bmaurer/x.pem;
ssl_certificate_key /home/bmaurer/x.pem;
root /tmp;
}
}
Then I did a benchmark with the following command:
ab -c500 -n20000 https://localhost:8666/
After doing this, the server uses ~ 30 MB of RSS. Running it once more,
it uses ~ 40 MB of RSS. Valgrind claims that there are no “leaks”, it
seems that there’s just a really bad case of memory fragmentation.
I tried applying this to the SSL configuration:
ssl_session_cache builtin:2;
Doing so resulted in the memory use of the nginx server staying
relatively low (it appears the memory was reclaimed from the OS after it
was used).
It seems like it might be worth switching to something like the shared
memory cache by default. Keeping the long-lived session cache in a
different pool of memory avoids the risk of large amounts of memory
getting pinned in.
One other thing I noticed while investigating this stuff was that nginx
keeps a 16 KB buffer for each SSL connection for the entire duration of
the connection. I’ve attached a patch that keeps this buffer alive only
while there’s a pending write. Sadly, there are some relatively large
buffers internal to openssl as well, which means the overhead for SSL
keepalive connections is pretty high.
On Wed, Dec 26, 2007 at 12:34:39PM -0500, Ben Maurer wrote:
http {
}
I tried applying this to the SSL configuration:
getting pinned in.
Well, I will make shared session cache by default. It seems it’s quite
stable. Other possible drawback of builtin cache as I think: it uses
a hash to store sessions and cache cleaning may take a long time.
One other thing I noticed while investigating this stuff was that nginx
keeps a 16 KB buffer for each SSL connection for the entire duration of
the connection. I’ve attached a patch that keeps this buffer alive only
while there’s a pending write. Sadly, there are some relatively large
buffers internal to openssl as well, which means the overhead for SSL
keepalive connections is pretty high.
Thank you, I will allocate it on demand, and will free using special
function just before a request will go to keep-alive state.
As to OpenSSL it takes about 100K per connection.
Hi,
Igor S. wrote:
On Wed, Dec 26, 2007 at 12:34:39PM -0500, Ben Maurer wrote:
It seems like it might be worth switching to something like the shared
memory cache by default. Keeping the long-lived session cache in a
different pool of memory avoids the risk of large amounts of memory
getting pinned in.
Well, I will make shared session cache by default. It seems it’s quite
stable. Other possible drawback of builtin cache as I think: it uses
a hash to store sessions and cache cleaning may take a long time.
It might be worth having an option to disable the cache completely –
right now all you can do is have a very small builtin cache. I have a
configuration where caching is pretty rare (in most cases, we only serve
2 http requests to a user over a single keepalive connection).
As to OpenSSL it takes about 100K per connection.
That’s worse than I thought…
-b
On Wed, Dec 26, 2007 at 01:49:52PM -0500, Ben Maurer wrote:
stable. Other possible drawback of builtin cache as I think: it uses
a hash to store sessions and cache cleaning may take a long time.
It might be worth having an option to disable the cache completely –
right now all you can do is have a very small builtin cache. I have a
configuration where caching is pretty rare (in most cases, we only serve
2 http requests to a user over a single keepalive connection).
Well, I have added
ssl_session_cache off;
and made it by default, as Apache mod_ssl does.