Normal memory usage for SSL terminating, reverse proxy nginx?

Server NGINX
1

It has been a difficult topic to research. The nginx instance is doing
nothing more than what the subject stated. It reverse proxies to a
backend,
load balanced set of web app instances and terminates SSL for a large
number
of unique domains each with their own SSL cert. Here’s a ps aux of
nginx
running after a clean start and zero (out of rotation) traffic.

root 20 0 676052 598224 1848 S 0.0 16.5 0:00.06 nginx

nginx 20 0 675552 597204 1228 S 0.0 16.5 0:00.44 nginx

nginx 20 0 675552 596612 636 S 0.0 16.5 0:00.36 nginx

Looking at that process list, nginx is using about 676mb of RAM for ~400
vhosts each with their own unique SSL cert for a unique domain. Here’s
an
example of a vhost server config. They’re all generated based on the
same
base template:

server {
listen 443 ssl proxy_protocol;
server_name www.;
access_log /var/log/nginx/access_vhost_443.log accesslog;
error_log /var/log/nginx/error_vhost_443.log warn;
real_ip_header proxy_protocol;

ssl on;
ssl_certificate
/etc/nginx/ssl//.crt;
ssl_certificate_key
/etc/nginx/ssl//.key;

ssl_stapling on;
ssl_stapling_verify on;

resolver internal-dns.vpc valid=60s;

set $internal “upstream-load-balancer.vpc”;
location / {
if ($denied) {
return 444;
}
proxy_pass http://$internal;
}
}

Now, this wouldn’t be all that bad. 1.69mb of memory per vhost isn’t
horrible, high, but not unsustainable. However, if I do nginx -s reload or
restart via systemd service…

root 20 0 1370188 1.176g 3240 S 0.0 33.4 0:14.98 nginx

nginx 20 0 1370192 1.175g 2584 S 0.3 33.4 2:39.95 nginx

nginx 20 0 1370192 1.175g 2584 S 1.7 33.4 2:28.42 nginx

It doubles the memory consumption! It never goes up or down drastically
again. It’s as if it duplicates and never frees or releases unless you
do a
restart.

This was tested on a handful of AWS EC2 instance types using vanilla
Centos7
and both nginx 1.6.3 (stable in centos repos) and nginx 1.10.0
(nginx.org
repo).

In summary, my questions are thus:

  • Is it normal for nginx to use ~1.7mb per SSL vhost?

  • Is there a way to reduce that memory usage?

  • Am I the only one that experiences the doubling of nginx memory usage
    after a nginx reload?

  • Is that a bug?

Thanks!

Posted at Nginx Forum:

2

Hello!

On Fri, May 27, 2016 at 03:15:25AM -0400, onecrazymonkey wrote:

nginx 20 0 675552 596612 636 S 0.0 16.5 0:00.36 nginx
error_log /var/log/nginx/error_vhost_443.log warn;
real_ip_header proxy_protocol;

ssl on;
ssl_certificate /etc/nginx/ssl//.crt;
ssl_certificate_key /etc/nginx/ssl//.key;

ssl_stapling on;
ssl_stapling_verify on;

resolver internal-dns.vpc valid=60s;

Note: by using using own resolver in each server{} you add more
flexibility, but waste memory and other resources. Consider using
single resolver defined at http{} level instead.

horrible, high, but not unsustainable. However, if I do nginx -s reload or
restart.
As nginx allocates a new configuration before releasing the old
one, and this doubles memory consumption of the master process for a
while. The old configuration is then freed, but in many cases
this memory isn’t really returned to kernel by the allocator, and
hence seen in various stats.

This was tested on a handful of AWS EC2 instance types using vanilla Centos7
and both nginx 1.6.3 (stable in centos repos) and nginx 1.10.0 (nginx.org
repo).

In summary, my questions are thus:

  • Is it normal for nginx to use ~1.7mb per SSL vhost?

Not really. This is likely due to a particular configuration you
are using: own resolver in each vhost, own logs, etc. Consider
moving common settings to the http{} level, it should somewhat reduce
memory consumption.

Just for the reference: a simple test with 10k SSL vhosts like:

server { listen 8443; ssl on; server_name 1.foo; ssl_certificate
test.crt; ssl_certificate_key test.key; }
server { listen 8443; ssl on; server_name 2.foo; ssl_certificate
test.crt; ssl_certificate_key test.key; }
…
server { listen 8443; ssl on; server_name 10000.foo; ssl_certificate
test.crt; ssl_certificate_key test.key; }

takes about 200mb of memory (on a 32-bit host). That is, about
20k of memory per vhost.

With more options it will take more memory, but 1.7mb is a bit too
many - you are doing something wrong.

Just a guess: if you use ssl_trusted_certificate at http{} level,
make sure you are loading only needed certificates, not a full
OS-provided bundle.

  • Is there a way to reduce that memory usage?

Yes, see various hints above. In summary: configure things at
http{} level where possible, avoid memory-hungry things you don’t
need.

  • Am I the only one that experiences the doubling of nginx memory usage
    after a nginx reload?
  • Is that a bug?

No and no, see above.

–
Maxim D.
http://nginx.org/