Worker process and memory leak?

Server NGINX
1

Hi all,

For testing, I compile nginx with following options :
./configure --with-http_ssl_module --without-http_charset_module
–without-http_
ssi_module --without-http_auth_basic_module
–without-http_autoindex_module --wi
thout-http_geo_module --without-http_map_module
–without-http_limit_zone_module
–without-http_empty_gif_module --without-http_browser_module
–without-http_up
stream_ip_hash_module --without-http_gzip_module
–without-http_userid_module –
without-http_access_module --without-http_referer_module
–without-http_rewrite_
module --without-http_proxy_module --without-http_fastcgi_module
–without-http_
memcached_module --without-mail_pop3_module --without-mail_imap_module
–without
-mail_smtp_module

nginx.conf:

worker_processes 1;
error_log logs/error.log info;

events {
worker_connections 1024;
use epoll;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;

server {
listen 443;
server_name myip;

client_max_body_size 100M;
ssl on;
ssl_verify_client on;
ssl_certificate server.pem;
ssl_certificate_key server.key;
ssl_client_certificate ca.pem;
ssl_session_timeout 5m;

    ssl_protocols  TLSv1;
    ssl_ciphers

ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;

    location / {
        root   html;
        index  index.html index.htm;
    }
}

}

Without any request, the state of worker process is:
15027 nobody 18 0 4104 944 392 S 0 0.1 0:00.00 nginx

When I flood it with 50 bad request (GET /test.test => 404) the worker
process becomes:
15027 nobody 16 0 4584 2300 1308 S 0 0.2 0:04.99 nginx

With another 50 bad request:
15027 nobody 16 0 5012 2668 1308 S 0 0.3 0:09.94 nginx

etc…

Nginx is it concerned? Hence the problem can come?

Thanks in advance

2

After many test, I found the memory leak cause. It happens when
ssl_verify_client is activate. Nginx doesn’t seem have memory leak when
this parameter is off.

Chavelle V. wrote:

ssl_verify_client on;

So I run valgrind to have more information and I found this :

340 bytes in 2 blocks are possibly lost in loss record 4 of 10
at 0x4022AB8: malloc (vg_replace_malloc.c:207)
by 0x430448D: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4304AEE: CRYPTO_malloc (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x433A256: BN_MONT_CTX_new (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x433A362: BN_MONT_CTX_set_locked (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x434E886: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x435035D: RSA_public_decrypt (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4351149: RSA_verify (in /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x428D005: ssl3_get_cert_verify (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x4290268: ssl3_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x42A729C: SSL_do_handshake (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x8069048: ngx_ssl_handshake (ngx_event_openssl.c:408)

280,022 (57,932 direct, 222,090 indirect) bytes in 249 blocks are
definitely lost in loss record 9 of 10
at 0x4022AB8: malloc (vg_replace_malloc.c:207)
by 0x430448D: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4304AEE: CRYPTO_malloc (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x438BD84: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x438E9B4: ASN1_item_ex_d2i (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x438F124: ASN1_item_d2i (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4385CD4: d2i_X509 (in /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x428C8D3: ssl3_get_client_certificate (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x4290381: ssl3_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x42A729C: SSL_do_handshake (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x8069048: ngx_ssl_handshake (ngx_event_openssl.c:408)
by 0x8069404: ngx_ssl_handshake_handler (ngx_event_openssl.c:540)

I have no idea how to solve this problem, and if it is possible to free
pointer in nginx. I don’t found a lot of help on google.

For information : http://bugs.mysql.com/bug.php?id=14780

Thanks again

3

On Wed, Apr 23, 2008 at 06:22:04PM +0200, Chavelle V. wrote:

After many test, I found the memory leak cause. It happens when
ssl_verify_client is activate. Nginx doesn’t seem have memory leak when
this parameter is off.

Thank you for your inverstigation.
Could you test the attached patch ?

4

Igor S. wrote:

Thank you for your inverstigation.
Could you test the attached patch ?

The patch works fine, it resolves the problem.

Thanks a lot !!!

5

Visibly, there is another memory leak when ssl_verify_client is also
activate.

21,440 bytes in 80 blocks are definitely lost in loss record 6 of 8
at 0x4022AB8: malloc (vg_replace_malloc.c:207)
by 0x430448D: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4304AEE: CRYPTO_malloc (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x436113C: BUF_MEM_grow (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x439E9B2: X509_NAME_oneline (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x8069962: ngx_http_ssl_verify_callback (ngx_event_openssl.c:305)
by 0x43A086C: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x43A15F8: X509_verify_cert (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x42AB922: ssl_verify_cert_chain (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x428C9F9: ssl3_get_client_certificate (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x4290381: ssl3_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x42A78C9: SSL_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)

268 bytes in 1 blocks are possibly lost in loss record 5 of 9
at 0x4022AB8: malloc (vg_replace_malloc.c:207)
by 0x430448D: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4304AEE: CRYPTO_malloc (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x436113C: BUF_MEM_grow (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x439E9B2: X509_NAME_oneline (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x806998D: ngx_http_ssl_verify_callback (ngx_event_openssl.c:308)
by 0x43A086C: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x43A15F8: X509_verify_cert (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x42AB922: ssl_verify_cert_chain (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x428C9F9: ssl3_get_client_certificate (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x4290381: ssl3_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x42A729C: SSL_do_handshake (in
/usr/lib/i686/cmov/libssl.so.0.9.8)

Igor, I hope that this debug output will help you to solve the problem

Thanks

6

Igor S. wrote:

Try the attached patch.

[crit] 28813#0: *1 SSL_do_handshake() failed (SSL: error:140880DC:SSL
routines:SSL3_GET_CERT_VERIFY:signature for non signing certificate)
while reading client request line, client: x.x.x.x, server: x.x.x.x
[notice] 28812#0: signal 17 (SIGCHLD) received
[alert] 28812#0: worker process 28813 exited on signal 11
[notice] 28812#0: start worker process 28817

7

On Sun, Apr 27, 2008 at 07:01:54PM +0200, Chavelle V. wrote:

by 0x439E9B2: X509_NAME_oneline (in 
by 0x42A78C9: SSL_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x806998D: ngx_http_ssl_verify_callback (ngx_event_openssl.c:308)

Igor, I hope that this debug output will help you to solve the problem

Try the attached patch.

8

On Sun, Apr 27, 2008 at 08:04:44PM +0200, Chavelle V. wrote:

Igor S. wrote:

Try the attached patch.

[crit] 28813#0: *1 SSL_do_handshake() failed (SSL: error:140880DC:SSL
routines:SSL3_GET_CERT_VERIFY:signature for non signing certificate)
while reading client request line, client: x.x.x.x, server: x.x.x.x
[notice] 28812#0: signal 17 (SIGCHLD) received
[alert] 28812#0: worker process 28813 exited on signal 11
[notice] 28812#0: start worker process 28817

Backout the previous patch and try the new one.

9

Igor S. wrote:

Backout the previous patch and try the new one.

Ok, all is good. Thanks for your disponibility.