Worker process and memory leak?

Hi all,

For testing, I compile nginx with following options :
./configure --with-http_ssl_module --without-http_charset_module
–without-http_
ssi_module --without-http_auth_basic_module
–without-http_autoindex_module --wi
thout-http_geo_module --without-http_map_module
–without-http_limit_zone_module
–without-http_empty_gif_module --without-http_browser_module
–without-http_up
stream_ip_hash_module --without-http_gzip_module
–without-http_userid_module –
without-http_access_module --without-http_referer_module
–without-http_rewrite_
module --without-http_proxy_module --without-http_fastcgi_module
–without-http_
memcached_module --without-mail_pop3_module --without-mail_imap_module
–without
-mail_smtp_module

nginx.conf:

worker_processes 1;
error_log logs/error.log info;

events {
worker_connections 1024;
use epoll;
}

http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;

server {
listen 443;
server_name myip;

client_max_body_size 100M;
ssl on;
ssl_verify_client on;
ssl_certificate server.pem;
ssl_certificate_key server.key;
ssl_client_certificate ca.pem;
ssl_session_timeout 5m;

    ssl_protocols  TLSv1;
    ssl_ciphers

ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP;
ssl_prefer_server_ciphers on;

    location / {
        root   html;
        index  index.html index.htm;
    }
}

}

Without any request, the state of worker process is:
15027 nobody 18 0 4104 944 392 S 0 0.1 0:00.00 nginx

When I flood it with 50 bad request (GET /test.test => 404) the worker
process becomes:
15027 nobody 16 0 4584 2300 1308 S 0 0.2 0:04.99 nginx

With another 50 bad request:
15027 nobody 16 0 5012 2668 1308 S 0 0.3 0:09.94 nginx

etc…

Nginx is it concerned? Hence the problem can come?

Thanks in advance

After many test, I found the memory leak cause. It happens when
ssl_verify_client is activate. Nginx doesn’t seem have memory leak when
this parameter is off.

Chavelle V. wrote:

ssl_verify_client on;

So I run valgrind to have more information and I found this :

340 bytes in 2 blocks are possibly lost in loss record 4 of 10
at 0x4022AB8: malloc (vg_replace_malloc.c:207)
by 0x430448D: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4304AEE: CRYPTO_malloc (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x433A256: BN_MONT_CTX_new (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x433A362: BN_MONT_CTX_set_locked (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x434E886: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x435035D: RSA_public_decrypt (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4351149: RSA_verify (in /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x428D005: ssl3_get_cert_verify (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x4290268: ssl3_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x42A729C: SSL_do_handshake (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x8069048: ngx_ssl_handshake (ngx_event_openssl.c:408)

280,022 (57,932 direct, 222,090 indirect) bytes in 249 blocks are
definitely lost in loss record 9 of 10
at 0x4022AB8: malloc (vg_replace_malloc.c:207)
by 0x430448D: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4304AEE: CRYPTO_malloc (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x438BD84: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x438E9B4: ASN1_item_ex_d2i (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x438F124: ASN1_item_d2i (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4385CD4: d2i_X509 (in /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x428C8D3: ssl3_get_client_certificate (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x4290381: ssl3_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x42A729C: SSL_do_handshake (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x8069048: ngx_ssl_handshake (ngx_event_openssl.c:408)
by 0x8069404: ngx_ssl_handshake_handler (ngx_event_openssl.c:540)

I have no idea how to solve this problem, and if it is possible to free
pointer in nginx. I don’t found a lot of help on google.

For information : http://bugs.mysql.com/bug.php?id=14780

Thanks again

On Wed, Apr 23, 2008 at 06:22:04PM +0200, Chavelle V. wrote:

After many test, I found the memory leak cause. It happens when
ssl_verify_client is activate. Nginx doesn’t seem have memory leak when
this parameter is off.

Thank you for your inverstigation.
Could you test the attached patch ?

Igor S. wrote:

Thank you for your inverstigation.
Could you test the attached patch ?

The patch works fine, it resolves the problem.

Thanks a lot !!!

Visibly, there is another memory leak when ssl_verify_client is also
activate.

21,440 bytes in 80 blocks are definitely lost in loss record 6 of 8
at 0x4022AB8: malloc (vg_replace_malloc.c:207)
by 0x430448D: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4304AEE: CRYPTO_malloc (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x436113C: BUF_MEM_grow (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x439E9B2: X509_NAME_oneline (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x8069962: ngx_http_ssl_verify_callback (ngx_event_openssl.c:305)
by 0x43A086C: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x43A15F8: X509_verify_cert (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x42AB922: ssl_verify_cert_chain (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x428C9F9: ssl3_get_client_certificate (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x4290381: ssl3_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x42A78C9: SSL_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)

268 bytes in 1 blocks are possibly lost in loss record 5 of 9
at 0x4022AB8: malloc (vg_replace_malloc.c:207)
by 0x430448D: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x4304AEE: CRYPTO_malloc (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x436113C: BUF_MEM_grow (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x439E9B2: X509_NAME_oneline (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x806998D: ngx_http_ssl_verify_callback (ngx_event_openssl.c:308)
by 0x43A086C: (within /usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x43A15F8: X509_verify_cert (in
/usr/lib/i686/cmov/libcrypto.so.0.9.8)
by 0x42AB922: ssl_verify_cert_chain (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x428C9F9: ssl3_get_client_certificate (in
/usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x4290381: ssl3_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x42A729C: SSL_do_handshake (in
/usr/lib/i686/cmov/libssl.so.0.9.8)

Igor, I hope that this debug output will help you to solve the problem

Thanks

Igor S. wrote:

Try the attached patch.

[crit] 28813#0: *1 SSL_do_handshake() failed (SSL: error:140880DC:SSL
routines:SSL3_GET_CERT_VERIFY:signature for non signing certificate)
while reading client request line, client: x.x.x.x, server: x.x.x.x
[notice] 28812#0: signal 17 (SIGCHLD) received
[alert] 28812#0: worker process 28813 exited on signal 11
[notice] 28812#0: start worker process 28817

On Sun, Apr 27, 2008 at 07:01:54PM +0200, Chavelle V. wrote:

by 0x439E9B2: X509_NAME_oneline (in 
by 0x42A78C9: SSL_accept (in /usr/lib/i686/cmov/libssl.so.0.9.8)
by 0x806998D: ngx_http_ssl_verify_callback (ngx_event_openssl.c:308)

Igor, I hope that this debug output will help you to solve the problem

Try the attached patch.

On Sun, Apr 27, 2008 at 08:04:44PM +0200, Chavelle V. wrote:

Igor S. wrote:

Try the attached patch.

[crit] 28813#0: *1 SSL_do_handshake() failed (SSL: error:140880DC:SSL
routines:SSL3_GET_CERT_VERIFY:signature for non signing certificate)
while reading client request line, client: x.x.x.x, server: x.x.x.x
[notice] 28812#0: signal 17 (SIGCHLD) received
[alert] 28812#0: worker process 28813 exited on signal 11
[notice] 28812#0: start worker process 28817

Backout the previous patch and try the new one.

Igor S. wrote:

Backout the previous patch and try the new one.

Ok, all is good. Thanks for your disponibility.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs