Nginx: worker process: malloc(): memory corruption: 0x0000000000b6bdb0 ***

hi
on centos 6, nginx-1.2.2, nginx was compiled with:
–prefix=/usr/local/nginx --user=root --group=root
–with-http_ssl_module
–with-ipv6 --with-pcre=/home/nginx/src/pcre-8.20
–with-openssl=/home/nginx/src/openssl-1.0.1c
–with-zlib=/home/nginx/src/zlib-1.2.7
–add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_subs_filter
–add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_gunzip
–add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_lvs_live
–add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_sniper

some minutes later when launch nginx, i get error(no coredump file, and
workprcess hungs):

*** glibc detected *** nginx: worker process: malloc(): memory
corruption:
0x0000000000b6bdb0 ***

pstark info:
[[email protected] sbin]# pstack 30803
#0 0x0000003eb160cb1b in pthread_once () from /lib64/libpthread.so.0
#1 0x0000003eb0efe954 in backtrace () from /lib64/libc.so.6
#2 0x0000003eb0e707cb in __libc_message () from /lib64/libc.so.6
#3 0x0000003eb0e760e6 in malloc_printerr () from /lib64/libc.so.6
#4 0x0000003eb0e79b64 in _int_malloc () from /lib64/libc.so.6
#5 0x0000003eb0e7a5a6 in calloc () from /lib64/libc.so.6
#6 0x0000003eb0a0ad0f in _dl_new_object () from
/lib64/ld-linux-x86-64.so.2
#7 0x0000003eb0a0719e in _dl_map_object_from_fd () from
/lib64/ld-linux-x86-64.so.2
#8 0x0000003eb0a0835a in _dl_map_object () from
/lib64/ld-linux-x86-64.so.2
#9 0x0000003eb0a129b4 in dl_open_worker () from
/lib64/ld-linux-x86-64.so.2
#10 0x0000003eb0a0e196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#11 0x0000003eb0a1246a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x0000003eb0f26300 in do_dlopen () from /lib64/libc.so.6
#13 0x0000003eb0a0e196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#14 0x0000003eb0f26457 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x0000003eb0efe855 in init () from /lib64/libc.so.6
#16 0x0000003eb160cb23 in pthread_once () from /lib64/libpthread.so.0
#17 0x0000003eb0efe954 in backtrace () from /lib64/libc.so.6
#18 0x0000003eb0e707cb in __libc_message () from /lib64/libc.so.6
#19 0x0000003eb0e760e6 in malloc_printerr () from /lib64/libc.so.6
#20 0x0000003eb0e79b64 in _int_malloc () from /lib64/libc.so.6
#21 0x0000003eb0e7a911 in malloc () from /lib64/libc.so.6
#22 0x000000000041bb5e in ngx_alloc ()
#23 0x00000000004144ab in ngx_resolver_alloc ()
#24 0x0000000000414916 in ngx_resolver_calloc ()
#25 0x00000000004149af in ngx_resolve_start ()
#26 0x000000000043ece8 in ngx_http_upstream_init_request ()
#27 0x000000000043ee93 in ngx_http_upstream_init ()
#28 0x00000000004361c1 in ngx_http_read_client_request_body ()
#29 0x0000000000459973 in ngx_http_proxy_handler ()
#30 0x000000000042bd24 in ngx_http_core_content_phase ()
#31 0x00000000004269a3 in ngx_http_core_run_phases ()
#32 0x0000000000426a9e in ngx_http_handler ()
#33 0x0000000000430662 in ngx_http_process_request ()
#34 0x0000000000430d98 in ngx_http_process_request_headers ()
#35 0x0000000000431369 in ngx_http_process_request_line ()
#36 0x000000000042e6a6 in ngx_http_init_request ()
#37 0x000000000042e825 in ngx_http_keepalive_handler ()
#38 0x00000000004198f2 in ngx_event_process_posted ()
#39 0x00000000004197c2 in ngx_process_events_and_timers ()
#40 0x000000000041f410 in ngx_worker_process_cycle ()
#41 0x000000000041dc58 in ngx_spawn_process ()
#42 0x000000000041eadd in ngx_start_worker_processes ()
#43 0x000000000041f931 in ngx_master_process_cycle ()
#44 0x0000000000404e23 in main ()
[[email protected] sbin]# pstack 30804
#0 0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x0000000000420562 in ngx_epoll_process_events ()
#2 0x000000000041975b in ngx_process_events_and_timers ()
#3 0x000000000041f410 in ngx_worker_process_cycle ()
#4 0x000000000041dc58 in ngx_spawn_process ()
#5 0x000000000041eadd in ngx_start_worker_processes ()
#6 0x000000000041f931 in ngx_master_process_cycle ()
#7 0x0000000000404e23 in main ()
[[email protected] sbin]# pstack 30805
#0 0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x0000000000420562 in ngx_epoll_process_events ()
#2 0x000000000041975b in ngx_process_events_and_timers ()
#3 0x000000000041f410 in ngx_worker_process_cycle ()
#4 0x000000000041dc58 in ngx_spawn_process ()
#5 0x000000000041eadd in ngx_start_worker_processes ()
#6 0x000000000041f931 in ngx_master_process_cycle ()
#7 0x0000000000404e23 in main ()
[[email protected] sbin]# pstack 30806
#0 0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x0000000000420562 in ngx_epoll_process_events ()
#2 0x000000000041975b in ngx_process_events_and_timers ()
#3 0x000000000041f410 in ngx_worker_process_cycle ()
#4 0x000000000041dc58 in ngx_spawn_process ()
#5 0x000000000041eadd in ngx_start_worker_processes ()
#6 0x000000000041f931 in ngx_master_process_cycle ()
#7 0x0000000000404e23 in main ()

[[email protected] sbin]# ps aux | grep nginx
root 30802 0.0 0.0 17904 844 ? Ss 08:15 0:00 nginx:
master process ./nginx
root 30803 0.0 0.0 20524 4200 ? S 08:15 0:00 nginx:
worker process
root 30804 0.0 0.0 18876 2544 ? S 08:15 0:00 nginx:
worker process
root 30805 0.0 0.0 19320 2972 ? S 08:15 0:00 nginx:
worker process
root 30806 0.0 0.0 18716 1884 ? S 08:15 0:00 nginx:
worker process
root 30957 0.0 0.0 103244 860 pts/1 S+ 08:28 0:00 grep
nginx

any ideas for fix it? thanks in advance.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,237393,237393#msg-237393

Hello!

On Fri, Mar 15, 2013 at 04:39:03AM -0400, honwel wrote:

some minutes later when launch nginx, i get error(no coredump file, and
workprcess hungs):

Have you tried reproducing the problem without 3rd party modules
compiled in?

You may also want to upgrade nginx to at least nginx 1.2.7, 1.2.2
is rather old.


Maxim D.
http://nginx.org/en/donation.html

i use valgrind to check memory leak, and have detected some error:

==2243== Invalid write of size 1
==2243== at 0x4A08088: memcpy (mc_replace_strmem.c:628)
==2243== by 0x4448C9: ngx_http_proxy_subs_headers
(ngx_http_proxy_subs_filter.c:149)
==2243== by 0x45B2FB: ngx_http_proxy_create_request
(ngx_http_proxy_module.c:1235)
==2243== by 0x43EA7E: ngx_http_upstream_init_request
(ngx_http_upstream.c:505)
==2243== by 0x43EE92: ngx_http_upstream_init
(ngx_http_upstream.c:446)
==2243== by 0x4361C0: ngx_http_read_client_request_body
(ngx_http_request_body.c:59)
==2243== by 0x459972: ngx_http_proxy_handler
(ngx_http_proxy_module.c:703)
==2243== by 0x42BD23: ngx_http_core_content_phase
(ngx_http_core_module.c:1396)
==2243== by 0x4269A2: ngx_http_core_run_phases
(ngx_http_core_module.c:877)
==2243== by 0x426A9D: ngx_http_handler (ngx_http_core_module.c:860)
==2243== by 0x430661: ngx_http_process_request
(ngx_http_request.c:1874)
==2243== by 0x430D97: ngx_http_process_request_headers
(ngx_http_request.c:1318)
==2243== Address 0x5a1f29a is not stack’d, malloc’d or (recently)
free’d
==2243==
==2243== Invalid write of size 8
==2243== at 0x4A080B3: memcpy (mc_replace_strmem.c:628)
==2243== by 0x4448C9: ngx_http_proxy_subs_headers
(ngx_http_proxy_subs_filter.c:149)
==2243== by 0x45B2FB: ngx_http_proxy_create_request
(ngx_http_proxy_module.c:1235)
==2243== by 0x43EA7E: ngx_http_upstream_init_request
(ngx_http_upstream.c:505)
==2243== by 0x43EE92: ngx_http_upstream_init
(ngx_http_upstream.c:446)
==2243== by 0x4361C0: ngx_http_read_client_request_body
(ngx_http_request_body.c:59)
==2243== by 0x459972: ngx_http_proxy_handler
(ngx_http_proxy_module.c:703)
==2243== by 0x42BD23: ngx_http_core_content_phase
(ngx_http_core_module.c:1396)
==2243== by 0x4269A2: ngx_http_core_run_phases
(ngx_http_core_module.c:877)
==2243== by 0x426A9D: ngx_http_handler (ngx_http_core_module.c:860)
==2243== by 0x430661: ngx_http_process_request
(ngx_http_request.c:1874)

due to ngx_copy() out of bound, and caused by my code. so, i modify the
corrspongding code, it’s running ok until now.

thanks for Maxim D. !

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,237393,237778#msg-237778

thanks, i will try as you mention and report it on forum.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,237393,237397#msg-237397

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs