Nginx: worker process: malloc(): memory corruption: 0x0000000000b6bdb0 ***

hi
on centos 6, nginx-1.2.2, nginx was compiled with:
–prefix=/usr/local/nginx --user=root --group=root
–with-http_ssl_module
–with-ipv6 --with-pcre=/home/nginx/src/pcre-8.20
–with-openssl=/home/nginx/src/openssl-1.0.1c
–with-zlib=/home/nginx/src/zlib-1.2.7
–add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_subs_filter
–add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_gunzip
–add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_lvs_live
–add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_sniper

some minutes later when launch nginx, i get error(no coredump file, and
workprcess hungs):

*** glibc detected *** nginx: worker process: malloc(): memory
corruption:
0x0000000000b6bdb0 ***

pstark info:
[root@rs1 sbin]# pstack 30803
#0 0x0000003eb160cb1b in pthread_once () from /lib64/libpthread.so.0
#1 0x0000003eb0efe954 in backtrace () from /lib64/libc.so.6
#2 0x0000003eb0e707cb in __libc_message () from /lib64/libc.so.6
#3 0x0000003eb0e760e6 in malloc_printerr () from /lib64/libc.so.6
#4 0x0000003eb0e79b64 in _int_malloc () from /lib64/libc.so.6
#5 0x0000003eb0e7a5a6 in calloc () from /lib64/libc.so.6
#6 0x0000003eb0a0ad0f in _dl_new_object () from
/lib64/ld-linux-x86-64.so.2
#7 0x0000003eb0a0719e in _dl_map_object_from_fd () from
/lib64/ld-linux-x86-64.so.2
#8 0x0000003eb0a0835a in _dl_map_object () from
/lib64/ld-linux-x86-64.so.2
#9 0x0000003eb0a129b4 in dl_open_worker () from
/lib64/ld-linux-x86-64.so.2
#10 0x0000003eb0a0e196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#11 0x0000003eb0a1246a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x0000003eb0f26300 in do_dlopen () from /lib64/libc.so.6
#13 0x0000003eb0a0e196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#14 0x0000003eb0f26457 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x0000003eb0efe855 in init () from /lib64/libc.so.6
#16 0x0000003eb160cb23 in pthread_once () from /lib64/libpthread.so.0
#17 0x0000003eb0efe954 in backtrace () from /lib64/libc.so.6
#18 0x0000003eb0e707cb in __libc_message () from /lib64/libc.so.6
#19 0x0000003eb0e760e6 in malloc_printerr () from /lib64/libc.so.6
#20 0x0000003eb0e79b64 in _int_malloc () from /lib64/libc.so.6
#21 0x0000003eb0e7a911 in malloc () from /lib64/libc.so.6
#22 0x000000000041bb5e in ngx_alloc ()
#23 0x00000000004144ab in ngx_resolver_alloc ()
#24 0x0000000000414916 in ngx_resolver_calloc ()
#25 0x00000000004149af in ngx_resolve_start ()
#26 0x000000000043ece8 in ngx_http_upstream_init_request ()
#27 0x000000000043ee93 in ngx_http_upstream_init ()
#28 0x00000000004361c1 in ngx_http_read_client_request_body ()
#29 0x0000000000459973 in ngx_http_proxy_handler ()
#30 0x000000000042bd24 in ngx_http_core_content_phase ()
#31 0x00000000004269a3 in ngx_http_core_run_phases ()
#32 0x0000000000426a9e in ngx_http_handler ()
#33 0x0000000000430662 in ngx_http_process_request ()
#34 0x0000000000430d98 in ngx_http_process_request_headers ()
#35 0x0000000000431369 in ngx_http_process_request_line ()
#36 0x000000000042e6a6 in ngx_http_init_request ()
#37 0x000000000042e825 in ngx_http_keepalive_handler ()
#38 0x00000000004198f2 in ngx_event_process_posted ()
#39 0x00000000004197c2 in ngx_process_events_and_timers ()
#40 0x000000000041f410 in ngx_worker_process_cycle ()
#41 0x000000000041dc58 in ngx_spawn_process ()
#42 0x000000000041eadd in ngx_start_worker_processes ()
#43 0x000000000041f931 in ngx_master_process_cycle ()
#44 0x0000000000404e23 in main ()
[root@rs1 sbin]# pstack 30804
#0 0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x0000000000420562 in ngx_epoll_process_events ()
#2 0x000000000041975b in ngx_process_events_and_timers ()
#3 0x000000000041f410 in ngx_worker_process_cycle ()
#4 0x000000000041dc58 in ngx_spawn_process ()
#5 0x000000000041eadd in ngx_start_worker_processes ()
#6 0x000000000041f931 in ngx_master_process_cycle ()
#7 0x0000000000404e23 in main ()
[root@rs1 sbin]# pstack 30805
#0 0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x0000000000420562 in ngx_epoll_process_events ()
#2 0x000000000041975b in ngx_process_events_and_timers ()
#3 0x000000000041f410 in ngx_worker_process_cycle ()
#4 0x000000000041dc58 in ngx_spawn_process ()
#5 0x000000000041eadd in ngx_start_worker_processes ()
#6 0x000000000041f931 in ngx_master_process_cycle ()
#7 0x0000000000404e23 in main ()
[root@rs1 sbin]# pstack 30806
#0 0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1 0x0000000000420562 in ngx_epoll_process_events ()
#2 0x000000000041975b in ngx_process_events_and_timers ()
#3 0x000000000041f410 in ngx_worker_process_cycle ()
#4 0x000000000041dc58 in ngx_spawn_process ()
#5 0x000000000041eadd in ngx_start_worker_processes ()
#6 0x000000000041f931 in ngx_master_process_cycle ()
#7 0x0000000000404e23 in main ()

[root@rs1 sbin]# ps aux | grep nginx
root 30802 0.0 0.0 17904 844 ? Ss 08:15 0:00 nginx:
master process ./nginx
root 30803 0.0 0.0 20524 4200 ? S 08:15 0:00 nginx:
worker process
root 30804 0.0 0.0 18876 2544 ? S 08:15 0:00 nginx:
worker process
root 30805 0.0 0.0 19320 2972 ? S 08:15 0:00 nginx:
worker process
root 30806 0.0 0.0 18716 1884 ? S 08:15 0:00 nginx:
worker process
root 30957 0.0 0.0 103244 860 pts/1 S+ 08:28 0:00 grep
nginx

any ideas for fix it? thanks in advance.

Posted at Nginx Forum:

Hello!

On Fri, Mar 15, 2013 at 04:39:03AM -0400, honwel wrote:

some minutes later when launch nginx, i get error(no coredump file, and
workprcess hungs):

Have you tried reproducing the problem without 3rd party modules
compiled in?

You may also want to upgrade nginx to at least nginx 1.2.7, 1.2.2
is rather old.

–
Maxim D.
http://nginx.org/en/donation.html

i use valgrind to check memory leak, and have detected some error:

==2243== Invalid write of size 1
==2243== at 0x4A08088: memcpy (mc_replace_strmem.c:628)
==2243== by 0x4448C9: ngx_http_proxy_subs_headers
(ngx_http_proxy_subs_filter.c:149)
==2243== by 0x45B2FB: ngx_http_proxy_create_request
(ngx_http_proxy_module.c:1235)
==2243== by 0x43EA7E: ngx_http_upstream_init_request
(ngx_http_upstream.c:505)
==2243== by 0x43EE92: ngx_http_upstream_init
(ngx_http_upstream.c:446)
==2243== by 0x4361C0: ngx_http_read_client_request_body
(ngx_http_request_body.c:59)
==2243== by 0x459972: ngx_http_proxy_handler
(ngx_http_proxy_module.c:703)
==2243== by 0x42BD23: ngx_http_core_content_phase
(ngx_http_core_module.c:1396)
==2243== by 0x4269A2: ngx_http_core_run_phases
(ngx_http_core_module.c:877)
==2243== by 0x426A9D: ngx_http_handler (ngx_http_core_module.c:860)
==2243== by 0x430661: ngx_http_process_request
(ngx_http_request.c:1874)
==2243== by 0x430D97: ngx_http_process_request_headers
(ngx_http_request.c:1318)
==2243== Address 0x5a1f29a is not stack’d, malloc’d or (recently)
free’d
==2243==
==2243== Invalid write of size 8
==2243== at 0x4A080B3: memcpy (mc_replace_strmem.c:628)
==2243== by 0x4448C9: ngx_http_proxy_subs_headers
(ngx_http_proxy_subs_filter.c:149)
==2243== by 0x45B2FB: ngx_http_proxy_create_request
(ngx_http_proxy_module.c:1235)
==2243== by 0x43EA7E: ngx_http_upstream_init_request
(ngx_http_upstream.c:505)
==2243== by 0x43EE92: ngx_http_upstream_init
(ngx_http_upstream.c:446)
==2243== by 0x4361C0: ngx_http_read_client_request_body
(ngx_http_request_body.c:59)
==2243== by 0x459972: ngx_http_proxy_handler
(ngx_http_proxy_module.c:703)
==2243== by 0x42BD23: ngx_http_core_content_phase
(ngx_http_core_module.c:1396)
==2243== by 0x4269A2: ngx_http_core_run_phases
(ngx_http_core_module.c:877)
==2243== by 0x426A9D: ngx_http_handler (ngx_http_core_module.c:860)
==2243== by 0x430661: ngx_http_process_request
(ngx_http_request.c:1874)

due to ngx_copy() out of bound, and caused by my code. so, i modify the
corrspongding code, it’s running ok until now.

thanks for Maxim D. !

Posted at Nginx Forum:

thanks, i will try as you mention and report it on forum.

Posted at Nginx Forum: