Forum: NGINX nginx: worker process: malloc(): memory corruption: 0x0000000000b6bdb0 ***

hi
    on centos 6, nginx-1.2.2, nginx was compiled with:
--prefix=/usr/local/nginx --user=root --group=root 
--with-http_ssl_module
--with-ipv6 --with-pcre=/home/nginx/src/pcre-8.20
--with-openssl=/home/nginx/src/openssl-1.0.1c
--with-zlib=/home/nginx/src/zlib-1.2.7
--add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_subs_filter
--add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_gunzip
--add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_lvs_live
--add-module=/home/nginx/svn/nginx-1.2.2/src/add-on/nginx_sniper

some minutes later when  launch nginx, i get error(no coredump file, and
workprcess hungs)：

*** glibc detected *** nginx: worker process: malloc(): memory 
corruption:
0x0000000000b6bdb0 ***

pstark info:
[root@rs1 sbin]# pstack 30803
#0  0x0000003eb160cb1b in pthread_once () from /lib64/libpthread.so.0
#1  0x0000003eb0efe954 in backtrace () from /lib64/libc.so.6
#2  0x0000003eb0e707cb in __libc_message () from /lib64/libc.so.6
#3  0x0000003eb0e760e6 in malloc_printerr () from /lib64/libc.so.6
#4  0x0000003eb0e79b64 in _int_malloc () from /lib64/libc.so.6
#5  0x0000003eb0e7a5a6 in calloc () from /lib64/libc.so.6
#6  0x0000003eb0a0ad0f in _dl_new_object () from
/lib64/ld-linux-x86-64.so.2
#7  0x0000003eb0a0719e in _dl_map_object_from_fd () from
/lib64/ld-linux-x86-64.so.2
#8  0x0000003eb0a0835a in _dl_map_object () from
/lib64/ld-linux-x86-64.so.2
#9  0x0000003eb0a129b4 in dl_open_worker () from
/lib64/ld-linux-x86-64.so.2
#10 0x0000003eb0a0e196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#11 0x0000003eb0a1246a in _dl_open () from /lib64/ld-linux-x86-64.so.2
#12 0x0000003eb0f26300 in do_dlopen () from /lib64/libc.so.6
#13 0x0000003eb0a0e196 in _dl_catch_error () from
/lib64/ld-linux-x86-64.so.2
#14 0x0000003eb0f26457 in __libc_dlopen_mode () from /lib64/libc.so.6
#15 0x0000003eb0efe855 in init () from /lib64/libc.so.6
#16 0x0000003eb160cb23 in pthread_once () from /lib64/libpthread.so.0
#17 0x0000003eb0efe954 in backtrace () from /lib64/libc.so.6
#18 0x0000003eb0e707cb in __libc_message () from /lib64/libc.so.6
#19 0x0000003eb0e760e6 in malloc_printerr () from /lib64/libc.so.6
#20 0x0000003eb0e79b64 in _int_malloc () from /lib64/libc.so.6
#21 0x0000003eb0e7a911 in malloc () from /lib64/libc.so.6
#22 0x000000000041bb5e in ngx_alloc ()
#23 0x00000000004144ab in ngx_resolver_alloc ()
#24 0x0000000000414916 in ngx_resolver_calloc ()
#25 0x00000000004149af in ngx_resolve_start ()
#26 0x000000000043ece8 in ngx_http_upstream_init_request ()
#27 0x000000000043ee93 in ngx_http_upstream_init ()
#28 0x00000000004361c1 in ngx_http_read_client_request_body ()
#29 0x0000000000459973 in ngx_http_proxy_handler ()
#30 0x000000000042bd24 in ngx_http_core_content_phase ()
#31 0x00000000004269a3 in ngx_http_core_run_phases ()
#32 0x0000000000426a9e in ngx_http_handler ()
#33 0x0000000000430662 in ngx_http_process_request ()
#34 0x0000000000430d98 in ngx_http_process_request_headers ()
#35 0x0000000000431369 in ngx_http_process_request_line ()
#36 0x000000000042e6a6 in ngx_http_init_request ()
#37 0x000000000042e825 in ngx_http_keepalive_handler ()
#38 0x00000000004198f2 in ngx_event_process_posted ()
#39 0x00000000004197c2 in ngx_process_events_and_timers ()
#40 0x000000000041f410 in ngx_worker_process_cycle ()
#41 0x000000000041dc58 in ngx_spawn_process ()
#42 0x000000000041eadd in ngx_start_worker_processes ()
#43 0x000000000041f931 in ngx_master_process_cycle ()
#44 0x0000000000404e23 in main ()
[root@rs1 sbin]# pstack 30804
#0  0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x0000000000420562 in ngx_epoll_process_events ()
#2  0x000000000041975b in ngx_process_events_and_timers ()
#3  0x000000000041f410 in ngx_worker_process_cycle ()
#4  0x000000000041dc58 in ngx_spawn_process ()
#5  0x000000000041eadd in ngx_start_worker_processes ()
#6  0x000000000041f931 in ngx_master_process_cycle ()
#7  0x0000000000404e23 in main ()
[root@rs1 sbin]# pstack 30805
#0  0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x0000000000420562 in ngx_epoll_process_events ()
#2  0x000000000041975b in ngx_process_events_and_timers ()
#3  0x000000000041f410 in ngx_worker_process_cycle ()
#4  0x000000000041dc58 in ngx_spawn_process ()
#5  0x000000000041eadd in ngx_start_worker_processes ()
#6  0x000000000041f931 in ngx_master_process_cycle ()
#7  0x0000000000404e23 in main ()
[root@rs1 sbin]# pstack 30806
#0  0x0000003eb0ee8ee3 in __epoll_wait_nocancel () from /lib64/libc.so.6
#1  0x0000000000420562 in ngx_epoll_process_events ()
#2  0x000000000041975b in ngx_process_events_and_timers ()
#3  0x000000000041f410 in ngx_worker_process_cycle ()
#4  0x000000000041dc58 in ngx_spawn_process ()
#5  0x000000000041eadd in ngx_start_worker_processes ()
#6  0x000000000041f931 in ngx_master_process_cycle ()
#7  0x0000000000404e23 in main ()




[root@rs1 sbin]# ps aux | grep nginx
root     30802  0.0  0.0  17904   844 ?        Ss   08:15   0:00 nginx:
master process ./nginx
root     30803  0.0  0.0  20524  4200 ?        S    08:15   0:00 nginx:
worker process
root     30804  0.0  0.0  18876  2544 ?        S    08:15   0:00 nginx:
worker process
root     30805  0.0  0.0  19320  2972 ?        S    08:15   0:00 nginx:
worker process
root     30806  0.0  0.0  18716  1884 ?        S    08:15   0:00 nginx:
worker process
root     30957  0.0  0.0 103244   860 pts/1    S+   08:28   0:00 grep 
nginx


any ideas for fix it? thanks in advance.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,237393,237393#msg-237393

Hello!

On Fri, Mar 15, 2013 at 04:39:03AM -0400, honwel wrote:

>
> some minutes later when  launch nginx, i get error(no coredump file, and
> workprcess hungs)：

Have you tried reproducing the problem without 3rd party modules
compiled in?

You may also want to upgrade nginx to at least nginx 1.2.7, 1.2.2
is rather old.

--
Maxim Dounin
http://nginx.org/en/donation.html

thanks, i will try as you mention and report it on forum.

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,237393,237397#msg-237397

i use valgrind to check memory leak, and have detected some error:

==2243== Invalid write of size 1
==2243==    at 0x4A08088: memcpy (mc_replace_strmem.c:628)
==2243==    by 0x4448C9: ngx_http_proxy_subs_headers
(ngx_http_proxy_subs_filter.c:149)
==2243==    by 0x45B2FB: ngx_http_proxy_create_request
(ngx_http_proxy_module.c:1235)
==2243==    by 0x43EA7E: ngx_http_upstream_init_request
(ngx_http_upstream.c:505)
==2243==    by 0x43EE92: ngx_http_upstream_init 
(ngx_http_upstream.c:446)
==2243==    by 0x4361C0: ngx_http_read_client_request_body
(ngx_http_request_body.c:59)
==2243==    by 0x459972: ngx_http_proxy_handler
(ngx_http_proxy_module.c:703)
==2243==    by 0x42BD23: ngx_http_core_content_phase
(ngx_http_core_module.c:1396)
==2243==    by 0x4269A2: ngx_http_core_run_phases
(ngx_http_core_module.c:877)
==2243==    by 0x426A9D: ngx_http_handler (ngx_http_core_module.c:860)
==2243==    by 0x430661: ngx_http_process_request 
(ngx_http_request.c:1874)
==2243==    by 0x430D97: ngx_http_process_request_headers
(ngx_http_request.c:1318)
==2243==  Address 0x5a1f29a is not stack'd, malloc'd or (recently) 
free'd
==2243==
==2243== Invalid write of size 8
==2243==    at 0x4A080B3: memcpy (mc_replace_strmem.c:628)
==2243==    by 0x4448C9: ngx_http_proxy_subs_headers
(ngx_http_proxy_subs_filter.c:149)
==2243==    by 0x45B2FB: ngx_http_proxy_create_request
(ngx_http_proxy_module.c:1235)
==2243==    by 0x43EA7E: ngx_http_upstream_init_request
(ngx_http_upstream.c:505)
==2243==    by 0x43EE92: ngx_http_upstream_init 
(ngx_http_upstream.c:446)
==2243==    by 0x4361C0: ngx_http_read_client_request_body
(ngx_http_request_body.c:59)
==2243==    by 0x459972: ngx_http_proxy_handler
(ngx_http_proxy_module.c:703)
==2243==    by 0x42BD23: ngx_http_core_content_phase
(ngx_http_core_module.c:1396)
==2243==    by 0x4269A2: ngx_http_core_run_phases
(ngx_http_core_module.c:877)
==2243==    by 0x426A9D: ngx_http_handler (ngx_http_core_module.c:860)
==2243==    by 0x430661: ngx_http_process_request 
(ngx_http_request.c:1874)


due to ngx_copy() out of bound, and caused by my code.  so, i modify the
corrspongding code, it's running ok until now.

thanks for Maxim Dounin !

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,237393,237778#msg-237778