Forum: NGINX nginx crash only when using Chromium (in ubuntu)

Posted by gadh (Guest)
on 2012-12-30 17:10
(Received via mailing list)
i could not find the cause that only when using Chromium i get a crash 
but
when using Firefox i never don't.
some hints to the nginx experts that might help:
1. i use my handler module + filter module. (when module is disabled - 
no
crash)
2. i use C++ code in shared lib and sometimes the crash is in the c++ 
object
deconstructor . the object is allocated on the stack (not ptr, just 
regular
declaration like: obj_t obj1) and freed automatically and end of 
function.
3. i attach here the headers of FF / CHR browsers.
4. when using valgrind - i get some warnings (see below) but never 
crash,
even in CHR
5. the nginx runs on vurtual machine (centos 6.3) under ubuntu 12.10. 
the
browser runs on the ubuntu.
6. the response handler runs when subrequest returns from an upstream
server, then the handler continues and goes to the filter module.
7. sometimes when using palloc i got alignment errors so i used pnalloc. 
is
it the source of the bug ? when to use palloc and when to use pnalloc ? 
(see
below the function that uses pnalloc)
8. when restarting nginx and doing CTRL+F5 in CHR browser (right after 
the
previous crash) - its easy to get another crash again with the same 
stack
trace, while when browsing to anbother page - it takes time to reproduce 
the
crash.

 ===============

Thread [1] (Suspended: Signal 'SIGABRT' received. Description: Aborted.)
  15 raise()  0x00007ffff64e18a5
  14 abort()  0x00007ffff64e3085
  13 __libc_message()  0x00007ffff651efe7
  12 malloc_printerr()  0x00007ffff6524916
  11 _int_free()  0x00007ffff6527443
  10 ngx_destroy_pool() ngx_palloc.c:87 0x0000000000406a22
  9 ngx_http_free_request() ngx_http_request.c:3081 0x000000000044dbfb
  8 ngx_http_close_request() ngx_http_request.c:3006 0x000000000044d9b3
  7 ngx_http_terminate_handler() ngx_http_request.c:2176 
0x000000000044bc38
  6 ngx_http_run_posted_requests() ngx_http_request.c:1903
0x000000000044b1ad
  5 ngx_http_request_handler() ngx_http_request.c:1869 
0x000000000044b0b6
  4 ngx_epoll_process_events() ngx_epoll_module.c:683 0x00000000004377d6
  3 ngx_process_events_and_timers() ngx_event.c:247 0x00000000004281f4
  2 ngx_single_process_cycle() ngx_process_cycle.c:316 
0x0000000000434442
  1 main() nginx.c:409 0x0000000000403cdc

valgrind:
==27496==  Address 0x90c0b2d is 29 bytes inside a block of size 3,366
free'd
==27496==    at 0x4C2645F: operator delete(void*) 
(vg_replace_malloc.c:387)
==27496==    by 0x59B73AD: SBB::ResponseBean::~ResponseBean() (in
/usr/local/lib/libClientAPI-C-Lib.so)
==27496==    by 0x57ABB04: ngx_sbb_med_handle_va_response (in
/usr/local/lib/libngx_sbb_mediator.so)
==27496==    by 0x4A933D: ngx_sbb_va_response_handler
(ngx_sbb_module.c:274)
==27496==    by 0x4AA372: ngx_sbb_post_subrequest_handler
(ngx_sbb_mod_utils.c:89)
==27496==    by 0x44B3C0: ngx_http_finalize_request
(ngx_http_request.c:1961)
==27496==    by 0x465407: ngx_http_upstream_finalize_request
(ngx_http_upstream.c:3095)


CHR headers:
GET /index.php?cat=1&pag=1&det=108 HTTP/1.1
Host: ---
Connection: keep-alive
Cache-Control: max-age=0
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.4 (KHTML, 
like
Gecko) Ubuntu/12.10 Chromium/22.0.1229.94 Chrome/22.0.1229.94 
Safari/537.4
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: http://yellowmockup.com/index.php?cat=1
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8,he;q=0.6
Accept-Charset: UTF-8,*;q=0.5
Cookie: adOtr=4aYP5; PRLST=Ya; 
UTGv2=h4a59e6b096ada50ad0a1243f0549366c032;
x-autozoom=150f; SPSI=56aa48be644d6ac8ccec5dd82ade576d


FF headers:
GET /index.php?cat=1&pag=1&det=108 HTTP/1.1
Host: ---
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:16.0) 
Gecko/20100101
Firefox/16.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cookie: UTGv2=h430c577bc94965b18d99cd502407af14a80;
SPSI=63c40df4be7823f2acbc8e966a8817df; PRLST=zi/Jv/DT; adOtr=04Hd6
Pragma: no-cache
Cache-Control: no-cache

another crash dump:
Thread [1] (Suspended: Signal 'SIGSEGV' received. Description: 
Segmentation
fault.)
  16 memcpy()  0x00007ffff65381ab
  15 sbb_strncpy() ngx_sbb_utils.c:12 0x00000000004a9e5f
  14 ngx_sbb_utils_str2char() ngx_sbb_mod_utils.c:253 0x00000000004aaab7
  13 ngx_sbb_med_prepare_va_request()  0x00007ffff725d7b4
  12 ngx_sbb_handler() ngx_sbb_module.c:229 0x00000000004a913d
  11 ngx_http_core_rewrite_phase() ngx_http_core_module.c:931
0x000000000043d2a1
  10 ngx_http_core_run_phases() ngx_http_core_module.c:877
0x000000000043d103
  9 ngx_http_handler() ngx_http_core_module.c:860 0x000000000043d07a
  8 ngx_http_process_request() ngx_http_request.c:1687 
0x000000000044ac51
  7 ngx_http_process_request_headers() ngx_http_request.c:1135
0x0000000000449809
  6 ngx_http_process_request_line() ngx_http_request.c:933
0x0000000000448fbe
  5 ngx_http_init_request() ngx_http_request.c:519 0x000000000044873f
  4 ngx_epoll_process_events() ngx_epoll_module.c:683 0x00000000004377d6
  3 ngx_process_events_and_timers() ngx_event.c:247 0x00000000004281f4
  2 ngx_single_process_cycle() ngx_process_cycle.c:316 
0x0000000000434442
  1 main() nginx.c:409 0x0000000000403cdc
=============

// copies exactly n bytes from src to dest, then adds null in n+1 (alloc 
dst
to n+1 first !)
u_char * sbb_strncpy(u_char *dst, u_char *src, size_t n)
{
  memcpy(dst, src, n);
  dst[n] = '\0';

  return dst;
}

// allocate, copy and add terminating null. do not return null but 
null_str
to avoid segmentation fault later (dereferencing null ptr)
u_char* ngx_sbb_utils_str2char(ngx_http_request_t *r, ngx_str_t 
*ngx_str)
{
  u_char *res = NULL;

  if ( (!ngx_str) || (!r))
    return (u_char*)gv_null_str;

    res = ngx_pnalloc(r->pool, ngx_str->len+1);
    if (!res)
      return (u_char*)gv_null_str;

    return sbb_strncpy(res, ngx_str->data, ngx_str->len); // adds
terminating null
}

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,234580,234580#msg-234580
Posted by gadh (Guest)
on 2012-12-30 21:39
(Received via mailing list)
forgot to add my nginx version: 1.2.5

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,234580,234583#msg-234583
Posted by gadh (Guest)
on 2012-12-31 10:11
(Received via mailing list)
i found that in some cases of the crash, the source of th crash was that
nginx pnalloc() returned invalid ptr address

0x6632333834643264 <Address 0x6632333834643264 out of bounds>

i use 64 bit system, but all of my pointers are in the 32 bits bounds. 
is it
related to the c/c++ code sharing ?
any help please ?

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,234580,234592#msg-234592
Posted by gadh (Guest)
on 2013-01-01 16:06
(Received via mailing list)
i think i found the source of the crash - i often hibernate my vbox 
(virtual
machine) and also my ubuntu (the host machine) so it appears that the 
memory
was garbaged. after revooting only the vnox all is normal now, no crash.
the one thing  i cuold not understand is why i got the crash only when 
using
Chromium and not in other browsers ?

Posted at Nginx Forum: 
http://forum.nginx.org/read.php?2,234580,234607#msg-234607
Please log in before posting. Registration is free and takes only a minute.
Existing account (Switch to SSL-encrypted connection)
NEW: Do you have a Google/GoogleMail or Yahoo account? No registration required!
Log in with Google account | Log in with Yahoo account
No account? Register here.