Glibc detected *** nginx: master process /usr/sbin/nginx: double free or corruption (out) -- what?

dcian · June 21, 2010, 11:27pm

Hi all,

I am getting entries like this in my nginx error log, and I have no idea
what they might mean.

(I am also getting /phpmyadmin/index.php served as a file, and not
passed to php-fastcgi, and they might be related).

Regards

Ian

*** glibc detected *** nginx: master process /usr/sbin/nginx: double
free or corruption (out): 0x099b2fa8 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x17b591]
/lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x17cde8]
/lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x17fecd]
nginx: master process /usr/sbin/nginx[0x80d9237]
[0x99a6cd0]
======= Memory map: ========
00110000-00263000 r-xp 00000000 08:01 524339
/lib/tls/i686/cmov/libc-2.11.1.so
00263000-00264000 —p 00153000 08:01 524339
/lib/tls/i686/cmov/libc-2.11.1.so
00264000-00266000 r-xp 00153000 08:01 524339
/lib/tls/i686/cmov/libc-2.11.1.so
00266000-00267000 rwxp 00155000 08:01 524339
/lib/tls/i686/cmov/libc-2.11.1.so
00267000-0026a000 rwxp 00000000 00:00 0
0026a000-0027d000 r-xp 00000000 08:01 524399
/lib/tls/i686/cmov/libnsl-2.11.1.so
0027d000-0027e000 r-xp 00012000 08:01 524399
/lib/tls/i686/cmov/libnsl-2.11.1.so
0027e000-0027f000 rwxp 00013000 08:01 524399
/lib/tls/i686/cmov/libnsl-2.11.1.so
0027f000-00281000 rwxp 00000000 00:00 0
002de000-002e4000 r-xp 00000000 08:01 524401
/lib/tls/i686/cmov/libnss_compat-2.11.1.so
002e4000-002e5000 r-xp 00006000 08:01 524401
/lib/tls/i686/cmov/libnss_compat-2.11.1.so
002e5000-002e6000 rwxp 00007000 08:01 524401
/lib/tls/i686/cmov/libnss_compat-2.11.1.so
00339000-0033b000 rwxp 00000000 00:00 0
005a6000-005a7000 rwxp 00000000 00:00 0
005ce000-005e9000 r-xp 00000000 08:01 532090 /lib/ld-2.11.1.so
005e9000-005ea000 r-xp 0001a000 08:01 532090 /lib/ld-2.11.1.so
005ea000-005eb000 rwxp 0001b000 08:01 532090 /lib/ld-2.11.1.so
0099d000-009a7000 r-xp 00000000 08:01 524405
/lib/tls/i686/cmov/libnss_files-2.11.1.so
009a7000-009a8000 r-xp 00009000 08:01 524405
/lib/tls/i686/cmov/libnss_files-2.11.1.so
009a8000-009a9000 rwxp 0000a000 08:01 524405
/lib/tls/i686/cmov/libnss_files-2.11.1.so
009b0000-009b1000 rwxs 00000000 00:04 11285 /dev/zero (deleted)
00b32000-00b33000 rwxp 00000000 00:00 0
00b9e000-00ba0000 r-xp 00000000 08:01 524353
/lib/tls/i686/cmov/libdl-2.11.1.so
00ba0000-00ba1000 r-xp 00001000 08:01 524353
/lib/tls/i686/cmov/libdl-2.11.1.so
00ba1000-00ba2000 rwxp 00002000 08:01 524353
/lib/tls/i686/cmov/libdl-2.11.1.so
00bd2000-00bef000 r-xp 00000000 08:01 524373 /lib/libgcc_s.so.1
00bef000-00bf0000 r-xp 0001c000 08:01 524373 /lib/libgcc_s.so.1
00bf0000-00bf1000 rwxp 0001d000 08:01 524373 /lib/libgcc_s.so.1
00d3e000-00d47000 r-xp 00000000 08:01 524347
/lib/tls/i686/cmov/libcrypt-2.11.1.so
00d47000-00d48000 r-xp 00008000 08:01 524347
/lib/tls/i686/cmov/libcrypt-2.11.1.so
00d48000-00d49000 rwxp 00009000 08:01 524347
/lib/tls/i686/cmov/libcrypt-2.11.1.so
00d49000-00d70000 rwxp 00000000 00:00 0
00e59000-00e61000 r-xp 00000000 08:01 524415
/lib/tls/i686/cmov/libnss_nis-2.11.1.so
00e61000-00e62000 r-xp 00007000 08:01 524415
/lib/tls/i686/cmov/libnss_nis-2.11.1.so
00e62000-00e63000 rwxp 00008000 08:01 524415
/lib/tls/i686/cmov/libnss_nis-2.11.1.so
00f29000-00f2a000 r-xp 00000000 00:00 0 [vdso]
00f2a000-02f2a000 rwxs 00000000 00:04 11283 /dev/zero (deleted)
08048000-0820e000 r-xp 00000000 08:01 787713 /usr/sbin/nginx
0820e000-0820f000 r-xp 001c5000 08:01 787713 /usr/sbin/nginx
0820f000-0821f000 rwxp 001c6000 08:01 787713 /usr/sbin/nginx
0821f000-0822d000 rwxp 00000000 00:00 0
099a4000-099ec000 rwxp 00000000 00:00 0 [heap]
b7600000-b7621000 rwxp 00000000 00:00 0
b7621000-b7700000 —p 00000000 00:00 0
bfd57000-bfd6c000 rwxp 00000000 00:00 0 [stack]
*** glibc detected *** nginx: master process /usr/sbin/nginx: double
free or corruption (out): 0x09b26fa8 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x6ab591]
/lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x6acde8]
/lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x6afecd]
nginx: master process /usr/sbin/nginx[0x80d9237]
[0x9b1acd0]
======= Memory map: ========
00110000-00111000 rwxs 00000000 00:04 12105 /dev/zero (deleted)
00111000-0012e000 r-xp 00000000 08:01 524373 /lib/libgcc_s.so.1
0012e000-0012f000 r-xp 0001c000 08:01 524373 /lib/libgcc_s.so.1
0012f000-00130000 rwxp 0001d000 08:01 524373 /lib/libgcc_s.so.1
002ff000-00300000 rwxp 00000000 00:00 0
0041e000-00431000 r-xp 00000000 08:01 524399
/lib/tls/i686/cmov/libnsl-2.11.1.so
00431000-00432000 r-xp 00012000 08:01 524399
/lib/tls/i686/cmov/libnsl-2.11.1.so
00432000-00433000 rwxp 00013000 08:01 524399
/lib/tls/i686/cmov/libnsl-2.11.1.so
00433000-00435000 rwxp 00000000 00:00 0
00461000-00467000 r-xp 00000000 08:01 524401
/lib/tls/i686/cmov/libnss_compat-2.11.1.so
00467000-00468000 r-xp 00006000 08:01 524401
/lib/tls/i686/cmov/libnss_compat-2.11.1.so
00468000-00469000 rwxp 00007000 08:01 524401
/lib/tls/i686/cmov/libnss_compat-2.11.1.so
0053e000-00559000 r-xp 00000000 08:01 532090 /lib/ld-2.11.1.so
00559000-0055a000 r-xp 0001a000 08:01 532090 /lib/ld-2.11.1.so
0055a000-0055b000 rwxp 0001b000 08:01 532090 /lib/ld-2.11.1.so
00635000-00636000 r-xp 00000000 00:00 0 [vdso]
00640000-00793000 r-xp 00000000 08:01 524339
/lib/tls/i686/cmov/libc-2.11.1.so
00793000-00794000 —p 00153000 08:01 524339
/lib/tls/i686/cmov/libc-2.11.1.so
00794000-00796000 r-xp 00153000 08:01 524339
/lib/tls/i686/cmov/libc-2.11.1.so
00796000-00797000 rwxp 00155000 08:01 524339
/lib/tls/i686/cmov/libc-2.11.1.so
00797000-0079a000 rwxp 00000000 00:00 0
00822000-0082c000 r-xp 00000000 08:01 524405
/lib/tls/i686/cmov/libnss_files-2.11.1.so
0082c000-0082d000 r-xp 00009000 08:01 524405
/lib/tls/i686/cmov/libnss_files-2.11.1.so
0082d000-0082e000 rwxp 0000a000 08:01 524405
/lib/tls/i686/cmov/libnss_files-2.11.1.so
0091e000-00927000 r-xp 00000000 08:01 524347
/lib/tls/i686/cmov/libcrypt-2.11.1.so
00927000-00928000 r-xp 00008000 08:01 524347
/lib/tls/i686/cmov/libcrypt-2.11.1.so
00928000-00929000 rwxp 00009000 08:01 524347
/lib/tls/i686/cmov/libcrypt-2.11.1.so
00929000-00950000 rwxp 00000000 00:00 0
00aa4000-00aac000 r-xp 00000000 08:01 524415
/lib/tls/i686/cmov/libnss_nis-2.11.1.so
00aac000-00aad000 r-xp 00007000 08:01 524415
/lib/tls/i686/cmov/libnss_nis-2.11.1.so
00aad000-00aae000 rwxp 00008000 08:01 524415
/lib/tls/i686/cmov/libnss_nis-2.11.1.so
00b24000-00b26000 r-xp 00000000 08:01 524353
/lib/tls/i686/cmov/libdl-2.11.1.so
00b26000-00b27000 r-xp 00001000 08:01 524353
/lib/tls/i686/cmov/libdl-2.11.1.so
00b27000-00b28000 rwxp 00002000 08:01 524353
/lib/tls/i686/cmov/libdl-2.11.1.so
00f0e000-00f0f000 rwxp 00000000 00:00 0
00f30000-00f32000 rwxp 00000000 00:00 0
00f32000-02f32000 rwxs 00000000 00:04 12103 /dev/zero (deleted)
08048000-0820e000 r-xp 00000000 08:01 787713 /usr/sbin/nginx
0820e000-0820f000 r-xp 001c5000 08:01 787713 /usr/sbin/nginx
0820f000-0821f000 rwxp 001c6000 08:01 787713 /usr/sbin/nginx
0821f000-0822d000 rwxp 00000000 00:00 0
09b18000-09b60000 rwxp 00000000 00:00 0 [heap]
b7700000-b7721000 rwxp 00000000 00:00 0
b7721000-b7800000 —p 00000000 00:00 0
bfbca000-bfbdf000 rwxp 00000000 00:00 0 [stack]
*** glibc detected *** nginx: master process /usr/sbin/nginx: double
free or corruption (out): 0x0875ffa8 ***
======= Backtrace: =========
/lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x17b591]

etc for 3 more all slightly different.

dcian · June 22, 2010, 12:03am

I am getting entries like this in my nginx error log, and I have no
idea what they might mean.

*** glibc detected *** nginx: master process /usr/sbin/nginx: double
free or corruption (out): 0x099b2fa8 ***

Which distro, how did you acquire nginx, etc?

Jeff

–
Ubuntu’s Bleeding Edge

Ye shall be cursed to fall in love so easily, and yet be so cold of
                   heart as never to express it.

dcian · June 22, 2010, 12:33am

Hello!

On Mon, Jun 21, 2010 at 10:26:35PM +0100, Ian H. wrote:

I am getting entries like this in my nginx error log, and I have no
idea what they might mean.

What does nginx -V show?

Also you may try running with MALLOC_CHECK_ set to 2 (see

for details) and provide backtrace.

See http://wiki.nginx.org/NginxDebugging for general nginx
debugging guidelines.

Maxim D.

dcian · June 22, 2010, 12:38am

On 21/06/2010 23:02, Jeff W. wrote:

Which distro, how did you acquire nginx, etc?

Jeff

Ubuntu 10.4 and compiled from sources today.

nginx-0.7.67
nginx_http_push_module-0.69
openssl-1.0.0a
pcre-8.02
zlib-1.2.5

./configure
–sbin-path=/usr/sbin
–conf-path=/etc/nginx/nginx.conf
–pid-path=/usr/local/nginx/nginx.pid
–with-http_ssl_module
–with-pcre=…/pcre-8.02
–with-zlib=…/zlib-1.2.5
–with-openssl=…/openssl-1.0.0a
–add-module=/home/ian/Sources/nginx_http_push_module-0.69
make
sudo make install

Configuration summary

using PCRE library: …/pcre-8.02
using OpenSSL library: …/openssl-1.0.0a
md5: using OpenSSL library
sha1 library is not used
using zlib library: …/zlib-1.2.5

nginx path prefix: “/usr/local/nginx”
nginx binary file: “/usr/sbin”
nginx configuration prefix: “/etc/nginx”
nginx configuration file: “/etc/nginx/nginx.conf”
nginx pid file: “/usr/local/nginx/nginx.pid”
nginx error log file: “/usr/local/nginx/logs/error.log”
nginx http access log file: “/usr/local/nginx/logs/access.log”
nginx http client request body temporary files: “client_body_temp”
nginx http proxy temporary files: “proxy_temp”
nginx http fastcgi temporary files: “fastcgi_temp”

dcian · June 22, 2010, 12:54am

On Mon, 2010-06-21 at 23:38 +0100, Ian H. wrote:

nginx_http_push_module-0.69
–with-zlib=…/zlib-1.2.5
–with-openssl=…/openssl-1.0.0a
–add-module=/home/ian/Sources/nginx_http_push_module-0.69
make
sudo make install

Have you tried with the distro’s versions of pcre, zlib, and openssl?

Cliff

dcian · June 22, 2010, 1:09am

On 21/06/2010 23:32, Maxim D. wrote:

Hello!

On Mon, Jun 21, 2010 at 10:26:35PM +0100, Ian H. wrote:

I am getting entries like this in my nginx error log, and I have no
idea what they might mean.

What does nginx -V show?

ian@anake:~$ nginx -V
nginx version: nginx/0.7.67
built by gcc 4.4.3 (Ubuntu 4.4.3-4ubuntu5)
TLS SNI support enabled
configure arguments: --sbin-path=/usr/sbin
–conf-path=/etc/nginx/nginx.conf --pid-path=/usr/local/nginx/nginx.pid
–with-http_ssl_module --with-pcre=…/pcre-8.02
–with-zlib=…/zlib-1.2.5 --with-openssl=…/openssl-1.0.0a
–add-module=/home/ian/Sources/nginx_http_push_module-0.69
ian@anake:~$

Also you may try running with MALLOC_CHECK_ set to 2 (see
Heap Consistency Checking (The GNU C Library)
for details) and provide backtrace.

I’ll take a look at that tomorrow if I can.

dcian · June 22, 2010, 1:09am

–conf-path=/etc/nginx/nginx.conf
–pid-path=/usr/local/nginx/nginx.pid
–with-http_ssl_module
–with-pcre=…/pcre-8.02
–with-zlib=…/zlib-1.2.5
–with-openssl=…/openssl-1.0.0a \

^ you’ve manually built pcre, zlib and openssl, despite the fact that
all of
these are provided by your distro?

First option: Install the -dev packages for all of nginx’s dependencies.
An
easy way to do this is “apt-get build-dep nginx” (as long as you have
some
deb-src entries in your sources.list).

Second option: It’s highly unlikely that you’re the first person to want
the
most up-to-date nginx for Ubuntu 10.04 LTS… surely someone has made
some
packages for you already? Ta-da!

https://launchpad.net/~jdub/+archive/devel

Jeff

–
Ubuntu’s Bleeding Edge

“There are men with an appetite for fight and a sense of occasion,
and
then there is Waugh.” - SMH, about Steve

dcian · June 22, 2010, 1:15am

On 21/06/2010 23:53, Cliff W. wrote:

Ubuntu 10.4 and compiled from sources today.
–pid-path=/usr/local/nginx/nginx.pid \

No. How would I do that?
pcre and zlib are not packages apt-get recognises.

openssl claims to be at latest (0.9.8k 25 mar 2009).

Regards

Ian

dcian · June 22, 2010, 1:38am

Hi,

No. How would I do that?
pcre and zlib are not packages apt-get recognises.

As Jeff suggested above, either by doing apt-get build-dep, which will
install all the dependencies required to build nginx.

Or you can use apt-cache search to find packages.
apt-cache search pcre shows me that the full package name is
libpcre3-dev.

Try to ./configure without specifying where the libs are, and if
something is missing the error message should help you figure out what
to install in your distro.

Cheers,

Matthieu.

–
Matthieu T.

dcian · June 22, 2010, 3:17am

On Mon, 2010-06-21 at 18:08 -0700, Cliff W. wrote:

On Tue, 2010-06-22 at 00:14 +0100, Ian H. wrote:

pcre and zlib are not packages apt-get recognises.

Yes, they just follow the weird Debian naming “conventions”:

aptitude install libpcre3-dev libssl-dev zlib1g-dev

BTW, I highly suggest using aptitude in place of apt-get for all your
package needs on Debian/Ubuntu. You’d certainly have found these
packages on your own had you used “aptitude search” in place of
“apt-cache search”.

Regards,
Cliff

dcian · June 22, 2010, 3:09am

On Tue, 2010-06-22 at 00:14 +0100, Ian H. wrote:

–conf-path=/etc/nginx/nginx.conf \

No. How would I do that?
pcre and zlib are not packages apt-get recognises.

Yes, they just follow the weird Debian naming “conventions”:

aptitude install libpcre3-dev libssl-dev zlib1g-dev

Cliff

–

dcian · June 22, 2010, 11:39am

On 22/06/2010 00:09, Jeff W. wrote:

First option: Install the -dev packages for all of nginx’s dependencies. An
easy way to do this is “apt-get build-dep nginx” (as long as you have some
deb-src entries in your sources.list).

Did that - and it recompiled just fine. I’ve not finished testing yet,
but things are looking good so far.
phpmyadmin is working, and no crashes.

Second option: It’s highly unlikely that you’re the first person to want the
most up-to-date nginx for Ubuntu 10.04 LTS… surely someone has made some
packages for you already? Ta-da!

https://launchpad.net/~jdub/+archive/devel

Thanks Jeff. I need the long polling module, so I have to compile
myself.

And thanks to everyone else who contributed. I leaned more about linux’
tools.
This was only my second dive into compiling under linux.

Regards

Ian