# Nginx-0.8.53 segfault

**URL:** <https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687>\
**Category:** NGINX\
**Created:** [January 12, 2011, 10:24pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687 "2011-01-12T22:24:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\
**Post date:** [January 12, 2011, 10:24pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/1 "2011-01-12T22:24:54Z")

</div>

I noticed that nginx segfaulting while serving large files like .avi or  
.flv

# tail -f /var/log/kern.log:

Jan 13 00:10:36 hvosting nginx[5068]: segfault at e8 ip 0808aeeb sp  
bd565c80 error 4 in nginx[8048000+60000]  
Jan 13 00:10:36 hvosting nginx[5064]: segfault at e8 ip 0808aeeb sp  
bd565c80 error 4 in nginx[8048000+60000]  
Jan 13 00:17:30 hvosting nginx[7081]: segfault at e8 ip 0808aeeb sp  
bd565cc0 error 4 in nginx[8048000+60000]  
Jan 13 00:17:30 hvosting nginx[5065]: segfault at e8 ip 0808aeeb sp  
bd565c50 error 4 in nginx[8048000+60000]

# tail -f /var/log/nginx/error\_log

2011/01/13 00:10:36 [alert] 5060#0: worker process 5068 exited on signal  
11  
2011/01/13 00:10:36 [alert] 5060#0: worker process 5064 exited on signal  
11  
2011/01/13 00:17:30 [alert] 5060#0: worker process 7081 exited on signal  
11  
2011/01/13 00:17:30 [alert] 5060#0: worker process 5065 exited on signal  
11

nginx is running under “apache” user, /etc/security/limits.conf:  
apache soft nofile 65536  
apache hard nofile 65536

nginx instance has few virtual hosts configured as reverse proxy and few  
virtual hosts are serving static content directly from fs.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 13, 2011, 2:06pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/2 "2011-01-13T14:06:23Z")

</div>

Hello!

On Wed, Jan 12, 2011 at 10:24:54PM +0100, Maxim C. wrote:

> Jan 13 00:17:30 hvosting nginx[5065]: segfault at e8 ip 0808aeeb sp  
> 11
> 
> nginx is running under “apache” user, /etc/security/limits.conf:  
> apache soft nofile 65536  
> apache hard nofile 65536
> 
> nginx instance has few virtual hosts configured as reverse proxy and few  
> virtual hosts are serving static content directly from fs.

Please provide:

1. nginx -V output
2. backtrace from coredump
3. full config

See here for basic instruction:

> **[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki...](https://github.com/nginxinc/nginx-wiki)**
>
> ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com

Note that 0.8.53 has at least one known and already fixed segfault  
problem  
(fixed in 0.9.0 and 0.8.54):

```
*) Bugfix: a segmentation fault might occur in a worker process, if 

```

the  
“auth\_basic” directive was used.  
Thanks to Michail Laletin.

Maxim D.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 13, 2011, 9:15pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/3 "2011-01-13T21:15:25Z")

</div>

> BFD: Предупреждение: /var/spool/nginx/cores/core усечён: ожидался размер  
> ядра файла \>= 2099683328, найдено: 524439552.

Seems that you have limited core size.  
See what ‘ulimit -c’ outputs and should probably set it ‘ulimit -c  
unlimited’ and restart nginx for it to produce full coredump.

rr

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 13, 2011, 8:16pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/4 "2011-01-13T20:16:52Z")

</div>

Hi, Maxim!

My nginx.conf is in attachment, i read [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/Debugging)  
but i can’t provide debug backtrace, my gdb outputs following:

# gdb /usr/sbin/nginx /var/spool/nginx/cores/core

GNU gdb (Gentoo 7.2 p1) 7.2  
Copyright (C) 2010 Free Software Foundation, Inc.  
License GPLv3+: GNU GPL version 3 or later  
[http://gnu.org/licenses/gpl.html](http://gnu.org/licenses/gpl.html)  
This is free software: you are free to change and redistribute it.  
There is NO WARRANTY, to the extent permitted by law. Type “show  
copying”  
and “show warranty” for details.  
This GDB was configured as “i686-pc-linux-gnu”.  
For bug reporting instructions, please see:  
[http://bugs.gentoo.org/](http://bugs.gentoo.org/)…  
Reading symbols from /usr/sbin/nginx…(no debugging symbols  
found)…done.  
BFD: Предупреждение: /var/spool/nginx/cores/core усечён: ожидался размер  
ядра файла \>= 2099683328, найдено: 524439552.  
[New Thread 6670]  
Cannot access memory at address 0xb1496648  
Cannot access memory at address 0xb1496648  
Cannot access memory at address 0xb1496648  
Reading symbols from /lib/ld-linux.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/ld-linux.so.2  
Failed to read a valid object file image from memory.  
Core was generated by `nginx: worker process '.  
Program terminated with signal 11, Segmentation fault.  
#0 0x08093c37 in ?? ()  
(gdb) bt  
#0 0x08093c37 in ?? ()  
Cannot access memory at address 0xbbc4732c  
(gdb)

Linux kernel i’m using has chpax and grsec patches, if it helps.

# nginx -V

nginx version: nginx/0.8.53  
TLS SNI support enabled  
configure arguments: --prefix=/usr --sbin-path=/usr/sbin/nginx  
–conf-path=/etc/nginx/nginx.conf  
–error-log-path=/var/log/nginx/error\_log --pid-path=/var/run/nginx.pid  
–lock-path=/var/lock/nginx.lock --user=nginx --group=nginx  
–with-cc-opt=-I/usr/include --with-ld-opt=-L/usr/lib  
–http-log-path=/var/log/nginx/access\_log  
–http-client-body-temp-path=/var/tmp/nginx/client  
–http-proxy-temp-path=/var/tmp/nginx/proxy  
–http-fastcgi-temp-path=/var/tmp/nginx/fastcgi  
–http-scgi-temp-path=/var/tmp/nginx/scgi  
–http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --with-debug --with-pcre  
–without-http\_autoindex\_module --without-http\_browser\_module  
–without-http\_charset\_module --without-http\_map\_module  
–without-http\_memcached\_module --without-http\_scgi\_module  
–without-http\_ssi\_module --without-http\_split\_clients\_module  
–without-http\_upstream\_ip\_hash\_module --without-http\_userid\_module  
–without-http\_uwsgi\_module --with-http\_dav\_module  
–with-http\_degradation\_module --with-http\_flv\_module  
–with-http\_image\_filter\_module --with-http\_stub\_status\_module  
–with-http\_realip\_module --with-http\_ssl\_module  
–without-mail\_imap\_module --without-mail\_pop3\_module  
–without-mail\_smtp\_module

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 13, 2011, 9:29pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/5 "2011-01-13T21:29:57Z")

</div>

On 1/13/11 3:24 PM, “Maxim C.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Reading symbols from /usr/sbin/nginx…(no debugging symbols

You stripped the binary, so you won’t get much useful info. Try with an  
unstripped binary.

–  
Brian A.  
Chief Operations Engineer  
Turner Digital Media Technologies

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 13, 2011, 9:24pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/6 "2011-01-13T21:24:48Z")

</div>

# ulimit -c

0

# ulimit -c unlimited

# ulimit -c

unlimited

# /etc/init.d/nginx restart

wait a little for new coredump )  
then

# gdb /usr/sbin/nginx /var/spool/nginx/cores/core

GNU gdb (Gentoo 7.2 p1) 7.2  
Copyright (C) 2010 Free Software Foundation, Inc.  
License GPLv3+: GNU GPL version 3 or later  
[http://gnu.org/licenses/gpl.html](http://gnu.org/licenses/gpl.html)  
This is free software: you are free to change and redistribute it.  
There is NO WARRANTY, to the extent permitted by law. Type “show  
copying”  
and “show warranty” for details.  
This GDB was configured as “i686-pc-linux-gnu”.  
For bug reporting instructions, please see:  
[http://bugs.gentoo.org/](http://bugs.gentoo.org/)…  
Reading symbols from /usr/sbin/nginx…(no debugging symbols  
found)…done.  
BFD: Предупреждение: /var/spool/nginx/cores/core усечён: ожидался размер  
ядра файла \>= 2099732480, найдено: 357965824.  
[New Thread 19472]  
Cannot access memory at address 0xa51db648  
Cannot access memory at address 0xa51db648  
Cannot access memory at address 0xa51db648  
Reading symbols from /lib/ld-linux.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/ld-linux.so.2  
Failed to read a valid object file image from memory.  
Core was generated by `nginx: worker process '.  
Program terminated with signal 11, Segmentation fault.  
#0 0x08093c37 in ?? ()  
(gdb) bt  
#0 0x08093c37 in ?? ()  
Cannot access memory at address 0xbb587b2c  
(gdb)

i think it’s consequence of chpax/grsec… btw previous version of nginx  
that was installed on my system didn’t have this bug. it was 0.7.X  
(don’t clearly remember)

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 13, 2011, 9:37pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/7 "2011-01-13T21:37:19Z")

</div>

What’s unstripped binary? How can i get it? I compiled nginx  
–with-debug option, so i was expecting debug symbols and i can’t figure  
out why they are absent.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 13, 2011, 10:42pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/8 "2011-01-13T22:42:53Z")

</div>

Hello!

On Thu, Jan 13, 2011 at 08:16:52PM +0100, Maxim C. wrote:

> My nginx.conf is in attachment, i read [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/Debugging)  
> but i can’t provide debug backtrace, my gdb outputs following:

[…]

> Reading symbols from /usr/sbin/nginx…(no debugging symbols  
> found)…done.

Your nginx binary is stripped, you have to make sure no strip(1)  
is run during install process (and no “-s” argument for  
install(1) is used).

> BFD: : /var/spool/nginx/cores/core ޣ:
> 
> > = 2099683328, : 524439552.

Resulting core is huge, most likely due to

```
    proxy_cache_path /var/spool/nginx levels=1:2 

```

keys\_zone=one:1000m;  
proxy\_cache\_path /var/spool/nginx-preview levels=1:2  
keys\_zone=image-preview:1000m;

in your config. 1000M for cache keys is a bit too many, it has  
space for 16mln keys (64 bytes per key on 32bit platforms). I  
doubt you have so many requests per 10 min (default inactive  
timeout you haven’t changed). You may want to reduce key zone  
size to something reasonable (something like 10M is enough in most  
cases).

Alternatively, set worker\_rlimit\_core big enough (i.e. larger than  
your nginx process size).

[…]

> –http-log-path=/var/log/nginx/access\_log  
> –without-http\_uwsgi\_module --with-http\_dav\_module  
> –with-http\_degradation\_module --with-http\_flv\_module  
> –with-http\_image\_filter\_module --with-http\_stub\_status\_module  
> –with-http\_realip\_module --with-http\_ssl\_module  
> –without-mail\_imap\_module --without-mail\_pop3\_module  
> –without-mail\_smtp\_module
> 
> Attachments:  
> [http://www.ruby-forum.com/attachment/5736/nginx.conf](http://www.ruby-forum.com/attachment/5736/nginx.conf)

See no obvious problems.

Maxim D.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 12:34am UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/9 "2011-01-14T00:34:35Z")

</div>

On Thu, Jan 13, 2011 at 12:16, Maxim C. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Cannot access memory at address 0xb1496648  
> Cannot access memory at address 0xb1496648  
> Cannot access memory at address 0xb1496648  
> […]  
> #0 0x08093c37 in ?? ()  
> (gdb) bt  
> #0 0x08093c37 in ?? ()  
> Cannot access memory at address 0xbbc4732c  
> (gdb)
> 
> Linux kernel i’m using has chpax and grsec patches, if it helps.

IIRC under PaX you need to chpax/paxctl -m /usr/sbin/nginx (turn off  
mprotect restrictions) to let gdb get a stacktrace. If that does not  
work you might need -r or -x or just disable everything: -pemrxs.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 2:57am UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/10 "2011-01-14T02:57:39Z")

</div>

On 1/13/11 4:41 PM, “Maxim D.” [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> ```
> proxy_cache_path /var/spool/nginx levels=1:2 keys_zone=one:1000m;
> proxy_cache_path /var/spool/nginx-preview levels=1:2
> 
> ```
> 
> keys\_zone=image-preview:1000m;
> 
> in your config. 1000M for cache keys is a bit too many, it has  
> space for 16mln keys (64 bytes per key on 32bit platforms).

Okay, this is a lingering question that is fuzzy in the docs. Is this  
size  
in the key zone the size of the share memory lock for the keys or the  
size  
of the cache on disk? I looked through the code for file\_cache, but  
there  
are several lengths added and subtracted.

Thanks.

–  
Brian A.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 12:35am UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/11 "2011-01-14T00:35:24Z")

</div>

tried to compile nginx on a same system with the same configure and gcc  
options (added -ggdb option). hope it will be enought:

# gdb /usr/sbin/nginx /var/spool/nginx/cores/core

GNU gdb (Gentoo 7.2 p1) 7.2  
Copyright (C) 2010 Free Software Foundation, Inc.  
License GPLv3+: GNU GPL version 3 or later  
[http://gnu.org/licenses/gpl.html](http://gnu.org/licenses/gpl.html)  
This is free software: you are free to change and redistribute it.  
There is NO WARRANTY, to the extent permitted by law. Type “show  
copying”  
and “show warranty” for details.  
This GDB was configured as “i686-pc-linux-gnu”.  
For bug reporting instructions, please see:  
[http://bugs.gentoo.org/](http://bugs.gentoo.org/)…  
Reading symbols from /usr/sbin/nginx…(no debugging symbols  
found)…done.  
[New Thread 30038]

warning: Can’t read pathname for load map: Ошибка ввода/вывода.  
Reading symbols from /lib/libcrypt.so.1…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libcrypt.so.1  
Reading symbols from /lib/libpcre.so.0…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libpcre.so.0  
Reading symbols from /usr/lib/libssl.so.0.9.8…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libssl.so.0.9.8  
Reading symbols from /usr/lib/libcrypto.so.0.9.8…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libcrypto.so.0.9.8  
Reading symbols from /lib/libdl.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libdl.so.2  
Reading symbols from /lib/libz.so.1…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libz.so.1  
Reading symbols from /usr/lib/libgd.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libgd.so.2  
Reading symbols from /lib/libc.so.6…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libc.so.6  
Reading symbols from /lib/ld-linux.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/ld-linux.so.2  
Reading symbols from /usr/lib/libjpeg.so.62…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libjpeg.so.62  
Reading symbols from /usr/lib/libpng12.so.0…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libpng12.so.0  
Reading symbols from /lib/libm.so.6…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libm.so.6  
Reading symbols from /lib/libnss\_compat.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libnss\_compat.so.2  
Reading symbols from /lib/libnsl.so.1…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libnsl.so.1  
Reading symbols from /lib/libnss\_nis.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libnss\_nis.so.2  
Reading symbols from /lib/libnss\_files.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libnss\_files.so.2  
Core was generated by `nginx: worker process '.  
Program terminated with signal 11, Segmentation fault.  
#0 0x080aaca8 in ngx\_http\_file\_cache\_update ()  
(gdb) bt  
#0 0x080aaca8 in ngx\_http\_file\_cache\_update ()  
#1 0x0809f213 in ngx\_http\_upstream\_process\_request ()  
#2 0x0809f054 in ngx\_http\_upstream\_process\_upstream ()  
#3 0x0809e701 in ngx\_http\_upstream\_send\_response ()  
#4 0x0809cf89 in ngx\_http\_upstream\_process\_header ()  
#5 0x0809ba4a in ngx\_http\_upstream\_handler ()  
#6 0x08069fe5 in ngx\_event\_process\_posted ()  
#7 0x08068319 in ngx\_process\_events\_and\_timers ()  
#8 0x08073177 in ngx\_worker\_process\_cycle ()  
#9 0x0807063d in ngx\_spawn\_process ()  
#10 0x08072d57 in ngx\_reap\_children ()  
#11 0x08071e06 in ngx\_master\_process\_cycle ()  
#12 0x0804c730 in main ()

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 3:03am UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/12 "2011-01-14T03:03:18Z")

</div>

Hello,

> Okay, this is a lingering question that is fuzzy in the docs. Is this  
> size  
> in the key zone the size of the share memory lock for the keys or the size  
> of the cache on disk? I looked through the code for file\_cache, but there  
> are several lengths added and subtracted.

“keys\_zone” is size of the shared memory, “max\_size” is the maximum  
on-disk  
size of the cache, but AFAIK it’s not really working at the moment.

Best regards,  
Piotr S. \< [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) \>

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 8:38am UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/13 "2011-01-14T08:38:52Z")

</div>

Hi, guys!

Do you understand what causes the problem?  
Should I provide more info?

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 12:55pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/14 "2011-01-14T12:55:38Z")

</div>

Hello!

On Fri, Jan 14, 2011 at 12:35:24AM +0100, Maxim C. wrote:

> tried to compile nginx on a same system with the same configure and gcc  
> options (added -ggdb option). hope it will be enought:

nginx should automatically construct CFLAGS with appropriate debug  
options (-g for gcc) unless you have non-empty CFLAGS environment  
variable defined when running ./configure script.

Using -ggdb should be ok too.

> # gdb /usr/sbin/nginx /var/spool/nginx/cores/core

[…]

> Reading symbols from /usr/sbin/nginx…(no debugging symbols  
> found)…done.

Looks like binary is still strip’ed.

> #9 0x0807063d in ngx\_spawn\_process ()  
> #10 0x08072d57 in ngx\_reap\_children ()  
> #11 0x08071e06 in ngx\_master\_process\_cycle ()  
> #12 0x0804c730 in main ()

Backtrace suggests there is some problem in proxy\_cache, but it’s  
impossible to say more without debug symbols.

Maxim D.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 1:26pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/15 "2011-01-14T13:26:30Z")

</div>

Maxim, thank you.

My CFLAGS variable wasn’t empty, after setting it to CFLAGS=“-ggdb”  
before configure i got nginx binary with debugging symbols.

# gdb /usr/sbin/nginx /var/spool/nginx/cores/core

GNU gdb (Gentoo 7.2 p1) 7.2  
Copyright (C) 2010 Free Software Foundation, Inc.  
License GPLv3+: GNU GPL version 3 or later  
[http://gnu.org/licenses/gpl.html](http://gnu.org/licenses/gpl.html)  
This is free software: you are free to change and redistribute it.  
There is NO WARRANTY, to the extent permitted by law. Type “show  
copying”  
and “show warranty” for details.  
This GDB was configured as “i686-pc-linux-gnu”.  
For bug reporting instructions, please see:  
[http://bugs.gentoo.org/](http://bugs.gentoo.org/)…  
Reading symbols from /usr/sbin/nginx…done.  
[New Thread 27193]

warning: Can’t read pathname for load map: Ошибка ввода/вывода.  
Reading symbols from /lib/libcrypt.so.1…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libcrypt.so.1  
Reading symbols from /lib/libpcre.so.0…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libpcre.so.0  
Reading symbols from /usr/lib/libssl.so.0.9.8…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libssl.so.0.9.8  
Reading symbols from /usr/lib/libcrypto.so.0.9.8…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libcrypto.so.0.9.8  
Reading symbols from /lib/libdl.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libdl.so.2  
Reading symbols from /lib/libz.so.1…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libz.so.1  
Reading symbols from /usr/lib/libgd.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libgd.so.2  
Reading symbols from /lib/libc.so.6…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libc.so.6  
Reading symbols from /lib/ld-linux.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/ld-linux.so.2  
Reading symbols from /usr/lib/libjpeg.so.62…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libjpeg.so.62  
Reading symbols from /usr/lib/libpng12.so.0…(no debugging symbols  
found)…done.  
Loaded symbols for /usr/lib/libpng12.so.0  
Reading symbols from /lib/libm.so.6…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libm.so.6  
Reading symbols from /lib/libnss\_compat.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libnss\_compat.so.2  
Reading symbols from /lib/libnsl.so.1…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libnsl.so.1  
Reading symbols from /lib/libnss\_nis.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libnss\_nis.so.2  
Reading symbols from /lib/libnss\_files.so.2…(no debugging symbols  
found)…done.  
Loaded symbols for /lib/libnss\_files.so.2  
Core was generated by `nginx: worker process '.  
Program terminated with signal 11, Segmentation fault.  
#0 0x080aaca8 in ngx\_http\_file\_cache\_update (r=0x82085a8, tf=0x820a4ec)  
at src/http/ngx\_http\_file\_cache.c:790  
790 if (c-\>updated) {  
(gdb) bt  
#0 0x080aaca8 in ngx\_http\_file\_cache\_update (r=0x82085a8, tf=0x820a4ec)  
at src/http/ngx\_http\_file\_cache.c:790  
#1 0x0809f213 in ngx\_http\_upstream\_process\_request (r=0x82085a8)  
at src/http/ngx\_http\_upstream.c:2661  
#2 0x0809f054 in ngx\_http\_upstream\_process\_upstream (r=0x82085a8,  
u=0x8209d68)  
at src/http/ngx\_http\_upstream.c:2603  
#3 0x0809e701 in ngx\_http\_upstream\_send\_response (r=0x82085a8,  
u=0x8209d68)  
at src/http/ngx\_http\_upstream.c:2292  
#4 0x0809cf89 in ngx\_http\_upstream\_process\_header (r=0x82085a8,  
u=0x8209d68)  
at src/http/ngx\_http\_upstream.c:1589  
#5 0x0809ba4a in ngx\_http\_upstream\_handler (ev=0x818fc40)  
at src/http/ngx\_http\_upstream.c:895  
#6 0x08074f9a in ngx\_epoll\_process\_events (cycle=0x8100018, timer=500,  
flags=1) at src/event/modules/ngx\_epoll\_module.c:642  
#7 0x08068211 in ngx\_process\_events\_and\_timers (cycle=0x8100018)  
at src/event/ngx\_event.c:245  
#8 0x08073177 in ngx\_worker\_process\_cycle (cycle=0x8100018, data=0x0)  
at src/os/unix/ngx\_process\_cycle.c:795  
#9 0x0807063d in ngx\_spawn\_process (cycle=0x8100018,  
proc=0x807302c \<ngx\_worker\_process\_cycle\>, data=0x0,  
name=0x80cbe9a “worker process”, respawn=5)  
at src/os/unix/ngx\_process.c:196  
#10 0x08072d57 in ngx\_reap\_children (cycle=0x8100018)  
—Type to continue, or q to quit—  
at src/os/unix/ngx\_process\_cycle.c:612  
#11 0x08071e06 in ngx\_master\_process\_cycle (cycle=0x8100018)  
at src/os/unix/ngx\_process\_cycle.c:180  
#12 0x0804c730 in main (argc=3, argv=0xbb708f04) at src/core/nginx.c:401

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 4:56pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/16 "2011-01-14T16:56:24Z")

</div>

Hello!

On Fri, Jan 14, 2011 at 01:26:30PM +0100, Maxim C. wrote:

> Maxim, thank you.
> 
> My CFLAGS variable wasn’t empty, after setting it to CFLAGS=“-ggdb”  
> before configure i got nginx binary with debugging symbols.

[…]

> Program terminated with signal 11, Segmentation fault.  
> #0 0x080aaca8 in ngx\_http\_file\_cache\_update (r=0x82085a8, tf=0x820a4ec)  
> at src/http/ngx\_http\_file\_cache.c:790  
> 790 if (c-\>updated) {  
> (gdb) bt  
> #0 0x080aaca8 in ngx\_http\_file\_cache\_update (r=0x82085a8, tf=0x820a4ec)  
> at src/http/ngx\_http\_file\_cache.c:790

Could you please show output of the following gdb commands:

fr 0  
p c  
p \*r  
p \*c

?

It would be helpful to obtain debug log as well, but I don’t  
expect it would be trivial.

Maxim D.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 5:07pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/17 "2011-01-14T17:07:29Z")

</div>

(gdb) fr 0  
#0 0x080aaca8 in ngx\_http\_file\_cache\_update (r=0x1, tf=0x10)  
at src/http/ngx\_http\_file\_cache.c:790  
790 if (c-\>updated) {  
(gdb) p c  
$1 = (ngx\_http\_cache\_t \*) 0x10  
(gdb) p \*r  
Cannot access memory at address 0x1  
(gdb) p \*c  
Cannot access memory at address 0x10  
(gdb)

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 5:22pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/18 "2011-01-14T17:22:45Z")

</div>

Hello!

On Fri, Jan 14, 2011 at 05:07:29PM +0100, Maxim C. wrote:

> (gdb) fr 0  
> #0 0x080aaca8 in ngx\_http\_file\_cache\_update (r=0x1, tf=0x10)  
> at src/http/ngx\_http\_file\_cache.c:790  
> 790 if (c-\>updated) {

This doesn’t looks like the same core (actually, this one looks  
like corrupted core file, probably you happened to run gdb during  
coredump to the same file). In your previous message this stack  
frame looked like:

> > > #0 0x080aaca8 in ngx\_http\_file\_cache\_update (r=0x82085a8, tf=0x820a4ec)  
> > > at src/http/ngx\_http\_file\_cache.c:790

Note r and tf are valid pointers.

Maxim D.

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 6:15pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/19 "2011-01-14T18:15:50Z")

</div>

Maxim, thanks for your help. I saved fresh core 🙂 to anothe directory  
and ran gdb, output of the commands that you provided is:

(gdb) fr 0  
#0 0x080aaca8 in ngx\_http\_file\_cache\_update (r=0x8200528, tf=0x8201570)  
at src/http/ngx\_http\_file\_cache.c:790  
790 if (c-\>updated) {  
(gdb) p c  
$1 = (ngx\_http\_cache\_t \*) 0x0  
(gdb) p \*r  
$2 = {signature = 1347703880, connection = 0x8133e88, ctx = 0x82009d8,  
main\_conf = 0x8100a0c, srv\_conf = 0x8112e90, loc\_conf = 0x8113e38,  
read\_event\_handler = 0x809ba57  
\<ngx\_http\_upstream\_rd\_check\_broken\_connection\>, write\_event\_handler =  
0x808ba03 \<ngx\_http\_request\_empty\_handler\>,  
cache = 0x0, upstream = 0x8200dc0, upstream\_states = 0x8201378, pool =  
0x82007d0, header\_in = 0x81fbf1c, headers\_in = {headers = {last =  
0x8200560,  
part = {elts = 0x8200af4, nelts = 9, next = 0x0}, size = 24,  
nalloc = 20, pool = 0x82007d0}, host = 0x8200af4, connection = 0x0,  
if\_modified\_since = 0x0, user\_agent = 0x8200b0c, referer =  
0x8200bb4, content\_length = 0x0, content\_type = 0x0, range = 0x0,  
if\_range = 0x0,  
transfer\_encoding = 0x0, expect = 0x0, accept\_encoding = 0x8200b54,  
via = 0x0, authorization = 0x0, keep\_alive = 0x8200b84, x\_forwarded\_for  
= 0x0,  
x\_real\_ip = 0x0, depth = 0x0, destination = 0x0, overwrite = 0x0,  
date = 0x0, user = {len = 0, data = 0x0}, passwd = {len = 0, data =  
0x0}, cookies = {  
elts = 0x8200cd4, nelts = 0, size = 4, nalloc = 2, pool =  
0x82007d0}, server = {len = 13,  
data = 0x8200ce0  
“media.ifun.ruuser-agentacceptaccept-languageaccept-encodingaccept-charsetkeep-aliveconnectionreferer\034I\021\b”},  
content\_length\_n = -1, keep\_alive\_n = 115, connection\_type = 2, msie  
= 0, msie4 = 0, msie6 = 0, opera = 0, gecko = 1, chrome = 0, safari = 0,  
konqueror = 0}, headers\_out = {headers = {last = 0x820060c, part =  
{elts = 0x82007f8, nelts = 0, next = 0x0}, size = 24, nalloc = 20,  
pool = 0x82007d0}, status = 200, status\_line = {len = 0, data =  
0x0}, server = 0x0, date = 0x0, content\_length = 0x0, content\_encoding =  
0x0,  
location = 0x0, refresh = 0x0, last\_modified = 0x0, content\_range =  
0x0, accept\_ranges = 0x0, www\_authenticate = 0x0, expires = 0x0, etag =  
0x0,  
override\_charset = 0x0, content\_type\_len = 9, content\_type = {len =  
9, data = 0x80d4ce9 “image/gif”}, charset = {len = 0, data = 0x0},  
content\_type\_lowcase = 0x0, content\_type\_hash = 0, cache\_control =  
{elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},  
content\_length\_n = 43,  
date\_time = 0, last\_modified\_time = 23349600}, request\_body =  
0x820103c, lingering\_time = 0, start\_sec = 1295025115, start\_msec = 407,  
method = 2,  
http\_version = 1001, request\_line = {len = 30, data = 0x81fbfa8 “GET  
/preview/x045.jpg HTTP/1.1\r\nHost”}, uri = {len = 6, data = 0x8113d6c  
“/empty”},  
args = {len = 0, data = 0x0}, exten = {len = 0, data = 0x0},  
unparsed\_uri = {len = 17, data = 0x81fbfac “/preview/x045.jpg  
HTTP/1.1\r\nHost”},  
method\_name = {len = 3, data = 0x80cdf28 “GET “}, http\_protocol = {len  
= 8, data = 0x81fbfbe “HTTP/1.1\r\nHost”}, out = 0x0, main = 0x8200528,  
parent = 0x0, postponed = 0x0, post\_subrequest = 0x0, posted\_requests  
= 0x0, virtual\_names = 0x81336a4, phase\_handler = 12,  
content\_handler = 0x80c6618 \<ngx\_http\_empty\_gif\_handler\>, access\_code  
= 0, variables = 0x8200a54, ncaptures = 0, captures = 0x0, captures\_data  
= 0x0,  
limit\_rate = 0, header\_size = 217, request\_length = 423, err\_status =  
0, http\_connection = 0x81fbefc,  
log\_handler = 0x808c150 \<ngx\_http\_log\_error\_handler\>, cleanup =  
0x82013ac, subrequests = 51, count = 2, blocked = 0, aio = 0, http\_state  
= 2,  
—Type to continue, or q to quit—  
complex\_uri = 0, quoted\_uri = 0, plus\_in\_uri = 0, space\_in\_uri = 0,  
invalid\_header = 0, add\_uri\_to\_alias = 0, valid\_location = 1,  
valid\_unparsed\_uri = 1, uri\_changed = 0, uri\_changes = 10,  
request\_body\_in\_single\_buf = 0, request\_body\_in\_file\_only = 0,  
request\_body\_in\_persistent\_file = 0, request\_body\_in\_clean\_file = 0,  
request\_body\_file\_group\_access = 0, request\_body\_file\_log\_level = 5,  
subrequest\_in\_memory = 0, waited = 0, cached = 0, gzip\_tested = 0,  
gzip\_ok = 0, gzip\_vary = 0, proxy = 0, bypass\_cache = 0, no\_cache = 0,  
limit\_zone\_set = 0, limit\_req\_set = 0, pipeline = 0, plain\_http = 0,  
chunked = 0, header\_only = 0, keepalive = 1, lingering\_close = 0,  
discard\_body = 0,  
internal = 1, error\_page = 1, ignore\_content\_encoding = 0,  
filter\_finalize = 1, post\_action = 0, request\_complete = 0,  
request\_output = 1,  
header\_sent = 1, expect\_tested = 0, root\_tested = 0, done = 0, logged  
= 0, buffered = 0, main\_filter\_need\_in\_memory = 1, filter\_need\_in\_memory  
= 0,  
filter\_need\_temporary = 0, allow\_ranges = 0, stat\_reading = 0,  
stat\_writing = 1, state = 0, header\_hash = 3519315678, lowcase\_index =  
10,  
lowcase\_header = “connectionngthg”, ‘\000’ \<repeats 16 times\>,  
header\_name\_start = 0x82018a9 “\r\n\r\n404 Not  
Found\r\n\<body bgcolor=“white”\>\r\n

# 404 Not Found
\r\n
* * *
nginx/0.8.53\r\n\r\n\r\n\320\037  
\b\364” \b”,  
header\_name\_end = 0x82018a0 “: close\r\n\r\n\r\n404  
Not Found\r\n\<body bgcolor=“white”\>\r\n
# 404 Not Found
\r\n
* * *
nginx/0.8.53\r\n\r\n\r\n\320\037  
\b\364” \b”,  
header\_start = 0x82018a2 “close\r\n\r\n\r\n404 Not  
Found\r\n\<body bgcolor=“white”\>\r\n
# 404 Not Found
\r\n
* * *
nginx/0.8.53\r\n\r\n\r\n\320\037  
\b\364” \b",  
header\_end = 0x82018a9 “\r\n\r\n404 Not  
Found\r\n\<body bgcolor=“white”\>\r\n
# 404 Not Found
\r\n
* * *
nginx/0.8.53\r\n\r\n\r\n\320\037  
\b\364” \b", uri\_start = 0x81fbfac “/preview/x045.jpg  
HTTP/1.1\r\nHost”,  
uri\_end = 0x81fbfbd " HTTP/1.1\r\nHost", uri\_ext = 0x81fbfba “jpg  
HTTP/1.1\r\nHost”, args\_start = 0x0,  
request\_start = 0x81fbfa8 “GET /preview/x045.jpg HTTP/1.1\r\nHost”,  
request\_end = 0x81fbfc6 “\r\nHost”,  
method\_end = 0x81fbfaa “T /preview/x045.jpg HTTP/1.1\r\nHost”,  
schema\_start = 0x0, schema\_end = 0x0, host\_start = 0x0, host\_end = 0x0,  
port\_start = 0x0,  
port\_end = 0x0, http\_minor = 1, http\_major = 1}  
(gdb) p \*c  
Cannot access memory at address 0x0

---

<div class="post-metadata">

**Author:** ![maxim\_c](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/3bc359/32.png) [@maxim\_c](https://www.ruby-forum.com/u/maxim_c)\
**Post date:** [January 14, 2011, 7:56pm UTC](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687/20 "2011-01-14T19:56:36Z")

</div>

Hello!

On Fri, Jan 14, 2011 at 06:15:50PM +0100, Maxim C. wrote:

> $2 = {signature = 1347703880, connection = 0x8133e88, ctx = 0x82009d8,  
> […]

> http\_version = 1001, request\_line = {len = 30, data = 0x81fbfa8 “GET  
> /preview/x045.jpg HTTP/1.1\r\nHost”}, uri = {len = 6, data = 0x8113d6c  
> “/empty”},

[…]

> filter\_finalize = 1, post\_action = 0, request\_complete = 0,

Ah, understood, it’s image filter which triggers the problem. I’m  
able to reproduce the problem here with the following config:

```
server {
    listen 127.0.0.1:8080;

    location / {
        proxy_pass http://127.0.0.1:8080/bad/;
        proxy_cache one;
        proxy_cache_valid any 1h;

        image_filter resize 150 100;
        error_page 415 = /empty;
    }

    location /empty {
        return 204;
    }

    location /bad/ {
        return 404;
    }
}

```

Right now I think correct solution would be to move r-\>cache into  
upstream’s private data (r-\>upstream). Igor?

Maxim D.

[Next page](https://www.ruby-forum.com/t/nginx-0-8-53-segfault/201687.md?page=2)