Nginx-0.8.53 segfault

I noticed that nginx segfaulting while serving large files like .avi or
.flv

tail -f /var/log/kern.log:

Jan 13 00:10:36 hvosting nginx[5068]: segfault at e8 ip 0808aeeb sp
bd565c80 error 4 in nginx[8048000+60000]
Jan 13 00:10:36 hvosting nginx[5064]: segfault at e8 ip 0808aeeb sp
bd565c80 error 4 in nginx[8048000+60000]
Jan 13 00:17:30 hvosting nginx[7081]: segfault at e8 ip 0808aeeb sp
bd565cc0 error 4 in nginx[8048000+60000]
Jan 13 00:17:30 hvosting nginx[5065]: segfault at e8 ip 0808aeeb sp
bd565c50 error 4 in nginx[8048000+60000]

tail -f /var/log/nginx/error_log

2011/01/13 00:10:36 [alert] 5060#0: worker process 5068 exited on signal
11
2011/01/13 00:10:36 [alert] 5060#0: worker process 5064 exited on signal
11
2011/01/13 00:17:30 [alert] 5060#0: worker process 7081 exited on signal
11
2011/01/13 00:17:30 [alert] 5060#0: worker process 5065 exited on signal
11

nginx is running under “apache” user, /etc/security/limits.conf:
apache soft nofile 65536
apache hard nofile 65536

nginx instance has few virtual hosts configured as reverse proxy and few
virtual hosts are serving static content directly from fs.

Hello!

On Wed, Jan 12, 2011 at 10:24:54PM +0100, Maxim C. wrote:

Jan 13 00:17:30 hvosting nginx[5065]: segfault at e8 ip 0808aeeb sp
11

nginx is running under “apache” user, /etc/security/limits.conf:
apache soft nofile 65536
apache hard nofile 65536

nginx instance has few virtual hosts configured as reverse proxy and few
virtual hosts are serving static content directly from fs.

Please provide:

  1. nginx -V output
  2. backtrace from coredump
  3. full config

See here for basic instruction:

http://wiki.nginx.org/Debugging

Note that 0.8.53 has at least one known and already fixed segfault
problem
(fixed in 0.9.0 and 0.8.54):

*) Bugfix: a segmentation fault might occur in a worker process, if 

the
“auth_basic” directive was used.
Thanks to Michail Laletin.

Maxim D.

BFD: Предупреждение: /var/spool/nginx/cores/core усечён: ожидался размер
ядра файла >= 2099683328, найдено: 524439552.

Seems that you have limited core size.
See what ‘ulimit -c’ outputs and should probably set it ‘ulimit -c
unlimited’ and restart nginx for it to produce full coredump.

rr

Hi, Maxim!

My nginx.conf is in attachment, i read http://wiki.nginx.org/Debugging
but i can’t provide debug backtrace, my gdb outputs following:

gdb /usr/sbin/nginx /var/spool/nginx/cores/core

GNU gdb (Gentoo 7.2 p1) 7.2
Copyright © 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “i686-pc-linux-gnu”.
For bug reporting instructions, please see:
http://bugs.gentoo.org/
Reading symbols from /usr/sbin/nginx…(no debugging symbols
found)…done.
BFD: Предупреждение: /var/spool/nginx/cores/core усечён: ожидался размер
ядра файла >= 2099683328, найдено: 524439552.
[New Thread 6670]
Cannot access memory at address 0xb1496648
Cannot access memory at address 0xb1496648
Cannot access memory at address 0xb1496648
Reading symbols from /lib/ld-linux.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/ld-linux.so.2
Failed to read a valid object file image from memory.
Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
#0 0x08093c37 in ?? ()
(gdb) bt
#0 0x08093c37 in ?? ()
Cannot access memory at address 0xbbc4732c
(gdb)

Linux kernel i’m using has chpax and grsec patches, if it helps.

nginx -V

nginx version: nginx/0.8.53
TLS SNI support enabled
configure arguments: --prefix=/usr --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error_log --pid-path=/var/run/nginx.pid
–lock-path=/var/lock/nginx.lock --user=nginx --group=nginx
–with-cc-opt=-I/usr/include --with-ld-opt=-L/usr/lib
–http-log-path=/var/log/nginx/access_log
–http-client-body-temp-path=/var/tmp/nginx/client
–http-proxy-temp-path=/var/tmp/nginx/proxy
–http-fastcgi-temp-path=/var/tmp/nginx/fastcgi
–http-scgi-temp-path=/var/tmp/nginx/scgi
–http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --with-debug --with-pcre
–without-http_autoindex_module --without-http_browser_module
–without-http_charset_module --without-http_map_module
–without-http_memcached_module --without-http_scgi_module
–without-http_ssi_module --without-http_split_clients_module
–without-http_upstream_ip_hash_module --without-http_userid_module
–without-http_uwsgi_module --with-http_dav_module
–with-http_degradation_module --with-http_flv_module
–with-http_image_filter_module --with-http_stub_status_module
–with-http_realip_module --with-http_ssl_module
–without-mail_imap_module --without-mail_pop3_module
–without-mail_smtp_module

On 1/13/11 3:24 PM, “Maxim C.” [email protected] wrote:

Reading symbols from /usr/sbin/nginx…(no debugging symbols

You stripped the binary, so you won’t get much useful info. Try with an
unstripped binary.


Brian A.
Chief Operations Engineer
Turner Digital Media Technologies

ulimit -c

0

ulimit -c unlimited

ulimit -c

unlimited

/etc/init.d/nginx restart

wait a little for new coredump )
then

gdb /usr/sbin/nginx /var/spool/nginx/cores/core

GNU gdb (Gentoo 7.2 p1) 7.2
Copyright © 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “i686-pc-linux-gnu”.
For bug reporting instructions, please see:
http://bugs.gentoo.org/
Reading symbols from /usr/sbin/nginx…(no debugging symbols
found)…done.
BFD: Предупреждение: /var/spool/nginx/cores/core усечён: ожидался размер
ядра файла >= 2099732480, найдено: 357965824.
[New Thread 19472]
Cannot access memory at address 0xa51db648
Cannot access memory at address 0xa51db648
Cannot access memory at address 0xa51db648
Reading symbols from /lib/ld-linux.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/ld-linux.so.2
Failed to read a valid object file image from memory.
Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
#0 0x08093c37 in ?? ()
(gdb) bt
#0 0x08093c37 in ?? ()
Cannot access memory at address 0xbb587b2c
(gdb)

i think it’s consequence of chpax/grsec… btw previous version of nginx
that was installed on my system didn’t have this bug. it was 0.7.X
(don’t clearly remember)

What’s unstripped binary? How can i get it? I compiled nginx
–with-debug option, so i was expecting debug symbols and i can’t figure
out why they are absent.

Hello!

On Thu, Jan 13, 2011 at 08:16:52PM +0100, Maxim C. wrote:

My nginx.conf is in attachment, i read http://wiki.nginx.org/Debugging
but i can’t provide debug backtrace, my gdb outputs following:

[…]

Reading symbols from /usr/sbin/nginx…(no debugging symbols
found)…done.

Your nginx binary is stripped, you have to make sure no strip(1)
is run during install process (and no “-s” argument for
install(1) is used).

BFD: : /var/spool/nginx/cores/core ޣ:

= 2099683328, : 524439552.

Resulting core is huge, most likely due to

    proxy_cache_path  /var/spool/nginx  levels=1:2 

keys_zone=one:1000m;
proxy_cache_path /var/spool/nginx-preview levels=1:2
keys_zone=image-preview:1000m;

in your config. 1000M for cache keys is a bit too many, it has
space for 16mln keys (64 bytes per key on 32bit platforms). I
doubt you have so many requests per 10 min (default inactive
timeout you haven’t changed). You may want to reduce key zone
size to something reasonable (something like 10M is enough in most
cases).

Alternatively, set worker_rlimit_core big enough (i.e. larger than
your nginx process size).

[…]

–http-log-path=/var/log/nginx/access_log
–without-http_uwsgi_module --with-http_dav_module
–with-http_degradation_module --with-http_flv_module
–with-http_image_filter_module --with-http_stub_status_module
–with-http_realip_module --with-http_ssl_module
–without-mail_imap_module --without-mail_pop3_module
–without-mail_smtp_module

Attachments:
http://www.ruby-forum.com/attachment/5736/nginx.conf

See no obvious problems.

Maxim D.

On Thu, Jan 13, 2011 at 12:16, Maxim C. [email protected] wrote:

Cannot access memory at address 0xb1496648
Cannot access memory at address 0xb1496648
Cannot access memory at address 0xb1496648
[…]
#0 0x08093c37 in ?? ()
(gdb) bt
#0 0x08093c37 in ?? ()
Cannot access memory at address 0xbbc4732c
(gdb)

Linux kernel i’m using has chpax and grsec patches, if it helps.

IIRC under PaX you need to chpax/paxctl -m /usr/sbin/nginx (turn off
mprotect restrictions) to let gdb get a stacktrace. If that does not
work you might need -r or -x or just disable everything: -pemrxs.

On 1/13/11 4:41 PM, “Maxim D.” [email protected] wrote:

    proxy_cache_path  /var/spool/nginx  levels=1:2   keys_zone=one:1000m;
    proxy_cache_path /var/spool/nginx-preview levels=1:2

keys_zone=image-preview:1000m;

in your config. 1000M for cache keys is a bit too many, it has
space for 16mln keys (64 bytes per key on 32bit platforms).

Okay, this is a lingering question that is fuzzy in the docs. Is this
size
in the key zone the size of the share memory lock for the keys or the
size
of the cache on disk? I looked through the code for file_cache, but
there
are several lengths added and subtracted.

Thanks.


Brian A.

tried to compile nginx on a same system with the same configure and gcc
options (added -ggdb option). hope it will be enought:

gdb /usr/sbin/nginx /var/spool/nginx/cores/core

GNU gdb (Gentoo 7.2 p1) 7.2
Copyright © 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “i686-pc-linux-gnu”.
For bug reporting instructions, please see:
http://bugs.gentoo.org/
Reading symbols from /usr/sbin/nginx…(no debugging symbols
found)…done.
[New Thread 30038]

warning: Can’t read pathname for load map: Ошибка ввода/вывода.
Reading symbols from /lib/libcrypt.so.1…(no debugging symbols
found)…done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpcre.so.0…(no debugging symbols
found)…done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /usr/lib/libssl.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/libdl.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libz.so.1…(no debugging symbols
found)…done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /usr/lib/libgd.so.2…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libgd.so.2
Reading symbols from /lib/libc.so.6…(no debugging symbols
found)…done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libjpeg.so.62…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libjpeg.so.62
Reading symbols from /usr/lib/libpng12.so.0…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /lib/libm.so.6…(no debugging symbols
found)…done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libnss_compat.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1…(no debugging symbols
found)…done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/libnss_files.so.2
Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
#0 0x080aaca8 in ngx_http_file_cache_update ()
(gdb) bt
#0 0x080aaca8 in ngx_http_file_cache_update ()
#1 0x0809f213 in ngx_http_upstream_process_request ()
#2 0x0809f054 in ngx_http_upstream_process_upstream ()
#3 0x0809e701 in ngx_http_upstream_send_response ()
#4 0x0809cf89 in ngx_http_upstream_process_header ()
#5 0x0809ba4a in ngx_http_upstream_handler ()
#6 0x08069fe5 in ngx_event_process_posted ()
#7 0x08068319 in ngx_process_events_and_timers ()
#8 0x08073177 in ngx_worker_process_cycle ()
#9 0x0807063d in ngx_spawn_process ()
#10 0x08072d57 in ngx_reap_children ()
#11 0x08071e06 in ngx_master_process_cycle ()
#12 0x0804c730 in main ()

Hello,

Okay, this is a lingering question that is fuzzy in the docs. Is this
size
in the key zone the size of the share memory lock for the keys or the size
of the cache on disk? I looked through the code for file_cache, but there
are several lengths added and subtracted.

“keys_zone” is size of the shared memory, “max_size” is the maximum
on-disk
size of the cache, but AFAIK it’s not really working at the moment.

Best regards,
Piotr S. < [email protected] >

Hi, guys!

Do you understand what causes the problem?
Should I provide more info?

Hello!

On Fri, Jan 14, 2011 at 12:35:24AM +0100, Maxim C. wrote:

tried to compile nginx on a same system with the same configure and gcc
options (added -ggdb option). hope it will be enought:

nginx should automatically construct CFLAGS with appropriate debug
options (-g for gcc) unless you have non-empty CFLAGS environment
variable defined when running ./configure script.

Using -ggdb should be ok too.

gdb /usr/sbin/nginx /var/spool/nginx/cores/core

[…]

Reading symbols from /usr/sbin/nginx…(no debugging symbols
found)…done.

Looks like binary is still strip’ed.

#9 0x0807063d in ngx_spawn_process ()
#10 0x08072d57 in ngx_reap_children ()
#11 0x08071e06 in ngx_master_process_cycle ()
#12 0x0804c730 in main ()

Backtrace suggests there is some problem in proxy_cache, but it’s
impossible to say more without debug symbols.

Maxim D.

Maxim, thank you.

My CFLAGS variable wasn’t empty, after setting it to CFLAGS="-ggdb"
before configure i got nginx binary with debugging symbols.

gdb /usr/sbin/nginx /var/spool/nginx/cores/core

GNU gdb (Gentoo 7.2 p1) 7.2
Copyright © 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show
copying”
and “show warranty” for details.
This GDB was configured as “i686-pc-linux-gnu”.
For bug reporting instructions, please see:
http://bugs.gentoo.org/
Reading symbols from /usr/sbin/nginx…done.
[New Thread 27193]

warning: Can’t read pathname for load map: Ошибка ввода/вывода.
Reading symbols from /lib/libcrypt.so.1…(no debugging symbols
found)…done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/libpcre.so.0…(no debugging symbols
found)…done.
Loaded symbols for /lib/libpcre.so.0
Reading symbols from /usr/lib/libssl.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libssl.so.0.9.8
Reading symbols from /usr/lib/libcrypto.so.0.9.8…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libcrypto.so.0.9.8
Reading symbols from /lib/libdl.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libz.so.1…(no debugging symbols
found)…done.
Loaded symbols for /lib/libz.so.1
Reading symbols from /usr/lib/libgd.so.2…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libgd.so.2
Reading symbols from /lib/libc.so.6…(no debugging symbols
found)…done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libjpeg.so.62…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libjpeg.so.62
Reading symbols from /usr/lib/libpng12.so.0…(no debugging symbols
found)…done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /lib/libm.so.6…(no debugging symbols
found)…done.
Loaded symbols for /lib/libm.so.6
Reading symbols from /lib/libnss_compat.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/libnss_compat.so.2
Reading symbols from /lib/libnsl.so.1…(no debugging symbols
found)…done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /lib/libnss_nis.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/libnss_nis.so.2
Reading symbols from /lib/libnss_files.so.2…(no debugging symbols
found)…done.
Loaded symbols for /lib/libnss_files.so.2
Core was generated by `nginx: worker process '.
Program terminated with signal 11, Segmentation fault.
#0 0x080aaca8 in ngx_http_file_cache_update (r=0x82085a8, tf=0x820a4ec)
at src/http/ngx_http_file_cache.c:790
790 if (c->updated) {
(gdb) bt
#0 0x080aaca8 in ngx_http_file_cache_update (r=0x82085a8, tf=0x820a4ec)
at src/http/ngx_http_file_cache.c:790
#1 0x0809f213 in ngx_http_upstream_process_request (r=0x82085a8)
at src/http/ngx_http_upstream.c:2661
#2 0x0809f054 in ngx_http_upstream_process_upstream (r=0x82085a8,
u=0x8209d68)
at src/http/ngx_http_upstream.c:2603
#3 0x0809e701 in ngx_http_upstream_send_response (r=0x82085a8,
u=0x8209d68)
at src/http/ngx_http_upstream.c:2292
#4 0x0809cf89 in ngx_http_upstream_process_header (r=0x82085a8,
u=0x8209d68)
at src/http/ngx_http_upstream.c:1589
#5 0x0809ba4a in ngx_http_upstream_handler (ev=0x818fc40)
at src/http/ngx_http_upstream.c:895
#6 0x08074f9a in ngx_epoll_process_events (cycle=0x8100018, timer=500,
flags=1) at src/event/modules/ngx_epoll_module.c:642
#7 0x08068211 in ngx_process_events_and_timers (cycle=0x8100018)
at src/event/ngx_event.c:245
#8 0x08073177 in ngx_worker_process_cycle (cycle=0x8100018, data=0x0)
at src/os/unix/ngx_process_cycle.c:795
#9 0x0807063d in ngx_spawn_process (cycle=0x8100018,
proc=0x807302c <ngx_worker_process_cycle>, data=0x0,
name=0x80cbe9a “worker process”, respawn=5)
at src/os/unix/ngx_process.c:196
#10 0x08072d57 in ngx_reap_children (cycle=0x8100018)
—Type to continue, or q to quit—
at src/os/unix/ngx_process_cycle.c:612
#11 0x08071e06 in ngx_master_process_cycle (cycle=0x8100018)
at src/os/unix/ngx_process_cycle.c:180
#12 0x0804c730 in main (argc=3, argv=0xbb708f04) at src/core/nginx.c:401

Hello!

On Fri, Jan 14, 2011 at 01:26:30PM +0100, Maxim C. wrote:

Maxim, thank you.

My CFLAGS variable wasn’t empty, after setting it to CFLAGS="-ggdb"
before configure i got nginx binary with debugging symbols.

[…]

Program terminated with signal 11, Segmentation fault.
#0 0x080aaca8 in ngx_http_file_cache_update (r=0x82085a8, tf=0x820a4ec)
at src/http/ngx_http_file_cache.c:790
790 if (c->updated) {
(gdb) bt
#0 0x080aaca8 in ngx_http_file_cache_update (r=0x82085a8, tf=0x820a4ec)
at src/http/ngx_http_file_cache.c:790

Could you please show output of the following gdb commands:

fr 0
p c
p *r
p *c

?

It would be helpful to obtain debug log as well, but I don’t
expect it would be trivial.

Maxim D.

(gdb) fr 0
#0 0x080aaca8 in ngx_http_file_cache_update (r=0x1, tf=0x10)
at src/http/ngx_http_file_cache.c:790
790 if (c->updated) {
(gdb) p c
$1 = (ngx_http_cache_t *) 0x10
(gdb) p *r
Cannot access memory at address 0x1
(gdb) p *c
Cannot access memory at address 0x10
(gdb)

Hello!

On Fri, Jan 14, 2011 at 05:07:29PM +0100, Maxim C. wrote:

(gdb) fr 0
#0 0x080aaca8 in ngx_http_file_cache_update (r=0x1, tf=0x10)
at src/http/ngx_http_file_cache.c:790
790 if (c->updated) {

This doesn’t looks like the same core (actually, this one looks
like corrupted core file, probably you happened to run gdb during
coredump to the same file). In your previous message this stack
frame looked like:

#0 0x080aaca8 in ngx_http_file_cache_update (r=0x82085a8, tf=0x820a4ec)
at src/http/ngx_http_file_cache.c:790

Note r and tf are valid pointers.

Maxim D.

Maxim, thanks for your help. I saved fresh core :slight_smile: to anothe directory
and ran gdb, output of the commands that you provided is:

(gdb) fr 0
#0 0x080aaca8 in ngx_http_file_cache_update (r=0x8200528, tf=0x8201570)
at src/http/ngx_http_file_cache.c:790
790 if (c->updated) {
(gdb) p c
$1 = (ngx_http_cache_t *) 0x0
(gdb) p *r
$2 = {signature = 1347703880, connection = 0x8133e88, ctx = 0x82009d8,
main_conf = 0x8100a0c, srv_conf = 0x8112e90, loc_conf = 0x8113e38,
read_event_handler = 0x809ba57
<ngx_http_upstream_rd_check_broken_connection>, write_event_handler =
0x808ba03 <ngx_http_request_empty_handler>,
cache = 0x0, upstream = 0x8200dc0, upstream_states = 0x8201378, pool =
0x82007d0, header_in = 0x81fbf1c, headers_in = {headers = {last =
0x8200560,
part = {elts = 0x8200af4, nelts = 9, next = 0x0}, size = 24,
nalloc = 20, pool = 0x82007d0}, host = 0x8200af4, connection = 0x0,
if_modified_since = 0x0, user_agent = 0x8200b0c, referer =
0x8200bb4, content_length = 0x0, content_type = 0x0, range = 0x0,
if_range = 0x0,
transfer_encoding = 0x0, expect = 0x0, accept_encoding = 0x8200b54,
via = 0x0, authorization = 0x0, keep_alive = 0x8200b84, x_forwarded_for
= 0x0,
x_real_ip = 0x0, depth = 0x0, destination = 0x0, overwrite = 0x0,
date = 0x0, user = {len = 0, data = 0x0}, passwd = {len = 0, data =
0x0}, cookies = {
elts = 0x8200cd4, nelts = 0, size = 4, nalloc = 2, pool =
0x82007d0}, server = {len = 13,
data = 0x8200ce0
“media.ifun.ruuser-agentacceptaccept-languageaccept-encodingaccept-charsetkeep-aliveconnectionreferer\034I\021\b”},
content_length_n = -1, keep_alive_n = 115, connection_type = 2, msie
= 0, msie4 = 0, msie6 = 0, opera = 0, gecko = 1, chrome = 0, safari = 0,
konqueror = 0}, headers_out = {headers = {last = 0x820060c, part =
{elts = 0x82007f8, nelts = 0, next = 0x0}, size = 24, nalloc = 20,
pool = 0x82007d0}, status = 200, status_line = {len = 0, data =
0x0}, server = 0x0, date = 0x0, content_length = 0x0, content_encoding =
0x0,
location = 0x0, refresh = 0x0, last_modified = 0x0, content_range =
0x0, accept_ranges = 0x0, www_authenticate = 0x0, expires = 0x0, etag =
0x0,
override_charset = 0x0, content_type_len = 9, content_type = {len =
9, data = 0x80d4ce9 “image/gif”}, charset = {len = 0, data = 0x0},
content_type_lowcase = 0x0, content_type_hash = 0, cache_control =
{elts = 0x0, nelts = 0, size = 0, nalloc = 0, pool = 0x0},
content_length_n = 43,
date_time = 0, last_modified_time = 23349600}, request_body =
0x820103c, lingering_time = 0, start_sec = 1295025115, start_msec = 407,
method = 2,
http_version = 1001, request_line = {len = 30, data = 0x81fbfa8 “GET
/preview/x045.jpg HTTP/1.1\r\nHost”}, uri = {len = 6, data = 0x8113d6c
“/empty”},
args = {len = 0, data = 0x0}, exten = {len = 0, data = 0x0},
unparsed_uri = {len = 17, data = 0x81fbfac “/preview/x045.jpg
HTTP/1.1\r\nHost”},
method_name = {len = 3, data = 0x80cdf28 “GET “}, http_protocol = {len
= 8, data = 0x81fbfbe “HTTP/1.1\r\nHost”}, out = 0x0, main = 0x8200528,
parent = 0x0, postponed = 0x0, post_subrequest = 0x0, posted_requests
= 0x0, virtual_names = 0x81336a4, phase_handler = 12,
content_handler = 0x80c6618 <ngx_http_empty_gif_handler>, access_code
= 0, variables = 0x8200a54, ncaptures = 0, captures = 0x0, captures_data
= 0x0,
limit_rate = 0, header_size = 217, request_length = 423, err_status =
0, http_connection = 0x81fbefc,
log_handler = 0x808c150 <ngx_http_log_error_handler>, cleanup =
0x82013ac, subrequests = 51, count = 2, blocked = 0, aio = 0, http_state
= 2,
—Type to continue, or q to quit—
complex_uri = 0, quoted_uri = 0, plus_in_uri = 0, space_in_uri = 0,
invalid_header = 0, add_uri_to_alias = 0, valid_location = 1,
valid_unparsed_uri = 1, uri_changed = 0, uri_changes = 10,
request_body_in_single_buf = 0, request_body_in_file_only = 0,
request_body_in_persistent_file = 0, request_body_in_clean_file = 0,
request_body_file_group_access = 0, request_body_file_log_level = 5,
subrequest_in_memory = 0, waited = 0, cached = 0, gzip_tested = 0,
gzip_ok = 0, gzip_vary = 0, proxy = 0, bypass_cache = 0, no_cache = 0,
limit_zone_set = 0, limit_req_set = 0, pipeline = 0, plain_http = 0,
chunked = 0, header_only = 0, keepalive = 1, lingering_close = 0,
discard_body = 0,
internal = 1, error_page = 1, ignore_content_encoding = 0,
filter_finalize = 1, post_action = 0, request_complete = 0,
request_output = 1,
header_sent = 1, expect_tested = 0, root_tested = 0, done = 0, logged
= 0, buffered = 0, main_filter_need_in_memory = 1, filter_need_in_memory
= 0,
filter_need_temporary = 0, allow_ranges = 0, stat_reading = 0,
stat_writing = 1, state = 0, header_hash = 3519315678, lowcase_index =
10,
lowcase_header = “connectionngthg”, ‘\000’ <repeats 16 times>,
header_name_start = 0x82018a9 “\r\n\r\n404 Not
Found\r\n<body bgcolor=“white”>\r\n

404 Not
Found

\r\n
nginx/0.8.53\r\n\r\n\r\n\320\037
\b\364” \b”,
header_name_end = 0x82018a0 “: close\r\n\r\n\r\n404
Not Found\r\n<body bgcolor=“white”>\r\n

404
Not
Found

\r\n
nginx/0.8.53\r\n\r\n\r\n\320\037
\b\364” \b”,
header_start = 0x82018a2 “close\r\n\r\n\r\n404 Not
Found\r\n<body bgcolor=“white”>\r\n

404 Not
Found

\r\n
nginx/0.8.53\r\n\r\n\r\n\320\037
\b\364” \b",
header_end = 0x82018a9 “\r\n\r\n404 Not
Found\r\n<body bgcolor=“white”>\r\n

404 Not
Found

\r\n
nginx/0.8.53\r\n\r\n\r\n\320\037
\b\364” \b", uri_start = 0x81fbfac “/preview/x045.jpg
HTTP/1.1\r\nHost”,
uri_end = 0x81fbfbd " HTTP/1.1\r\nHost", uri_ext = 0x81fbfba “jpg
HTTP/1.1\r\nHost”, args_start = 0x0,
request_start = 0x81fbfa8 “GET /preview/x045.jpg HTTP/1.1\r\nHost”,
request_end = 0x81fbfc6 “\r\nHost”,
method_end = 0x81fbfaa “T /preview/x045.jpg HTTP/1.1\r\nHost”,
schema_start = 0x0, schema_end = 0x0, host_start = 0x0, host_end = 0x0,
port_start = 0x0,
port_end = 0x0, http_minor = 1, http_major = 1}
(gdb) p *c
Cannot access memory at address 0x0

Hello!

On Fri, Jan 14, 2011 at 06:15:50PM +0100, Maxim C. wrote:

$2 = {signature = 1347703880, connection = 0x8133e88, ctx = 0x82009d8,
[…]

http_version = 1001, request_line = {len = 30, data = 0x81fbfa8 “GET
/preview/x045.jpg HTTP/1.1\r\nHost”}, uri = {len = 6, data = 0x8113d6c
“/empty”},

[…]

filter_finalize = 1, post_action = 0, request_complete = 0,

Ah, understood, it’s image filter which triggers the problem. I’m
able to reproduce the problem here with the following config:

server {
    listen 127.0.0.1:8080;

    location / {
        proxy_pass http://127.0.0.1:8080/bad/;
        proxy_cache one;
        proxy_cache_valid any 1h;

        image_filter   resize  150 100;
        error_page     415   = /empty;
    }

    location /empty {
        return 204;
    }

    location /bad/ {
        return 404;
    }
}

Right now I think correct solution would be to move r->cache into
upstream’s private data (r->upstream). Igor?

Maxim D.