# Nginx 1.0.4 crashes

**URL:** <https://www.ruby-forum.com/t/nginx-1-0-4-crashes/209506>\
**Category:** NGINX\
**Created:** [July 15, 2011, 10:28am UTC](https://www.ruby-forum.com/t/nginx-1-0-4-crashes/209506 "2011-07-15T10:28:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [July 15, 2011, 10:28am UTC](https://www.ruby-forum.com/t/nginx-1-0-4-crashes/209506/1 "2011-07-15T10:28:39Z")

</div>

Hi,  
I’m experiencing some random nginx crashes. My server is Ubuntu 10.04,  
with glibc 2.11.1-0ubuntu7.8. Here is an excerpt from the log file:

2011/07/15 05:15:14 [error] 3299#0: \*2788234 open() “[…]” failed (2:  
No such file or directory), client: […],  
server: […], request: “GET /robots.txt HTTP/1.0”, host: “[…]”  
\*\*\* glibc detected \*\*\* nginx: worker process: malloc(): memory  
corruption: 0x0000000000c15dd0 \*\*\*  
======= Backtrace: =========  
/lib/libc.so.6(+0x775b6)[0x7fb125cf35b6]  
/lib/libc.so.6(+0x7b6d8)[0x7fb125cf76d8]  
/lib/libc.so.6(\_\_libc\_malloc+0x6e)[0x7fb125cf858e]  
nginx: worker process(ngx\_alloc+0x1e)[0x430bfe]  
nginx: worker process[0x418405]  
nginx: worker process(ngx\_pcalloc+0x16)[0x4185f6]  
nginx: worker process[0x44538e]  
nginx: worker process[0x436424]  
nginx: worker process(ngx\_process\_events\_and\_timers+0x5e)[0x42d54e]  
nginx: worker process[0x434ad5]  
nginx: worker process(ngx\_spawn\_process+0x194)[0x432de4]  
nginx: worker process[0x433f12]  
nginx: worker process(ngx\_master\_process\_cycle+0x1b4)[0x435034]  
nginx: worker process(main+0x947)[0x417517]  
/lib/libc.so.6(\_\_libc\_start\_main+0xfd)[0x7fb125c9ac4d]  
nginx: worker process[0x415d09]  
======= Memory map: ========  
00400000-004d1000 r-xp 00000000 09:01 16612  
/usr/sbin/nginx  
006d0000-006d1000 r–p 000d0000 09:01 16612  
/usr/sbin/nginx  
006d1000-006e6000 rw-p 000d1000 09:01 16612  
/usr/sbin/nginx  
006e6000-00e90000 rw-p 00000000 00:00 0  
[heap]  
[…]  
2011/07/15 07:29:59 [alert] 3294#0: worker process 3295 exited on  
signal 6 (core dumped)  
2011/07/15 08:04:28 [alert] 3294#0: worker process 3299 exited on  
signal 11 (core dumped)  
2011/07/15 08:04:59 [alert] 3294#0: worker process 3297 exited on  
signal 11 (core dumped)  
2011/07/15 08:05:33 [alert] 3294#0: worker process 3298 exited on  
signal 11 (core dumped)

(also notice the times)

There were also different messages, like:  
\*\*\* glibc detected \*\*\* nginx: worker process: double free or  
corruption (!prev): 0x00000000009f9ad0 \*\*\*  
\*\*\* glibc detected \*\*\* nginx: worker process: free(): corrupted  
unsorted chunks: 0x00000000009872c0 \*\*\*

After loading the last dumped core with gdb & doing “bt” I get this:

Core was generated by `nginx: worker pr’.  
Program terminated with signal 11, Segmentation fault.  
#0 0x000000000041ac42 in ngx\_output\_chain ()  
(gdb) bt  
#0 0x000000000041ac42 in ngx\_output\_chain ()  
#1 0x0000000000457723 in ?? ()  
#2 0x00000000004583dd in ?? ()  
#3 0x000000000044e965 in ngx\_http\_read\_client\_request\_body ()  
#4 0x000000000047f7f9 in ?? ()  
#5 0x00000000004408d4 in ngx\_http\_core\_content\_phase ()  
#6 0x000000000043d0ed in ngx\_http\_core\_run\_phases ()  
#7 0x000000000043f6d5 in ngx\_http\_named\_location ()  
#8 0x0000000000443467 in ngx\_http\_core\_try\_files\_phase ()  
#9 0x000000000043d0ed in ngx\_http\_core\_run\_phases ()  
#10 0x0000000000447f25 in ?? ()  
#11 0x0000000000448bf2 in ?? ()  
#12 0x0000000000436424 in ?? ()  
#13 0x000000000042d54e in ngx\_process\_events\_and\_timers ()  
#14 0x0000000000434ad5 in ?? ()  
#15 0x0000000000432de4 in ngx\_spawn\_process ()  
#16 0x0000000000433f12 in ?? ()  
#17 0x0000000000435034 in ngx\_master\_process\_cycle ()  
#18 0x0000000000417517 in main ()

More info about my nginx version:  
nginx -V  
nginx: nginx version: nginx/1.0.4  
nginx: TLS SNI support enabled  
nginx: configure arguments: --prefix=/etc/nginx  
–conf-path=/etc/nginx/nginx.conf  
–error-log-path=/var/log/nginx/error.log  
–http-client-body-temp-path=/var/lib/nginx/body  
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi  
–http-log-path=/var/log/nginx/access.log  
–http-proxy-temp-path=/var/lib/nginx/proxy  
–http-scgi-temp-path=/var/lib/nginx/scgi  
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi  
–lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid  
–with-debug --with-http\_addition\_module --with-http\_dav\_module  
–with-http\_flv\_module --with-http\_geoip\_module  
–with-http\_gzip\_static\_module --with-http\_image\_filter\_module  
–with-http\_perl\_module --with-http\_random\_index\_module  
–with-http\_realip\_module --with-http\_secure\_link\_module  
–with-http\_stub\_status\_module --with-http\_ssl\_module  
–with-http\_sub\_module --with-http\_xslt\_module --with-ipv6  
–with-sha1=/usr/include/openssl --with-md5=/usr/include/openssl  
–with-mail --with-mail\_ssl\_module  
–add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-development-kit  
–add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-upstream-fair  
–add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-echo  
–add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-lua  
–add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-http-push  
–add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-upload-progress  
–add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-secure-download

The biggest problem is that when it happens, the server process is still  
alive,  
but it no longer handles connections, i.e. when you connect, the  
connection hangs infinitely.  
It’s necessary to restart nginx manually to make it work again. I  
thought nginx had  
a master/worker architecture to prevent such cases - a worker would  
die, but the  
rest should remain safe.

Anyway - is there any way to fix it? What can cause it?

–  
Thanks  
meris

---

<div class="post-metadata">

**Author:** ![Q\_S\_A](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/q/3da27b/32.png) [@Q\_S\_A](https://www.ruby-forum.com/u/Q_S_A)\
**Post date:** [July 15, 2011, 10:31am UTC](https://www.ruby-forum.com/t/nginx-1-0-4-crashes/209506/2 "2011-07-15T10:31:46Z")

</div>

On Fri, Jul 15, 2011 at 10:28:14AM +0200, Q & A wrote:

> /lib/libc.so.6(+0x775b6)[0x7fb125cf35b6]  
> nginx: worker process[0x433f12]  
> /usr/sbin/nginx
> 
> #6 0x000000000043d0ed in ngx\_http\_core\_run\_phases ()  
> #17 0x0000000000435034 in ngx\_master\_process\_cycle ()  
> –http-client-body-temp-path=/var/lib/nginx/body  
> –with-http\_realip\_module --with-http\_secure\_link\_module  
> –add-module=/build/buildd/nginx-1.0.4/debian/modules/nginx-secure-download  
> Could you reproduce the crash without third-pary modules which are built  
> using --add-module ?

–  
Igor S.

---

<div class="post-metadata">

**Author:** ![Q\_S\_A](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/q/3da27b/32.png) [@Q\_S\_A](https://www.ruby-forum.com/u/Q_S_A)\
**Post date:** [July 15, 2011, 12:12pm UTC](https://www.ruby-forum.com/t/nginx-1-0-4-crashes/209506/3 "2011-07-15T12:12:06Z")

</div>

> Could you reproduce the crash without third-pary modules which are built  
> using --add-module ?

I don’t know. I’ll give it a try and report back if it still crashes.

---

<div class="post-metadata">

**Author:** ![Q\_S\_A](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/q/3da27b/32.png) [@Q\_S\_A](https://www.ruby-forum.com/u/Q_S_A)\
**Post date:** [July 18, 2011, 8:02pm UTC](https://www.ruby-forum.com/t/nginx-1-0-4-crashes/209506/4 "2011-07-18T20:02:30Z")

</div>

> > Could you reproduce the crash without third-pary modules which are built  
> > using --add-module ?
> 
> I don’t know. I’ll give it a try and report back if it still crashes.

Hi again,

Looks like it’s still happening. I had several crashes that didn’t  
block the server, and some that required it to be restarted.  
My current config is:  
nginx: nginx version: nginx/1.0.4  
nginx: TLS SNI support enabled  
nginx: configure arguments: --prefix=/etc/nginx  
–conf-path=/etc/nginx/nginx.conf  
–error-log-path=/var/log/nginx/error.log  
–http-client-body-temp-path=/var/lib/nginx/body  
–http-fastcgi-temp-path=/var/lib/nginx/fastcgi  
–http-log-path=/var/log/nginx/access.log  
–http-proxy-temp-path=/var/lib/nginx/proxy  
–http-scgi-temp-path=/var/lib/nginx/scgi  
–http-uwsgi-temp-path=/var/lib/nginx/uwsgi  
–lock-path=/var/lock/nginx.lock --pid-path=/var/run/nginx.pid  
–with-debug --with-http\_addition\_module --with-http\_dav\_module  
–with-http\_flv\_module --with-http\_geoip\_module  
–with-http\_gzip\_static\_module --with-http\_image\_filter\_module  
–with-http\_perl\_module --with-http\_random\_index\_module  
–with-http\_realip\_module --with-http\_secure\_link\_module  
–with-http\_stub\_status\_module --with-http\_ssl\_module  
–with-http\_sub\_module --with-http\_xslt\_module --with-file-aio  
–with-ipv6 --with-sha1=/usr/include/openssl  
–with-md5=/usr/include/openssl

(most of the extra options aren’t used in my configs)

Backtrace gives:  
Program terminated with signal 6, Aborted.  
#0 0x00007fb3895c7a75 in raise () from /lib/libc.so.6  
(gdb) bt  
#0 0x00007fb3895c7a75 in raise () from /lib/libc.so.6  
#1 0x00007fb3895cb5c0 in abort () from /lib/libc.so.6  
#2 0x00007fb3896014fb in ?? () from /lib/libc.so.6  
#3 0x00007fb38960b5b6 in ?? () from /lib/libc.so.6  
#4 0x00007fb38960f6d8 in ?? () from /lib/libc.so.6  
#5 0x00007fb38961058e in malloc () from /lib/libc.so.6  
#6 0x00007fb3896118f9 in posix\_memalign () from /lib/libc.so.6  
#7 0x000000000042b39c in ngx\_memalign ()  
#8 0x0000000000412b7c in ngx\_create\_pool ()  
#9 0x0000000000429150 in ngx\_event\_accept ()  
#10 0x00000000004285e4 in ngx\_event\_process\_posted ()  
#11 0x0000000000427ddc in ngx\_process\_events\_and\_timers ()  
#12 0x000000000042f335 in ?? ()  
#13 0x000000000042d644 in ngx\_spawn\_process ()  
#14 0x000000000042e772 in ?? ()  
#15 0x000000000042f894 in ngx\_master\_process\_cycle ()  
#16 0x0000000000411d17 in main ()

Log output is:  
2011/07/18 15:19:45 [alert] 26671#0: worker process 10016 exited on  
signal 6 (core dumped)  
2011/07/18 15:19:52 [alert] 26671#0: worker process 11105 exited on  
signal 11 (core dumped)  
\*\*\* glibc detected \*\*\* nginx: worker process: malloc(): memory  
corruption: 0x0000000000cb02c0 \*\*\*  
======= Backtrace: =========  
/lib/libc.so.6(+0x775b6)[0x7fb38960b5b6]  
/lib/libc.so.6(+0x7b6d8)[0x7fb38960f6d8]  
/lib/libc.so.6(\_\_libc\_malloc+0x6e)[0x7fb38961058e]  
/lib/libc.so.6(posix\_memalign+0x39)[0x7fb3896118f9]  
nginx: worker process(ngx\_memalign+0x2c)[0x42b39c]  
nginx: worker process(ngx\_create\_pool+0x1c)[0x412b7c]  
nginx: worker process(ngx\_event\_accept+0x1e0)[0x429150]  
nginx: worker process(ngx\_event\_process\_posted+0x44)[0x4285e4]  
nginx: worker process(ngx\_process\_events\_and\_timers+0x8c)[0x427ddc]  
nginx: worker process[0x42f335]  
nginx: worker process(ngx\_spawn\_process+0x194)[0x42d644]  
nginx: worker process[0x42e772]  
nginx: worker process(ngx\_master\_process\_cycle+0x1b4)[0x42f894]  
nginx: worker process(main+0x947)[0x411d17]  
/lib/libc.so.6(\_\_libc\_start\_main+0xfd)[0x7fb3895b2c4d]  
nginx: worker process[0x410509]  
======= Memory map: ========  
00400000-004a1000 r-xp 00000000 09:01 6082  
/usr/sbin/nginx  
006a0000-006a1000 r–p 000a0000 09:01 6082  
/usr/sbin/nginx  
006a1000-006b2000 rw-p 000a1000 09:01 6082  
/usr/sbin/nginx  
006b2000-00d3e000 rw-p 00000000 00:00 0  
[heap]  
[…]

I’ll try to downgrade to the latest 8.x version and see if it works.  
What else would you suggest to do?

Regards,  
meris

---

<div class="post-metadata">

**Author:** ![Q\_S\_A](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/q/3da27b/32.png) [@Q\_S\_A](https://www.ruby-forum.com/u/Q_S_A)\
**Post date:** [July 19, 2011, 2:22am UTC](https://www.ruby-forum.com/t/nginx-1-0-4-crashes/209506/5 "2011-07-19T02:22:30Z")

</div>

On Tue, Jul 19, 2011 at 2:01 AM, Q & A [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> –http-log-path=/var/log/nginx/access.log  
> –with-http\_sub\_module --with-http\_xslt\_module --with-file-aio  
> #0 0x00007fb3895c7a75 in raise () from /lib/libc.so.6  
> #11 0x0000000000427ddc in ngx\_process\_events\_and\_timers ()  
> 2011/07/18 15:19:52 [alert] 26671#0: worker process 11105 exited on  
> nginx: worker process(ngx\_event\_accept+0x1e0)[0x429150]  
> 00400000-004a1000 r-xp 00000000 09:01 6082  
> I’ll try to downgrade to the latest 8.x version and see if it works.  
> What else would you suggest to do?

Its gonna be hard to figure out the culprit since you’ve lots of 3rd  
party  
modules 🙂