# Nginx fails to accept new connection if active worker crashes **URL:** \ **Category:** NGINX\ **Created:** [November 15, 2011, 8:32am UTC](https://www.ruby-forum.com/t/nginx-fails-to-accept-new-connection-if-active-worker-crashes/213888 "2011-11-15T08:32:31Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\ **Post date:** [November 15, 2011, 8:32am UTC](https://www.ruby-forum.com/t/nginx-fails-to-accept-new-connection-if-active-worker-crashes/213888/1 "2011-11-15T08:32:31Z")
Hi All I use nginx configured with multiple workers. I also have an nginx module that crashed due to an error when I noticed that the module crash leaves nginx in a state where it cannot accept new calls. Removing my module and killing the “active” worker (the one which seems to take the new request) with a SIGHUP again caused nginx to hang. Killing the other worker(s) seem to be working just fine. Further investigations(with nginx at debug level) showed that all threads are fine but none of the workers are getting the ngx\_accept\_mutex\_lock. Master tries to release the ngx\_accept\_mutex\_lock if the dead process was holding it [ [https://svn.nginx.org/nginx/browser/nginx/trunk/src/os/unix/ngx\_process.c?annotate=blame#L503](https://svn.nginx.org/nginx/browser/nginx/trunk/src/os/unix/ngx_process.c?annotate=blame#L503)] but doesnt look like the value is set anywhere. I have been using nginx only for a couple of months now so I am not very sure of the diagnosis, please feel free to correct. uname -a: Linux faskiri-pc 2.6.32-24-generic #43-Ubuntu SMP Thu Sep 16 14:58:24 UTC 2010 x86\_64 GNU/Linux nginx -V: nginx: nginx version: nginx/1.0.5 nginx: built by gcc 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5) nginx: configure arguments: –without-http\_ssi\_module --without-http\_geo\_module –without-http\_fastcgi\_module --without-http\_uwsgi\_module –without-http\_scgi\_module --without-http\_memcached\_module –without-mail\_pop3\_module --without-mail\_imap\_module –without-mail\_smtp\_module --with-pcre --with-debug I will be grateful for any advice. Best Regards +Fasih Posted at Nginx Forum: > **[Nginx fails to accept new connection if active worker crashes](https://forum.nginx.org/read.php?2,218359,218359#msg-218359)** > > Hi All I use nginx configured with multiple workers. I also have an nginx module that crashed due to an error when I noticed that the module crash leaves nginx in a state where it cannot accept new calls. Removing my module and killing the... ---
**Author:** ![faskiri.devel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/958977/32.png) [@faskiri.devel](https://www.ruby-forum.com/u/faskiri.devel)\ **Post date:** [November 15, 2011, 10:16am UTC](https://www.ruby-forum.com/t/nginx-fails-to-accept-new-connection-if-active-worker-crashes/213888/2 "2011-11-15T10:16:38Z")
Fasih, On Nov 15, 2011, at 11:32 AM, faskiri.devel wrote: > Further investigations(with nginx at debug level) showed that all > threads are fine but none of the workers are getting the > ngx\_accept\_mutex\_lock. > > Master tries to release the ngx\_accept\_mutex\_lock if the dead process > was holding it [ [https://svn.nginx.org/nginx/browser/nginx/trunk/src/os/unix/ngx\_process.c?annotate=blame#L503](https://svn.nginx.org/nginx/browser/nginx/trunk/src/os/unix/ngx_process.c?annotate=blame#L503)] > but doesnt look like the value is set anywhere. > > I have been using nginx only for a couple of months now so I am not very > sure of the diagnosis, please feel free to correct. Thanks for spotting this one. It’s kind of a known issue and we’re working on a fix currently. In the meanwhile you can switch accept mutex off as a workaround (the only downside could potentially be in minor increase of CPU utilization). ---
**Author:** ![faskiri.devel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/958977/32.png) [@faskiri.devel](https://www.ruby-forum.com/u/faskiri.devel)\ **Post date:** [November 15, 2011, 12:37pm UTC](https://www.ruby-forum.com/t/nginx-fails-to-accept-new-connection-if-active-worker-crashes/213888/3 "2011-11-15T12:37:54Z")
Hi Andrew Thanks for the prompt reply. As a temporary fix I had created a variable in the shared memory to track which pid is holding the mutex so that the check in [[https://svn.nginx.org/nginx/browser/nginx/trunk/src/os/unix/ngx\_process.c?annotate=blame#L503](https://svn.nginx.org/nginx/browser/nginx/trunk/src/os/unix/ngx_process.c?annotate=blame#L503)] works. It works fine for me, hadnt realized I could switch it off. Are we talking about putting “accept\_mutex off” in the nginx.conf file? It will be great if you could explain what that will actually do, as in, how will the new requests be handed off to workers. Best Regards +Fasih Posted at Nginx Forum: > **[Re: Nginx fails to accept new connection if active worker crashes](https://forum.nginx.org/read.php?2,218359,218373#msg-218373)** > > Hi All I use nginx configured with multiple workers. I also have an nginx module that crashed due to an error when I noticed that the module crash leaves nginx in a state where it cannot accept new calls. Removing my module and killing the... ---
**Author:** ![faskiri.devel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/958977/32.png) [@faskiri.devel](https://www.ruby-forum.com/u/faskiri.devel)\ **Post date:** [November 15, 2011, 2:26pm UTC](https://www.ruby-forum.com/t/nginx-fails-to-accept-new-connection-if-active-worker-crashes/213888/4 "2011-11-15T14:26:50Z")
On Nov 15, 2011, at 3:37 PM, faskiri.devel wrote: > how will the new requests be handed off to workers. > Yes, accept\_mutex off. What accept mutex does is trying to prevent > workers from competing over accept from listening sockets (in the > kernel). In other words, without accept mutex workers may try to > simultaneously check for new events on sockets which may lead to a > slight increase in CPU usage. Depending on your OS and the event notification mechanisms the results may vary. Actually it’s quite safe to try it and we’d appreciate your feedback here! And as I mentioned, we’ve been working on fixing the situation with crashed workers and mutex lock-ups. ---
**Author:** ![faskiri.devel](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/958977/32.png) [@faskiri.devel](https://www.ruby-forum.com/u/faskiri.devel)\ **Post date:** [November 16, 2011, 9:51am UTC](https://www.ruby-forum.com/t/nginx-fails-to-accept-new-connection-if-active-worker-crashes/213888/5 "2011-11-16T09:51:23Z")
Thanks for your attention! It works fine with accept\_mutex off, I will run my stress test harness over the weekend to see the impact on the performance. If there is a significant difference in performance, will surely update the thread with the same. For my understanding, I had implemented a workaround to get around this problem. Is your solution along the same line? My patch: diff --git a/service/nginxServer/nginx-1.0.5/src/event/ngx\_event.c b/service/nginxServer/nginx-1.0.5/src/event/ngx\_event.c index c57d37e…a6ed725 100644 — a/service/nginxServer/nginx-1.0.5/src/event/ngx\_event.c +++ b/service/nginxServer/nginx-1.0.5/src/event/ngx\_event.c @@ -49,6 +49,10 @@ ngx\_atomic\_t \*ngx\_connection\_counter = &connection\_counter; ngx\_atomic\_t \*ngx\_accept\_mutex\_ptr; ngx\_shmtx\_t ngx\_accept\_mutex; +// This is shared var protected by ngx\_use\_accept\_mutex. Access only when +// ngx\_accept\_mutex is held. The var stores the PID of the process currently +// holding the mutex +ngx\_pid\_t \*ngx\_accept\_mutex\_held\_by; ngx\_uint\_t ngx\_use\_accept\_mutex; ngx\_uint\_t ngx\_accept\_events; ngx\_uint\_t ngx\_accept\_mutex\_held; @@ -254,6 +258,7 @@ ngx\_process\_events\_and\_timers(ngx\_cycle\_t \*cycle) } ``` if (ngx_accept_mutex_held) { ``` - ``` *ngx_accept_mutex_held_by = 0; ngx_shmtx_unlock(&ngx_accept_mutex); ``` } @@ -526,6 +531,9 @@ ngx\_event\_module\_init(ngx\_cycle\_t \*cycle) { return NGX\_ERROR; } - // cl = 128 bytes are available for us to use. ngx\_shmtx\_create uses - // ngx\_atomic\_t bytes to assign to mutex-\>lock, using the memory after that - ngx\_accept\_mutex\_held\_by = (ngx\_pid\_t\*) (shared + sizeof(ngx\_atomic\_t)); diff --git a/service/nginxServer/nginx-1.0.5/src/event/ngx\_event.h b/service/nginxServer/nginx-1.0.5/src/event/ngx\_event.h index 778da52…f1b06d4 100644 — a/service/nginxServer/nginx-1.0.5/src/event/ngx\_event.h +++ b/service/nginxServer/nginx-1.0.5/src/event/ngx\_event.h @@ -501,6 +501,7 @@ extern ngx\_atomic\_t \*ngx\_connection\_counter; extern ngx\_atomic\_t \*ngx\_accept\_mutex\_ptr; extern ngx\_shmtx\_t ngx\_accept\_mutex; +extern ngx\_pid\_t \*ngx\_accept\_mutex\_held\_by; extern ngx\_uint\_t ngx\_use\_accept\_mutex; extern ngx\_uint\_t ngx\_accept\_events; extern ngx\_uint\_t ngx\_accept\_mutex\_held; diff --git a/service/nginxServer/nginx-1.0.5/src/event/ngx\_event\_accept.c b/service/nginxServer/nginx-1.0.5/src/event/ngx\_event\_accept.c index 2355d1b…feb4568 100644 — a/service/nginxServer/nginx-1.0.5/src/event/ngx\_event\_accept.c +++ b/service/nginxServer/nginx-1.0.5/src/event/ngx\_event\_accept.c @@ -298,6 +298,10 @@ ngx\_trylock\_accept\_mutex(ngx\_cycle\_t \*cycle) ngx\_log\_debug0(NGX\_LOG\_DEBUG\_EVENT, cycle-\>log, 0, “accept mutex locked”); - ``` *ngx_accept_mutex_held_by = ngx_pid; ``` - - ``` // If the mutex was already held by me and we are using ``` RTSIG\_EVENT, no - ``` // need to enable accept_events if (ngx_accept_mutex_held && ngx_accept_events == 0 && !(ngx_event_flags & NGX_USE_RTSIG_EVENT)) ``` @@ -306,6 +310,8 @@ ngx\_trylock\_accept\_mutex(ngx\_cycle\_t \*cycle) } ``` if (ngx_enable_accept_events(cycle) == NGX_ERROR) { ``` - ``` // No one is holding the mutex now ``` - ``` *ngx_accept_mutex_held_by = 0; ngx_shmtx_unlock(&ngx_accept_mutex); return NGX_ERROR; } ``` @@ -317,8 +323,9 @@ ngx\_trylock\_accept\_mutex(ngx\_cycle\_t \*cycle) } ``` ngx_log_debug1(NGX_LOG_DEBUG_EVENT, cycle->log, 0, ``` - ``` "accept mutex lock failed: %ui", ``` ngx\_accept\_mutex\_held); - ``` "accept mutex lock failed: held by: %ui", ``` \*ngx\_accept\_mutex\_held\_by); - // If I held it earlier, but not anymore (ngx\_trylock\_accept\_mutex failed) if (ngx\_accept\_mutex\_held) { if (ngx\_disable\_accept\_events(cycle) == NGX\_ERROR) { return NGX\_ERROR; diff --git a/service/nginxServer/nginx-1.0.5/src/os/unix/ngx\_process.c b/service/nginxServer/nginx-1.0.5/src/os/unix/ngx\_process.c index 6055587…b66d4b3 100644 — a/service/nginxServer/nginx-1.0.5/src/os/unix/ngx\_process.c +++ b/service/nginxServer/nginx-1.0.5/src/os/unix/ngx\_process.c @@ -492,17 +492,18 @@ ngx\_process\_get\_status(void) } - ``` if (ngx_accept_mutex_ptr) { ``` - - ``` /* ``` - ``` * unlock the accept mutex if the abnormally exited ``` process - ``` * held it ``` - ``` */ ``` - - ``` ngx_atomic_cmp_set(ngx_accept_mutex_ptr, pid, 0); ``` - ``` // If the accept mutex is held by the abnormally exited ``` process - ``` // Note: If the process holding this has died, the mutex cannot ``` be - ``` // acquired by someone else, in which case, ``` ngx\_accept\_mutex\_held is - ``` // free to be accessed ``` - ``` if (ngx_accept_mutex_held_by != NULL && pid == ``` \*ngx\_accept\_mutex\_held\_by) { - ``` ngx_log_error(NGX_LOG_INFO, ngx_cycle->log, 0, ``` - ``` "PID %P held the accept mutex. Releasing", pid); ``` - ``` // Reset the value before unlocking ``` - ``` *ngx_accept_mutex_held_by = 0; ``` - ``` ngx_shmtx_unlock(&ngx_accept_mutex); } ``` - ``` one = 1; process = "unknown process"; ``` – 1.7.1 Posted at Nginx Forum: > **[Re: Nginx fails to accept new connection if active worker crashes](https://forum.nginx.org/read.php?2,218359,218425#msg-218425)** > > Hi All I use nginx configured with multiple workers. I also have an nginx module that crashed due to an error when I noticed that the module crash leaves nginx in a state where it cannot accept new calls. Removing my module and killing the...