# Call ngx\_localtime may cause deadlock? **URL:** \ **Category:** NGINX\ **Created:** [March 4, 2010, 11:10am UTC](https://www.ruby-forum.com/t/call-ngx-localtime-may-cause-deadlock/184550 "2010-03-04T11:10:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![georgexsh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/43a26b/32.png) [@georgexsh](https://www.ruby-forum.com/u/georgexsh)\ **Post date:** [March 4, 2010, 11:10am UTC](https://www.ruby-forum.com/t/call-ngx-localtime-may-cause-deadlock/184550/1 "2010-03-04T11:10:02Z")
I am writing a module based on nginx 0.7.65, ngx\_localtime() is called in the module. Problem is, if set “timer\_resolution 100ms;” in config file, nginx workers(\*4) will hangs randomly. then if “timer\_resolution” command was turned off , everything works just fine. here is the backtrace (coredump generated with gcore(1) on a hanged worker process): #0 0x007347a2 in ?? () #1 0x00823a9e in pthread\_cond\_init@@GLIBC\_2.3.2 () from /lib/tls/libc.so.6 #2 0x007c942d in \_\_wcsmbs\_named\_conv () from /lib/tls/libc.so.6 #3 0x080611c6 in ngx\_localtime (s=1267569890, tm=0x9ba2048) at src/os/unix/ngx\_time.c:59 #4 0x08055223 in ngx\_time\_update (sec=1267569890, msec=5) at src/core/ngx\_times.c:114 #5 0x0805e51a in ngx\_timer\_signal\_handler (signo=14) at src/event/ngx\_event.c:565 #6 0x00775898 in \_\_open\_catalog () from /lib/tls/libc.so.6 #7 0x007cc043 in \_\_tzfile\_read () from /lib/tls/libc.so.6 #8 0x007cb217 in tzset\_internal () from /lib/tls/libc.so.6 #9 0x007c942d in \_\_wcsmbs\_named\_conv () from /lib/tls/libc.so.6 #10 0x080611c6 in ngx\_localtime (s=1267569889, tm=0x875ff4) at src/os/unix/ngx\_time.c:59 #11 0x08088c9f in ngx\_foo\_time\_str (r=0x9dcdc28, time\_str=0xbfe18450, t=-4) at ./ngx\_foo\_module//uti.c:111 #12 0x08087d1a in ngx\_http\_foo\_record (r=0x9dcdc28, request\_arg=0x9dcddb0) at ./ngx\_foo\_module//foo.c:124 #13 0x0808788c in ngx\_http\_foo\_handler (r=0x9dcdc28) at ./ngx\_foo\_module//ngx\_http\_foo\_module.c:74 #14 0x0806d4f9 in ngx\_http\_core\_content\_phase (r=0x9dcdc28, ph=0x9bb9938) at src/http/ngx\_http\_core\_module.c:1262 strace shows (not on the same worker, but similar) : futex(0x2c2eec, FUTEX\_WAIT, 2, NULL) I guess this situation is a deadlock, happens beneath localtime\_r() of glibc. While my module is calling localtime\_r, a time update signal trigged localtime at the same time, then deadlocked. Is this correct? Now I employ ngx\_gmtime() to get formated time instead of ngx\_localtime(), works stable for weeks. Maybe ngx\_localtime() should not be used outside ngx\_time\_update(). Posted at Nginx Forum: > **[call ngx\_localtime may cause deadlock?](https://forum.nginx.org/read.php?2,59851,59851#msg-59851)** > > I am writing a module based on nginx 0.7.65, ngx\_localtime() is called in the module. Problem is, if set "timer\_resolution 100ms;" in config file, nginx workers(\*4) will hangs randomly. then if "timer\_resolution" command was... ---
**Author:** ![georgexsh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/43a26b/32.png) [@georgexsh](https://www.ruby-forum.com/u/georgexsh)\ **Post date:** [March 4, 2010, 1:22pm UTC](https://www.ruby-forum.com/t/call-ngx-localtime-may-cause-deadlock/184550/2 "2010-03-04T13:22:55Z")
On Thu, Mar 04, 2010 at 03:14:53PM +0300, Maxim D. wrote: > > #7 0x007cc043 in \_\_tzfile\_read () from /lib/tls/libc.so.6 > > eventport) nginx uses localtime\_r() in signal handler. This isn’t > it’s not glibc’s fault. > Here is third-party module which probably uses threads: #11 0x08088c9f in ngx\_foo\_time\_str (r=0x9dcdc28, time\_str=0xbfe18450, t=-4) at ./ngx\_foo\_module//uti.c:111 #12 0x08087d1a in ngx\_http\_foo\_record (r=0x9dcdc28, request\_arg=0x9dcddb0) at ./ngx\_foo\_module//foo.c:124 #13 0x0808788c in ngx\_http\_foo\_handler (r=0x9dcdc28) at ./ngx\_foo\_module//ngx\_http\_foo\_module.c:74 > > Now I employ ngx\_gmtime() to get formated time instead of > > ngx\_localtime(), works stable for weeks. Maybe ngx\_localtime() > > should not be used outside ngx\_time\_update(). > > You should use ngx\_time()/ngx\_timeofday() instead in your module. – Igor S. > **[Igor Sysoev](http://sysoev.ru/en/)** ---
**Author:** ![georgexsh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/43a26b/32.png) [@georgexsh](https://www.ruby-forum.com/u/georgexsh)\ **Post date:** [March 4, 2010, 1:15pm UTC](https://www.ruby-forum.com/t/call-ngx-localtime-may-cause-deadlock/184550/3 "2010-03-04T13:15:18Z")
Hello! On Thu, Mar 04, 2010 at 05:09:33AM -0500, georgexsh wrote: > #0 0x007347a2 in ?? () > #11 0x08088c9f in ngx\_foo\_time\_str (r=0x9dcdc28, time\_str=0xbfe18450, t=-4) at ./ngx\_foo\_module//uti.c:111 > localtime\_r() of glibc. > While my module is calling localtime\_r, a time update signal > trigged localtime at the same time, then deadlocked. Is this > correct? Yes, your analysis is correct. With timer\_resolution set and for event methods which does not support timers (all but kqueue, eventport) nginx uses localtime\_r() in signal handler. This isn’t generally safe thing to do as POSIX doesn’t list it as async-signal-safe function. Any other usage of localtime\_r() in the same process may cause deadlock. In fact it may even deadlock by itself as soon as new signal arrives before localtime\_r() will finish execution (though unlikely). I’m a bit surprised that this actually happens as nginx isn’t threaded and I see no reason for glibc to use locks. But anyway it’s not glibc’s fault. > Now I employ ngx\_gmtime() to get formated time instead of > ngx\_localtime(), works stable for weeks. Maybe ngx\_localtime() > should not be used outside ngx\_time\_update(). You should use ngx\_time()/ngx\_timeofday() instead in your module. Maxim D. ---
**Author:** ![georgexsh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/43a26b/32.png) [@georgexsh](https://www.ruby-forum.com/u/georgexsh)\ **Post date:** [March 4, 2010, 5:50pm UTC](https://www.ruby-forum.com/t/call-ngx-localtime-may-cause-deadlock/184550/4 "2010-03-04T17:50:26Z")
Hello! On Thu, Mar 04, 2010 at 10:20:29AM -0500, georgexsh wrote: > > (r=0x9dcdc28, request\_arg=0x9dcddb0) at > > [http://blogs.inxsasia.com/hiran/2009/07/stupid-futex-and-glibc.html](http://blogs.inxsasia.com/hiran/2009/07/stupid-futex-and-glibc.html) > > [I Challenge Thee](http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=301511) Ok, second link suggest that recent glibc uses futex even when there are no threads in application. This isn’t really relevant though as underlying problem is in nginx and should be fixed anyway. Only async-signal-safe functions may be used in signal handlers, anything else leads to undefined behaviour. For now - use ngx\_time() and don’t use timer\_resolution unless you have kqueue or eventport. Maxim D. ---
**Author:** ![georgexsh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/43a26b/32.png) [@georgexsh](https://www.ruby-forum.com/u/georgexsh)\ **Post date:** [March 4, 2010, 8:55pm UTC](https://www.ruby-forum.com/t/call-ngx-localtime-may-cause-deadlock/184550/5 "2010-03-04T20:55:27Z")
On Thu, Mar 04, 2010 at 07:49:57PM +0300, Maxim D. wrote: > > > #11 0x08088c9f in ngx\_foo\_time\_str (r=0x9dcdc28, > > nginx and should be fixed anyway. Only async-signal-safe > functions may be used in signal handlers, anything else leads to > undefined behaviour. Strickly speaking the generic signal handler calls localtime/\_r() too: ngx\_time\_update(0, 0); And if timer\_resolution bug can be fixed easy, nevertheless, I do not know how to log local time in singal handler without localtime(). – Igor S. > **[Igor Sysoev](http://sysoev.ru/en/)** ---
**Author:** ![georgexsh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/43a26b/32.png) [@georgexsh](https://www.ruby-forum.com/u/georgexsh)\ **Post date:** [March 5, 2010, 1:59am UTC](https://www.ruby-forum.com/t/call-ngx-localtime-may-cause-deadlock/184550/6 "2010-03-05T01:59:51Z")
Hello! On Thu, Mar 04, 2010 at 10:54:56PM +0300, Igor S. wrote: > On Thu, Mar 04, 2010 at 07:49:57PM +0300, Maxim D. wrote: […] > > This isn’t really relevant though as underlying problem is in > > nginx and should be fixed anyway. Only async-signal-safe > > functions may be used in signal handlers, anything else leads to > > undefined behaviour. > > Strickly speaking the generic signal handler calls localtime/\_r() too: > ngx\_time\_update(0, 0); > > And if timer\_resolution bug can be fixed easy, nevertheless, I do not > know how to log local time in singal handler without localtime(). Well, I see 3 basic options: 1. Handle signals in the event loop (at least kqueue supports it, for other methods we may use pipe() to deliver signals to event loop safely). This is basically how libevent does this, btw. 2. Limit signal handlers to use only async-signal-safe functions, and nothing more. This means that time won’t be correct (no localtime()), errno (if any) won’t be translated to text (no strerror\_r()) etc. 3. Pretend we never talked about it and problems shouldn’t happen as generic signals are rare (and we already try to follow (2), though not completely). Maxim D. ---
**Author:** ![georgexsh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/43a26b/32.png) [@georgexsh](https://www.ruby-forum.com/u/georgexsh)\ **Post date:** [March 4, 2010, 4:20pm UTC](https://www.ruby-forum.com/t/call-ngx-localtime-may-cause-deadlock/184550/7 "2010-03-04T16:20:52Z")
## Igor S. Wrote: > (r=0x9dcdc28) at > ./ngx\_foo\_module//ngx\_http\_foo\_module.c:74 No, ngx\_foo\_module/\* is my module which does’t use threads ethier. ref: [http://blogs.inxsasia.com/hiran/2009/07/stupid-futex-and-glibc.html](http://blogs.inxsasia.com/hiran/2009/07/stupid-futex-and-glibc.html) > **[\#301511 - sysklogd: hangs the whole system - Debian Bug report logs](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=301511)** Posted at Nginx Forum: [http://forum.nginx.org/read.php?2,59851,59942#msg-59942](http://forum.nginx.org/read.php?2,59851,59942#msg-59942)