# Nginx to use libatomic\_ops **URL:** \ **Category:** NGINX\ **Created:** [September 25, 2009, 9:54pm UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180 "2009-09-25T21:54:56Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [September 25, 2009, 9:54pm UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/1 "2009-09-25T21:54:56Z")
Dear developers, I have experienced SEGFAULTs on ARM using fastcgi and discovered it compiles with “NGX\_HAVE\_ATOMIC\_OPS 0” on ‘other’ architectures than x86, amd64, sparc and the such defined in src/os/unix/ngx\_atomic.h. Therefore I’d like to contribute the patch linked below [1], which introduces configure option “–with-libatomic” and which makes nginx use atomic operations of that library on these ‘other’ architectures. For more information on the library please see [2]. (Indeed, this does not result in additional runtime dependencies and the atomic ops compile to less code than those of newer GCC versions [3].) –with-libatomic compiles on ARM, does no SEGFAULT, and yields higher requests per second than with the otherwise used lock file. The option is disabled by default, and even if enabled still used as last resort. I hope that after a brief review (I have little experience with configure scripts) you can integrate it in the next release. – W-Mark K. > **[Ossdl](https://ossdl.de/)** > > Blog [1] [https://svn.hurrikane.de/all/ossdl/www-servers/nginx/files/nginx-0.8.16-libatomic.patch](https://svn.hurrikane.de/all/ossdl/www-servers/nginx/files/nginx-0.8.16-libatomic.patch) [2] [http://www.hpl.hp.com/research/linux/atomic\_ops/](http://www.hpl.hp.com/research/linux/atomic_ops/) [CVS Info for project bdwgc](http://bdwgc.cvs.sourceforge.net/viewvc/bdwgc/bdwgc/libatomic_ops-1.2/) [3] [Atomic Builtins - Using the GNU Compiler Collection (GCC)](http://gcc.gnu.org/onlinedocs/gcc-4.1.0/gcc/Atomic-Builtins.html) for gcc 4.1.0 and later ---
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [September 29, 2009, 10:27am UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/2 "2009-09-29T10:27:38Z")
On Fri, Sep 25, 2009 at 09:46:33PM +0200, W-Mark K. wrote: > ## not result in additional runtime dependencies and the atomic ops > > W-Mark K. > [http://mark.ossdl.de/](http://mark.ossdl.de/) > > [1] [https://svn.hurrikane.de/all/ossdl/www-servers/nginx/files/nginx-0.8.16-libatomic.patch](https://svn.hurrikane.de/all/ossdl/www-servers/nginx/files/nginx-0.8.16-libatomic.patch) > [2] [http://www.hpl.hp.com/research/linux/atomic\_ops/](http://www.hpl.hp.com/research/linux/atomic_ops/) > [http://bdwgc.cvs.sourceforge.net/viewvc/bdwgc/bdwgc/libatomic\_ops-1.2/](http://bdwgc.cvs.sourceforge.net/viewvc/bdwgc/bdwgc/libatomic_ops-1.2/) > [3] [Atomic Builtins - Using the GNU Compiler Collection (GCC)](http://gcc.gnu.org/onlinedocs/gcc-4.1.0/gcc/Atomic-Builtins.html) > for gcc 4.1.0 and later Thank you for the patch and information about gcc 4.1. I’m going to add gcc builtins, since they are slighty lesser than my code at least on x86. I’m not sure should libatomic be added, since gcc 4.1+ is common compiler these days. Could you show backtrace of the segfault ? I could not reproduce it on x86 with disabled atomic ops. ---
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [September 29, 2009, 1:37pm UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/3 "2009-09-29T13:37:57Z")
2009/9/29 Igor S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > Thank you for the patch and information about gcc 4.1. > I’m going to add gcc builtins, since they are slighty lesser than > my code at least on x86. > I’m not sure should libatomic be added, since gcc 4.1+ is common compiler > these days. > Could you show backtrace of the segfault ? I could not reproduce it > on x86 with disabled atomic ops. The segfaults happened on ARM architecture, not x86. Eliminating lock\_file by introducing atomic ops the only code which got not called after that were the emulations at the bottom of ngx\_atomic.h. GCC builtins work for me, too, but they make nginx be linked against a specific libgcc (that of the particular GCC version) and yield more overhead (2kb libatomic\_ops w/o any link vs. 13kb GCC builtins). Finally libatomic\_ops can be compiled by other means (such as MSVC, ICC, SUNC for example) therefore I prefer them for portability. What do you think about the compromise using GCC builtins if –with-libatomic is not set? Only architectures not covered by the #ifs would be affected and you could remove lock\_file code entirely in future. ---
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [October 8, 2009, 5:51pm UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/4 "2009-10-08T17:51:58Z")
On Tue, Sep 29, 2009 at 12:26:43PM +0200, W-Mark K. wrote: > > > > got not called after that were the emulations at the bottom of > ngx\_atomic.h. I understand that segfaults was happened on ARM. However, I tried to reproduce them on x86 with disabled atomic operations. > GCC builtins work for me, too, but they make nginx be linked against a > specific libgcc (that of the particular GCC version) and yield more > overhead (2kb libatomic\_ops w/o any link vs. 13kb GCC builtins). > Finally libatomic\_ops can be compiled by other means (such as MSVC, > ICC, SUNC for example) therefore I prefer them for portability. > > What do you think about the compromise using GCC builtins if > –with-libatomic is not set? > Only architectures not covered by the #ifs would be affected and you > could remove lock\_file code entirely in future. On i386/amd64 gcc builtins are just inlines and do not require libgcc. As to libatomic, I’m not sure now. I prefer to use complex libraries such as OpenSSL, zlib, etc. ---
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [October 9, 2009, 12:04pm UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/5 "2009-10-09T12:04:56Z")
2009/10/8 Igor S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid): > such as OpenSSL, zlib, etc. > On x86/amd64 the shipped assembler code works fine, and my libatomic > patch was intended for ‘other’ architectures. > Despite of the latter I believe reinventing the wheel (as did php-fpm) > is no good engineering - and you need to maintain that code yourself > rather than have the ‘community’ do that for you (and other projects). Unfortunately I don’t know the “complex” libraries good enough to tell whether they contain atomic ops nor whether they export it. Igor, if it was my project I would have deleted the asm part completely and used GCC atomics (the same goes for php-fpm) where available, and libatomic (or any better replacement, I am not dogmatic here) else. As far as I can tell the latter are inlined, too. In the meantime I’ve created modified Gentoo ebuilds [1] (for all archs) and binaries [2] (for ARM) - with the option for libatomic. Find it attached as the link mentioned previously does not work anymore because I replaced subversion by GIT. – W-Mark K. > **[Open Source Software Entwicklung Heidelberg | OSSDL](https://ossdl.de/)** > > OSSDL — Open Source Software Development Labs Heidelberg. Softwareentwicklung, Architektur, Linux und DevOps auf Basis offener Technologien. [1] [http://git.ossdl.de/?p=ossdl-overlay.git;a=summary](http://git.ossdl.de/?p=ossdl-overlay.git;a=summary) [2] [Open Source Software Entwicklung Heidelberg | OSSDL](http://binhost.ossdl.de/armv5tel-softfloat-linux-gnueabi/) (set this as binhost in Gentoo and the packages will be pulled automatically) ---
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [October 9, 2009, 1:29pm UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/6 "2009-10-09T13:29:13Z")
Hello! On Fri, Oct 09, 2009 at 11:57:24AM +0200, W-Mark K. wrote: > > As to libatomic, I’m not sure now. I prefer to use complex libraries > > such as OpenSSL, zlib, etc. > > On x86/amd64 the shipped assembler code works fine, and my libatomic > patch was intended for ‘other’ architectures. > Despite of the latter I believe reinventing the wheel (as did php-fpm) > is no good engineering - and you need to maintain that code yourself > rather than have the ‘community’ do that for you (and other projects). > > Unfortunately I don’t know the “complex” libraries good enough to tell > whether they contain atomic ops nor whether they export it. The idea is that for simple and rarely changing things like atomic ops it’s better to have them in-place instead of in external library. As external library may change, broke, disappear and so on, as a result requiring much more attention then simple in-place code. Note well: I believe the actual problem isn’t atomic ops that isn’t here. By design nginx should work without atomic ops. The real problem is SIGSEGV you see without them. It should be fixed, but unfortunately we have no hardware to reproduce the problem. Maxim D. ---
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [October 9, 2009, 3:13pm UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/7 "2009-10-09T15:13:24Z")
On Fri, Oct 09, 2009 at 11:57:24AM +0200, W-Mark K. wrote: > > As to libatomic, I’m not sure now. I prefer to use complex libraries > > – > W-Mark K. > [http://mark.ossdl.de/](http://mark.ossdl.de/) > > [1] [http://git.ossdl.de/?p=ossdl-overlay.git;a=summary](http://git.ossdl.de/?p=ossdl-overlay.git;a=summary) > [2] [Open Source Software Entwicklung Heidelberg | OSSDL](http://binhost.ossdl.de/armv5tel-softfloat-linux-gnueabi/) > (set this as binhost in Gentoo and the packages will be pulled > automatically) I agree that reinventing wheel is not good engineering and I prefer to use libraries, however, in reason. I want to minimize external dependances for general usage as much as possible. nginx may depend on OpenSSL, libXSLT/libXML, perl, libGD, GeoIP, but usually it requires just zlib and PCRE. And you can build nginx even without these libraries. I’m not sure that community maintenance is always better. PCRE and zlib support is certainly very good, however, I can say this about any library. As to gcc 4.1 builtin/libatomic only solution, now nginx can be built by gcc 2.8 and 2.95, and I do not think that installing libatomic on FreeBSD 4 (gcc 2.95) is necessary thing to run nginx. Besides, for example, MacOSX/Darwin has atomic(3) and barrier(3) operations optimized at run-time for UP and SMP (they does to use lock on UP). I will add libatomic as last resort (gcc4.1 built-ins, my atomic code, libatomic) and will add building procedures since it seems Linux distributions are the only platform that has libatomic package. BTW, could you please show coredump backtrace of segfault without atomic ops ? I still want to fix the bug to allow nginx run even on platforms without atomic ops at all. ---
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [October 12, 2009, 3:01am UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/8 "2009-10-12T03:01:08Z")
Hello! On Tue, Sep 29, 2009 at 12:26:43PM +0200, W-Mark K. wrote: > > > > got not called after that were the emulations at the bottom of > ngx\_atomic.h. Just for the record: I’ve tried to reproduce the problem on QEMU-emulated ARM but failed. Everything works fine for me. Could you please provide some more details? Hardware, OS, nginx -V output, config and request pattern which triggers fault, backtrace, etc… Maxim D. ---
**Author:** ![W-Mark\_K](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/w/b9bd4f/32.png) [@W-Mark\_K](https://www.ruby-forum.com/u/W-Mark_K)\ **Post date:** [October 12, 2009, 1:25pm UTC](https://www.ruby-forum.com/t/nginx-to-use-libatomic-ops/176180/9 "2009-10-12T13:25:43Z")
Hello! I will send you the debug informations asap, but unfortunately no sooner than by the end of this week. Let me open a separate thread for this, then, with more details. From what I remember, lockfile was on /dev/shm, vanilla configuration files. Nginx was compiled with zlib and flv, only the former was actually used and later on even none of them. Segfaults began as soon as concurrent requests occured (e.g. by ab -c 5, one to two workers, epoll), even to static files. The logfile was flooded by “worker… segfault” for every (unhandled due to the fault) request. (Linux 2.6.29-rc5, 2.6.30; deadlock scheduler)