# "Hacking" the event model of Nginx **URL:** \ **Category:** NGINX\ **Created:** [April 12, 2008, 12:20pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582 "2008-04-12T12:20:56Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 12, 2008, 12:20pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/1 "2008-04-12T12:20:56Z")
Hi All! I’m new on this nice discussion list but I’m a long time lurker. I’m working on a very specific module for Nginx: a complete CMS. It may sounds strange since upstream servers and scripting languages are the norm for the CMS, but I want (and need) speed. So far I’ve done many things without too much trouble, but I’m a little bit stuck with the processing of events in Nginx. I would like to process a specific event which is not connection related but created by one of my worker threads. I hope this short example will be clear: // My specific Nginx http handler int my\_http\_handler (ngx\_request\_t \* r) { if (r-\>my\_state == 0) // first step: initiate the work to do { r-\>my\_state ++ ; my\_sendmsg (myqueue,r) ; // send a message to worker ``` return NGX_AGAIN ; // please call me back when done ``` } else // second step: results are ready { // produce xhtml output from results return NGX\_OK ; // finished } } // The worker running on a specific thread void my\_http\_worker (void \* arg) { ngx\_http\_request\_t \* r ; ngx\_event\_t \* ev ; while (1) { my\_recvmsg (myqueue,r) ; // processing the request // … // wake up my\_http\_handler ngx\_post\_event (ev, (ngx\_event\_t \* \*) & ngx\_posted\_events) ; } } But I don’t know how to fill the ngx\_event\_t (in particular the handlers) in order to call again my\_http\_handler on Nginx’s context. I believe it’s possible to do so from what I’ve seen, but Nginx’s code is not so easy to enter on (it’s not a critic). Sorry for this (first) long message but I think it would be nice to be able to develop clean and non blocking modules for Nginx. BTW forgive my English, I’m French 😉 ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 13, 2008, 6:27am UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/2 "2008-04-13T06:27:08Z")
Manlio P. \ writes: > If you need “speed”, try my WSGI module. > [http://hg.mperillo.ath.cx/nginx/mod\_wsgi/](http://hg.mperillo.ath.cx/nginx/mod_wsgi/) First, thanks a lot for your reply. Well, I believe Python is a great language but I will keep C (I was considering using assembly language!) because I will be able to have _zero copy_ between my database buffers and my xml to xhtml translation, I already know where the bottleneck is and that’s why I made this choice. > Give at look at: > [http://hg.mperillo.ath.cx/nginx/mod\_wsgi/file/tip/src/ngx\_wsgi.c](http://hg.mperillo.ath.cx/nginx/mod_wsgi/file/tip/src/ngx_wsgi.c) > row 850. Well, I’m not too sure it’s relevant to my case but I will investigate in depth; thanks to share your knowledge. > Is my\_http\_worker in a separate thread? > Then this will not work, Nginx is not thread safe. Yes there are separate threads launched for each Nginx worker process, it’s not an issue as a request is linked to a worker process, just a matter of mutex; nothing to scare me! The mix using non preemptive model and worker thread is very interesting but not so easy to do. Thanks for your kind reply, Best regards ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 12, 2008, 1:10pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/3 "2008-04-12T13:10:16Z")
François Battail ha scritto: > Hi All! > > I’m new on this nice discussion list but I’m a long time lurker. > > I’m working on a very specific module for Nginx: a complete CMS. It may > sounds strange since upstream servers and scripting languages are the > norm for the CMS, but I want (and need) speed. If you need “speed”, try my WSGI module. [http://hg.mperillo.ath.cx/nginx/mod\_wsgi/](http://hg.mperillo.ath.cx/nginx/mod_wsgi/) It enables you to write Python applications embedded in Nginx. Of course there are some overheads, but usually the problems are elsewhere. [http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all](http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all) Python is not really that bad, unless, of course, you start to use a lot of for cycles or recursion (but in this case you can always write the function in C). > So far I’ve done many things without too much trouble, but I’m a little > bit stuck with the processing of events in Nginx. I would like to > process a specific event which is not connection related but created by > one of my worker threads. Give at look at: [http://hg.mperillo.ath.cx/nginx/mod\_wsgi/file/tip/src/ngx\_wsgi.c](http://hg.mperillo.ath.cx/nginx/mod_wsgi/file/tip/src/ngx_wsgi.c) row 850. You need to obtain a valid file descriptor `s`, then call: c = ngx\_get\_connection(s, log); The event for read and write notifications are: c-\>read and c-\>write Note however that in my code, the file descriptor is assumed as already “connected” to the peer. > ``` > return NGX_AGAIN ; // please call me back when done > > ``` > > { > } > } > > But I don’t know how to fill the ngx\_event\_t (in particular the > handlers) in order to call again my\_http\_handler on Nginx’s context. > I believe it’s possible to do so from what I’ve seen, but Nginx’s code > is not so easy to enter on (it’s not a critic). Is my\_http\_worker in a separate thread? Then this will not work, Nginx is not thread safe. > Sorry for this (first) long message but I think it would be nice to be > able to develop clean and non blocking modules for Nginx. > > BTW forgive my English, I’m French 😉 Regards Manlio P. ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 13, 2008, 11:15am UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/4 "2008-04-13T11:15:12Z")
François Battail ha scritto: > Ok. > > Give at look at: > > [http://hg.mperillo.ath.cx/nginx/mod\_wsgi/file/tip/src/ngx\_wsgi.c](http://hg.mperillo.ath.cx/nginx/mod_wsgi/file/tip/src/ngx_wsgi.c) > > row 850. > > Well, I’m not too sure it’s relevant to my case but I will investigate in depth; > thanks to share your knowledge. What type of connection do you want to create? > > Is my\_http\_worker in a separate thread? > > Then this will not work, Nginx is not thread safe. > > Yes there are separate threads launched for each Nginx worker process, it’s not > an issue as a request is linked to a worker process, just a matter of mutex; > nothing to scare me! The problem is with: ngx\_post\_event as far as I know it is not thread safe, unless you enable threads support in Nginx (but in current version it is broken). > The mix using non preemptive model and worker thread is > very interesting but not so easy to do. What are you trying to do? > Thanks for your kind reply, > > Best regards Regards Manlio P. ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 13, 2008, 3:23pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/5 "2008-04-13T15:23:04Z")
Manlio P. \ writes: > What type of connection do you want to create? I don’t want to create a connection, just to split the processing of a HTTP request in two without blocking Nginx. HTTP request -\> handler send a message to one of my workers returns NGX\_AGAIN -\> worker (running on an independant thread) wait for a message do the job (using blocking calls) “wake up” the HTTP handler with a Nginx event -\> handler process the worker’s results finalize the HTTP reply Hope it’s more clear written that way. > as far as I know it is not thread safe, unless you enable threads > support in Nginx (but in current version it is broken). Yes, you’re correct. But I don’t want to activate thread support in Nginx, I would like to create my specific threads _inside_ a Nginx worker. I just need to protect the event queue of Nginx from my threads with a mutex as far I’ve seen. > > The mix using non preemptive model and worker thread is > > very interesting but not so easy to do. > > What are you trying to do? Simply to call blocking functions during a HTTP request without blocking Nginx and without using an upstream server. In my project the database (Sqlite3) is embedded into the web server and share commons memory pools, that way I can process XML documents stored in the database without allocating or copying data but only by using a list of fragment buffers for the reply which fit nicely with ngx\_chain\_t. I hope it will be very fast to process a dynamic request. Your code is a great help for me, thank you. Best regards ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 13, 2008, 4:13pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/6 "2008-04-13T16:13:15Z")
François Battail ha scritto: > send a message to one of my workers > returns NGX\_AGAIN > > -\> worker (running on an independant thread) > wait for a message > do the job (using blocking calls) > “wake up” the HTTP handler with a Nginx event Two questions: - how many threads do you have for each worker process? - how do you plan to implement the “wake up”? > > > matter of mutex; nothing to scare me! No. You need to protect the event queue not only from your threads, but also from the main thread of the worker process, and here you don’t have any control. > for the reply which fit nicely with ngx\_chain\_t. I hope it will be very > fast to process a dynamic request. > > Your code is a great help for me, thank you. You’re welcome. > Best regards Regards Manlio P. ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 13, 2008, 5:56pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/7 "2008-04-13T17:56:17Z")
Manlio P. \ writes: > Two questions: > > - how many threads do you have for each worker process? > - how do you plan to implement the “wake up”? I believe that 4 threads by worker should be sufficient but I need to do some benchmarks. For the second question, the wake up will be done by injecting a “fake” Nginx event into the queue, that’s my main problem and that’s why I’ve used the word “hacking” in the title of this subject. > No. > You need to protect the event queue not only from your threads, but also > from the main thread of the worker process, and here you don’t have any > control. I will modify Nginx to ensure it will use the same mutex as my threads when accessing the event queue. In fact it’s already in Nginx’s code (ngx\_posted\_event\_mutex) but only active when compiling with the NGX\_THREADS directive. My project is a very specific module, so I will do any modification required in Nginx’s code. Best regards ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 15, 2008, 7:21am UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/8 "2008-04-15T07:21:43Z")
Le lundi 14 avril 2008 à 16:27 +0200, Manlio P. a écrit : > Note that this it not sufficient to “wake up” the main thread. > In fact, the main event loop can be waiting on the select/poll/epoll/kqueue. Yes, you’re right. I’ve rewritten the epoll module to have the epoll\_wait calls on a separate thread and to have the main event loop waiting on a queue which receive the epoll events and the events from my threads. But as the initial request can be cancelled while still being processed by my threads or still in the queue, some care is needed before destroying the request structure. Best regards ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 14, 2008, 4:37pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/9 "2008-04-14T16:37:31Z")
François Battail ha scritto: > Manlio P. \ writes: > > > Two questions: > > > > - how many threads do you have for each worker process? > > - how do you plan to implement the “wake up”? > > I believe that 4 threads by worker should be sufficient but I need to do some > benchmarks. For the second question, the wake up will be done by injecting a > “fake” Nginx event into the queue, Note that this it not sufficient to “wake up” the main thread. In fact, the main event loop can be waiting on the select/poll/epoll/kqueue. A solution is to use a pipe: the read end is registered in the event loop, and the write end is used by threads to wake the main event loop. Manlio P. ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 15, 2008, 10:32am UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/10 "2008-04-15T10:32:44Z")
On Die 15.04.2008 07:16, François Battail wrote: > Le lundi 14 avril 2008 à 16:27 +0200, Manlio P. a écrit : > > > Note that this it not sufficient to “wake up” the main thread. > > In fact, the main event loop can be waiting on the select/poll/epoll/kqueue. > > Yes, you’re right. I’ve rewritten the epoll module to have the > epoll\_wait calls on a separate thread and to have the main event loop > waiting on a queue which receive the epoll events and the events from > my threads. Have you taken a look into the libev [http://software.schmorp.de/pkg/libev.html](http://software.schmorp.de/pkg/libev.html) or libevent [libevent](http://monkey.org/~provos/libevent/) before you have implement your own epoll handlers 😉 cheers Aleks ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [December 29, 2010, 2:43am UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/11 "2010-12-29T02:43:51Z")
François Battail wrote in post #660685: > Aleksandar L. \ writes: > > > Have you taken a look into the libev > > [http://software.schmorp.de/pkg/libev.html](http://software.schmorp.de/pkg/libev.html) or libevent > > [libevent](http://monkey.org/~provos/libevent/) before you have implement your own > > epoll handlers > > Thank you for these hints, but I’ve already done this layer! I’ve > bookmarked > libev, maybe for an another project. > > Best regards Is the implement of yours open？ I want to see how do you rewrite the epoll module，thanks ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [April 15, 2008, 12:44pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/12 "2008-04-15T12:44:50Z")
Aleksandar L. \ writes: > Have you taken a look into the libev > [libev](http://software.schmorp.de/pkg/libev.html) or libevent > [libevent](http://monkey.org/~provos/libevent/) before you have implement your own > epoll handlers Thank you for these hints, but I’ve already done this layer! I’ve bookmarked libev, maybe for an another project. Best regards ---
**Author:** ![FranSSSSois\_Battail](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/f/8e8cbc/32.png) [@FranSSSSois\_Battail](https://www.ruby-forum.com/u/FranSSSSois_Battail)\ **Post date:** [December 9, 2012, 12:26pm UTC](https://www.ruby-forum.com/t/hacking-the-event-model-of-nginx/134582/13 "2012-12-09T12:26:14Z")
> // My specific Nginx http handler > int my\_http\_handler (ngx\_request\_t \* r) > { > if (r-\>my\_state == 0) // first step: initiate the work to do I’m trying to do something similar, however r-\>my\_state is unknown here. Did you create your own struct?