Implementing a non-blocking delay between request and response

Server NGINX
1

Hi there,

I’m trying to figure out how to implement a non blocking delay between
when a request comes into a module, and when it is responded too.

For example, consider a blocking delay:

static ngx_int_t
ngx_http_my_module_handler(ngx_http_request_t *r)
{
r->headers_out.status = NGX_HTTP_OK
r->header_only = 1;
r->headers_out.content_length_n = 0;

ngx_sleep(10);

rc = ngx_http_send_header(r);
ngx_http_finalize_request(r, rc);
return NGX_DONE;
}

Which would allow a client to connect, wait for 10 seconds, and then
send the response. That’s the behaviour I want, but such a wait as
implemented here is blocking, that is, while the first request is
waiting to be served, all the other requests are blocked until it
completed. I want to make it non blocking so that while one is being
processed (and waiting), others can come into the system and be handled
(which may themselves wait, or may not).

So after some research, I found out about nginx timers, and it seems I
should be able to set a timer/event hander to implement the wait and be
non-blocking:

static void ngx_http_my_module_delay_handler(ngx_event_t *ev);

static ngx_int_t
ngx_http_my_module_handler(ngx_http_request_t *r)
{
r->headers_out.status = NGX_HTTP_OK
r->header_only = 1;
r->headers_out.content_length_n = 0;

ngx_event_t *wakeupFromSleep;
wakeupFromSleep = ngx_pcalloc(r->pool, sizeof(ngx_event_t));

if (wakeupFromSleep == NULL)
ngx_log_stderr(0, “Wakeupfromsleep null”);

wakeupFromSleep->handler = ngx_http_my_module_delay_handler;
wakeupFromSleep->data = r;
ngx_log_stderr(0, “Sleeping for 30 seconds”);
ngx_add_timer(wakeupFromSleep, 30);

return NGX_DONE;
}

static void
ngx_http_my_module_delay_handler(ngx_event_t *ev)
{
int rc;
ngx_http_request_t *r = ev->data;
ngx_log_stderr(0, “Done waiting. Sending response.”);

rc = ngx_http_send_header(r);
ngx_http_finalize_request(r, rc);
ngx_del_timer(ev);
}

Unfortunately, when I connect (say with telnet), and send my request,
the connection immediately closes instead of waiting. In addition, I
never see the “Done waiting. Sending response.” message, indicating to
me that the event isn’t waking up.

My guess as to why the connection is closing immediately has to do with
my returning NGX_DONE in ngx_http_my_module_handler, but I can’t figure
out what else to return that would allow it not to close, and wait for
the timer. I can’t figure out why the timer event isn’t raising, though
I suspect it has to do with the connection immediately returning. I
think it’s just something I’m missing in the setup, but I’m not sure (My
view of how timers should work like this could also be flawed).

Any advice you have would be helpful! Thanks.

Posted at Nginx Forum:

2

Hi,

Tronman wrote:

r->headers_out.status = NGX_HTTP_OK

r->headers_out.status = NGX_HTTP_OK
r->header_only = 1;
r->headers_out.content_length_n = 0;

ngx_event_t *wakeupFromSleep;
wakeupFromSleep = ngx_pcalloc(r->pool, sizeof(ngx_event_t));

You should use ngx_add_event () here instead of allocating the space for
the event struct directly.

if (wakeupFromSleep == NULL)
ngx_log_stderr(0, “Wakeupfromsleep null”);

As a side-note here, this should have included returning a failure - you
would have got a segfault on failure otherwise

if (wakeupFromSleep == NULL) {
ngx_log_stderr(0, “Wakeupfromsleep null”);
return NGX_ERROR;
}

wakeupFromSleep->handler = ngx_http_my_module_delay_handler;
wakeupFromSleep->data = r;
ngx_log_stderr(0, “Sleeping for 30 seconds”);
ngx_add_timer(wakeupFromSleep, 30);

If you do nothing else here, then your request will return immediately.
(Check out the code in ngx_http_finalize_request() to understand why.)
Adding

r->count++;

is one way that should prevent it from returning the first time.

rc = ngx_http_send_header®;
ngx_http_finalize_request(r, rc);
ngx_del_timer(ev);

You should add a ngx_del_event() too here. I’m not sure whether you
should delete the event and timer before finalizing the request (I don’t
think so), but it may be required to avoid segfaulting.

}
By the way, this question is probably better asked in the nginx-devel
mailing list rather than the main one.

Hope that helps,

Marcus.

3

On Fri, Jan 15, 2010 at 3:26 AM, Tronman [email protected] wrote:

Hi there,

I’m trying to figure out how to implement a non blocking delay between when a request comes into a module, and when it is responded too.

Heh, it reminds me of the echo_sleep and echo_blocking_sleep
directives in the ngx_echo module:

http://wiki.nginx.org/NginxHttpEchoModule#echo_sleep

The former is a non-blocking sleep using custom events and timers.
There’s a known issue though. I should have registered a request
cleanup handler to remove my own timers when requests abort
abnormally.

Hope it helps.

Cheers,
-agentzh

P.S. I’ve cc’d the nginx-devel mailing list.

4

On Fri, Jan 15, 2010 at 9:47 AM, agentzh [email protected] wrote:

There’s a known issue though. I should have registered a request
cleanup handler to remove my own timers when requests abort
abnormally.

My bad. I’ve just added such a test case to ngx_echo’s test suite and
found that nginx won’t check broken downstream connections by default
(at least for 0.8.32 and 0.7.64). So even if the client shut the
connection prematurely, nginx will go on waiting for our timers get
properly expired and quit the current session normally. If we do want
to quit eagerly we will have to register our own read/write event
handler to check broken connections ourselves, just like the
ngx_http_upstream_rd_check_broken_connection function used by
ngx_http_upstream.

Anyway, I’ve added cleanup handler to ngx_echo in my local checkout
and it will appear in mainstream soon.

Cheers,
-agentzh

5

Hi folks, thanks for your responses! As a quick note, I’ve been using
ngxin 0.7.64

Marcus C. wrote:

As a side-note here, this should have included returning a failure - you
would have got a segfault on failure otherwise

For sure, I’m just experimenting for now, not too much error handling.

You should use ngx_add_event () here instead of allocating the space for
the event struct directly.

I spent some time looking at ngx_event.h, but can’t figure out the
semantics of that call. It doesn’t look like it actually allocates space
for the event that I can tell, so I still have to pcalloc. Even then, I
tried ngx_add_event(wakeupFromSleep, NGX_ONESHOT_EVENT, 0) (and READ,
WRITE events), but it still just seg faults.

If you do nothing else here, then your request will return immediately.
(Check out the code in ngx_http_finalize_request() to understand why.)
Adding

r->count++;

is one way that should prevent it from returning the first time.

Since I’m using 0.7.64 (sorry for not mentioning that sooner), my
ngx_http_request doesn’t have a count element. Is there something
similar to prevent return in 0.7.64?

Also, I know ngx_http_finalize_request() never even gets called after I
add the timer and the request returns. So I don’t see how it could be
causing it to return immediately.

You should add a ngx_del_event() too here.

Yep, but I have to figure out how to use add event first.

By the way, this question is probably better asked in the nginx-devel
mailing list rather than the main one.

Sorry, it can be tricky to figure out where something should go, and I
don’t have the mailing list set up.
Thanks for the CC agentzh.

agentzh wrote:

Heh, it reminds me of the echo_sleep and echo_blocking_sleep
directives in the ngx_echo module:

I took a look at that module, and it does seem somewhat similar, except
I want the sleep to be specifiable from the code, not the configuration
file. I’m not sure what’s being done differently to implement it that
I’m not doing.

Thanks again.

Posted at Nginx Forum:

6

agentzh wrote:

Heh, it reminds me of the echo_sleep and echo_blocking_sleep
directives in the ngx_echo module:

Tronman wrote:

I took a look at that module, and it does seem somewhat similar, except I want the sleep to be specifiable >from the code, not the configuration file. I’m not sure what’s being done differently to implement it that >I’m not doing.

Ah, it appears your module does to r->main->count++, if nginx_version >=
8011, which I can’t under 0.7.64, but in the module docs, it doesn’t say
that you can’t use the non-blocking sleep with 0.7.64, only that you
can’t use it after echo_location or echo_subrequest, which I’m not
trying to do. So what am I missing for how the non-blocking works in
0.7.64?

Posted at Nginx Forum:

7

Shoot, I just realized my

ngx_add_timer(wakeupFromSleep, 30);

call was never returning. Looking in the logs it looks like a
segfault…but why? Hmm…

Posted at Nginx Forum:

8

Two problems solved, one more found:

Solved:

  1. I wasn’t specifying a wakeupFromSleep->log, so I was seg faulting on
    the:

    ngx_log_debug3(NGX_LOG_DEBUG_EVENT, ev->log, 0,
    “event timer add: %d: %M:%M”,
    ngx_event_ident(ev->data), timer, ev->timer.key);

line in ngx_event_add_timer from ngx_event_timer.h

  1. I was specifying seconds, not ms. The line should correctly read:

ngx_add_timer(wakeupFromSleep, 30000);

Found:

Like before, the connection is closing immediately, even though the
event is in fact waking up. The only way I seem to be able to have it
not close immediate is to set r->keepalive = 1, which closes when the
event times out, but doesn’t send my headers. So, like my original
question, I still need a way to keep the connection open until the event
times out and can send the headers and finalize the response (in 0.7.64,
I’m hoping to not have to move to a later version if I can avoid it).

Thanks!

Posted at Nginx Forum:

9

Almost there!

I found a ngx_http_finalize_request I was forgetting about. That’s gone
and now:

  1. I can connect with one client, have it wait a specified number of
    seconds, then send the headers and finalize the request.

  2. I can connect with another client, while the first is waiting, and it
    doesn’t get blocked. If it’s a non-waiting connection it returns like it
    should.

  3. But if I try to connect with another waiting connection, connection 2
    returns when the first one returns (instead of after it’s time out) and
    doesn’t send any headers.

I’ll keep working on and post my results, though again if you have any
ideas let me know! Also if someone could explain how ngx_add/del_event
work I’d appreciate it. Thanks!

Posted at Nginx Forum: