Forum: NGINX NGX_AGAIN and POST data

Announcement (2017-05-07): www.ruby-forum.com is now read-only since I unfortunately do not have the time to support and maintain the forum any more. Please see rubyonrails.org/community and ruby-lang.org/en/community for other Rails- und Ruby-related community platforms.
0fe8a1a614282dc4b89281431f8e7f41?d=identicon&s=25 Brian Bruns (Guest)
on 2008-10-06 06:22
(Received via mailing list)
Hi all,

I'm writing a module for nginx and am hoping someone could enlighten me.

The module takes in some parameters via either GET or POST methods.
Everything works fine except when using Python's urllib which appears
to send the HTTP POST data as two TCP packets.

The short of it is that ngx_http_read_client_request_body() returns
NGX_AGAIN because c->read->ready is not set.  My module in turn passes
NGX_AGAIN back up to the calling routine.

The confusing part to me is the following lines from
src/http/ngx_http_core_module.c:

    if (r->content_handler) {
        r->write_event_handler = ngx_http_request_empty_handler;
        ngx_http_finalize_request(r, r->content_handler(r));
        return NGX_OK;
    }

which appears to (according the debug stuff in the log) simply close
out the connection regardless of the return value from
content_handler().  Shouldn't my module be getting re-called when more
data is available from the connection or am I misunderstanding
something?

Thanks in advance,

Brian
Abbd9d5312c5d54114a96a35dc94fdb1?d=identicon&s=25 Valery Kholodkov (Guest)
on 2008-10-06 13:40
(Received via mailing list)
> out the connection regardless of the return value from
> content_handler().  Shouldn't my module be getting re-called when more
> data is available from the connection or am I misunderstanding
> something?

No. post_handler (the second argument of
ngx_http_read_client_request_body) should be called when the body will
be
successfully received.

ngx_http_finalize_request does not close the connection whenever rc ==
NGX_AGAIN. Instead ngx_http_read_client_request_body sets read handler
to
ngx_http_read_client_request_body_handler. Therefore the read handler
will
be recalled, not your handler.
0fe8a1a614282dc4b89281431f8e7f41?d=identicon&s=25 Brian Bruns (Guest)
on 2008-10-06 14:46
(Received via mailing list)
Ok, now I'm really confused, here is my call to
ngx_http_read_client_request_body:

    rc = ngx_http_read_client_request_body(r,
ngx_http_mymodule_request_body_handler);

If I use any other client (firefox, links, wget)  then
ngx_http_mymodule_request_body_handler gets called.

Using urllib, which causes the NGX_AGAIN return, I get the following
in the log (the first statement is mine after the call to
ngx_http_read_client_request_body prior to returning NGX_AGAIN to the
caller).

2008/10/06 07:23:37 [alert] 34159#0: *5 failed to read client request
body, client: 192.168.16.1, server: localhost, request: "POST /sql
HTTP/1.0", host: "192.168.16.1:8081"
2008/10/06 07:23:37 [debug] 34159#0: *5 http finalize request: -2,
"/sql?"
2008/10/06 07:23:37 [debug] 34159#0: *5 event timer add: 6:
60000:3525165053
2008/10/06 07:23:37 [debug] 34159#0: timer delta: 1
2008/10/06 07:23:37 [debug] 34159#0: posted events 00000000
2008/10/06 07:23:37 [debug] 34159#0: worker cycle
2008/10/06 07:23:37 [debug] 34159#0: kevent timer: 60000, changes: 0
2008/10/06 07:23:37 [debug] 34159#0: kevent events: 1
2008/10/06 07:23:37 [debug] 34159#0: kevent: 6: ft:-1 fl:0025
ff:00000000 d:84 ud:002280B5
2008/10/06 07:23:37 [debug] 34159#0: *5 http test reading
2008/10/06 07:23:37 [debug] 34159#0: timer delta: 1
2008/10/06 07:23:37 [debug] 34159#0: posted events 00000000
2008/10/06 07:23:37 [debug] 34159#0: worker cycle
2008/10/06 07:23:37 [debug] 34159#0: kevent timer: 59999, changes: 0


My request_body_handler never gets called and the client eventually
times out.

Thanks again,

Brian

2008/10/6 Valery Kholodkov <valery+nginxen@grid.net.ru>:
Abbd9d5312c5d54114a96a35dc94fdb1?d=identicon&s=25 Valery Kholodkov (Guest)
on 2008-10-06 16:05
(Received via mailing list)
Hey, sorry for confusing you. What I do in this case is:

static ngx_int_t some_handler(ngx_http_request_t *r)
{

   [ ...blah... ]

    rc = ngx_http_read_client_request_body(r, my_handler);

    if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
        return rc;
    }

    return NGX_DONE;
}

So, basically I check the return value of
ngx_http_read_client_request_body and if it is weird, simply return it,
otherwise always return NGX_DONE.
0fe8a1a614282dc4b89281431f8e7f41?d=identicon&s=25 Brian Bruns (Guest)
on 2008-10-06 17:15
(Received via mailing list)
Yes that is basically what I'm doing:

rc = ngx_http_read_client_request_body(r, my_handler);

if (rc != NGX_OK) {
   ngx_log_error( ... );
   return rc;
}

I see the message in the log, so I know I returned from here with
NGX_AGAIN, but the request_body_handler never gets invoked.  Is there
some state I should be setting on the ngx_http_request_t before
returning control?

Thanks again for your help,

Brian

2008/10/6 Valery Kholodkov <valery+nginxen@grid.net.ru>:
Abbd9d5312c5d54114a96a35dc94fdb1?d=identicon&s=25 Valery Kholodkov (Guest)
on 2008-10-06 18:09
(Received via mailing list)
> I see the message in the log, so I know I returned from here with
> NGX_AGAIN, but the request_body_handler never gets invoked.  Is there
> some state I should be setting on the ngx_http_request_t before
> returning control?

post_handler is the state. It is being written into request. But you
should return NGX_DONE as I wrote before.
0fe8a1a614282dc4b89281431f8e7f41?d=identicon&s=25 Brian Bruns (Guest)
on 2008-10-07 19:24
(Received via mailing list)
Hi Valery,

Thank you, that fixes that problem.

I'm a little unsure what I should be doing at the end of my
post_handler.

When my code was in the main handler I returned with:

    return ngx_http_output_filter(r, &out);

Most examples of using ngx_http_read_client_request_body seem to be
upstream modules which really doesn't apply to my situation.  The DAV
module does this at the end of the post_handler
(ngx_http_dav_put_handler):

    ngx_http_finalize_request(r, ngx_http_send_header(r));
    return;

What is the correct thing to do at the end of the handler?  I've tried
both methods and neither seems to have any positive results.

Thanks again for all your help,

Brian

2008/10/6 Valery Kholodkov <valery+nginxen@grid.net.ru>:
Abbd9d5312c5d54114a96a35dc94fdb1?d=identicon&s=25 Valery Kholodkov (Guest)
on 2008-10-08 10:19
(Received via mailing list)
> Most examples of using ngx_http_read_client_request_body seem to be
> upstream modules which really doesn't apply to my situation.  The DAV
> module does this at the end of the post_handler
> (ngx_http_dav_put_handler):
>
>     ngx_http_finalize_request(r, ngx_http_send_header(r));
>     return;
>
> What is the correct thing to do at the end of the handler?  I've tried
> both methods and neither seems to have any positive results.

What result do you actually want to accomplish?
0fe8a1a614282dc4b89281431f8e7f41?d=identicon&s=25 Brian Bruns (Guest)
on 2008-10-09 16:23
(Received via mailing list)
Hi Valery,

Basically, I am returning some JSON objects as the text/plain mime type.

This had been working fine (creating the out buffer, etc...) as long
as I did it in the main handler, but it doesn't seem to work quite
right in the request_body_handler.  The client hangs on the response,
if I kill the nginx server, the client returns with the data, but
obviously that's not what I'm looking for.  I would like for the
output to be passed through the filter chain for gzipping and such.

What I want to do is "return ngx_http_output_filter(r, &out);" just
like before, but that seems to not work, my question is should it work
from the request_body_handler and is there any special trick to doing
so or better yet an existing module I can look at for pointers? I
couldn't find one, virtually everything that uses
ngx_http_read_client_request_body is an upstream module.

Thanks,

Brian

2008/10/8 Valery Kholodkov <valery+nginxen@grid.net.ru>:
Abbd9d5312c5d54114a96a35dc94fdb1?d=identicon&s=25 Valery Kholodkov (Guest)
on 2008-10-09 17:32
(Received via mailing list)
> What I want to do is "return ngx_http_output_filter(r, &out);" just
> like before, but that seems to not work, my question is should it work
> from the request_body_handler and is there any special trick to doing
> so or better yet an existing module I can look at for pointers? I
> couldn't find one, virtually everything that uses
> ngx_http_read_client_request_body is an upstream module.

I don't know whether a module doing similar thing exists, but I assume
you
invoke ngx_http_send_header, interpret the result, call
ngx_http_output_filter and it should be ok.
0fe8a1a614282dc4b89281431f8e7f41?d=identicon&s=25 Brian Bruns (Guest)
on 2008-10-15 05:18
(Received via mailing list)
Hi Valery,

First, thanks for all your help, I'm slowing getting a handle on the
internal workings of nginx but it's been a bit of a learning curve.

I did some more digging and what I'm finding is that everything is
indeed working correctly except the connection is not closed after the
results are returned.  Here is the transaction with my module in the
loop:

request:

POST /hello HTTP/1.0
Content-Length: 10

abcd=12345

response:

HTTP/1.1 200 OK
Server: nginx/0.7.19
Date: Wed, 15 Oct 2008 03:02:41 GMT
Content-Type: text/plain
Content-Length: 11
Last-Modified: Mon, 28 Sep 1970 06:00:00 GMT
Connection: close

hello world

The connection then hangs until another character is sent from the
client to the server at which time it closes.

I've replicated the same request using fastcgi/perl and it works
flawlessly.

Any thoughts on why my module would require additional input from the
client to close the connection?

Thanks again,

Brian

2008/10/9 Valery Kholodkov <valery+nginxen@grid.net.ru>:
0fe8a1a614282dc4b89281431f8e7f41?d=identicon&s=25 Brian Bruns (Guest)
on 2008-10-15 05:50
(Received via mailing list)
I believe I have solved my problem.  At the end of my request body
post_handler, I am now doing this:

    ngx_http_send_header(r);
    rc = ngx_http_output_filter(r, &out);
    ngx_http_finalize_request(r, rc);
    return;

I still not sure this is quite right, particularly on which value I
should be sending for the second argument to
ngx_http_finalize_request(), but it does seem to work reliably.

Thank you for all your help,

Brian

2008/10/14 Brian Bruns <brian.bruns@gmail.com>:
This topic is locked and can not be replied to.