# Trouble getting the Request Body of a HTTP Post

**URL:** <https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463>\
**Category:** NGINX\
**Created:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463 "2009-12-19T15:25:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/1 "2009-12-19T15:25:42Z")

</div>

Hi there,

I recently created a custom Nginx module, and successfully compiled it  
into the Nginx source and used a location directive to direct traffic to  
my module (good job me).

In my modules handler, I of course have access to the ngx\_http\_request\_t  
\*r object passed into the function.

Using this object, I have no trouble:

1. Getting the request headers (r-\>headers\_in)
2. Setting and sending the response headers (r-\>headers\_out,  
ngx\_http\_send\_header(r)😉
3. Send response body (return ngx\_http\_output\_filter(r, &out)😉

However, I’m having a very difficult time accessing the request body.  
Let me explain:

1. I run nginx
2. I telnet to my localhost at port 80
3. I enter something like this:

POST /mylocation HTTP/1.0  
Host: 127.0.0.1  
Content-Length: 13  
value=Tronman

But in my hander function,  
r-\>request\_body is completely null!

However, I know the body is being read since if I look at  
“r-\>request\_line.data” and simply offset the pointer by the length of  
the headers, I clearly see the “value=Tronman”. But I don’t feel this is  
a very safe thing to do unless I can always be sure that the Request  
Body will indeed be accessible this way and I know the total length of  
the headers. I’m not making any calls to  
ngx\_http\_discard\_request\_body(r); in the handler or anywhere else I’m  
aware of.

I also tried using the ngx\_http\_read\_client\_request\_body, but this  
ultimately just makes r-\>request\_header-\>buf point to gibberish. I used  
gdb to trace through the code and it ultimately leads to a failed “recv”  
socket function, for which the failure would make sence if the body had  
already been read (which it seems to have been).

A few other details:  
System: Ubuntu 9.10 32-bit  
Nginx Version: 0.7.64

So ultimately it comes down to:

1. Why isn’t the response body in r-\>response\_body like I would expect?
2. Would it be safe to access it by looking past request\_line.data, and  
if so, how do I know the total header length?
3. Is there another method I should be using to access the response  
body?
4. Any other advice you might have would be welcome!

Thanks.

Posted at Nginx Forum:

> **[Trouble getting the Request Body of a HTTP Post](https://forum.nginx.org/read.php?2,31312,31312#msg-31312)**
>
> Hi there, I recently created a custom Nginx module, and successfully compiled it into the Nginx source and used a location directive to direct traffic to my module (good job me). In my modules handler, I of course have access to the...

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/2 "2009-12-19T15:25:42Z")

</div>

Hello!

On Thu, Dec 17, 2009 at 11:35:16AM -0500, Tronman wrote:

> I recently created a custom Nginx module, and successfully compiled it into the Nginx source and used a location directive to direct traffic to my module (good job me).
> 
> In my modules handler, I of course have access to the ngx\_http\_request\_t \*r object passed into the function.

[…]

> So ultimately it comes down to:
> 
> 1. Why isn’t the response body in r-\>response\_body like I would expect?

Your expectation is wrong. Handler functions are called before  
request body has been read from client.

If you expect request body and want to read it, you should use  
ngx\_http\_read\_client\_request\_body().

> 1. Would it be safe to access it by looking past request\_line.data, and if so, how do I know the total header length?

No.

> 1. Is there another method I should be using to access the response body?

You mean request body? Use ngx\_http\_read\_client\_request\_body().

> 1. Any other advice you might have would be welcome!

To see the difference between (2) and (3) you may consider using  
some big POST (e.g. 10M) slowly uploaded by client. This also  
should give you some idea about how to handle multiple request  
body buffers and disk-buffered data.

Maxim D.

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/3 "2009-12-19T15:25:42Z")

</div>

Hi there, Thanks for your response!

I’m still trying to get the hang of the I/O model going on here. Bare  
with me to see if I understand:

When a connection comes in, the “ngx\_http\_my\_module\_handler” is called.  
This function has the request headers, but not the request body. In  
“ngx\_http\_my\_module\_handler”, we can use  
“ngx\_http\_read\_client\_request\_body(r, ngx\_http\_my\_module\_body\_handler);”  
to set a callback (handler) to be run when the Request Body is read  
(yes, I did mean Request Body, sorry for my typo). Since the response is  
dependant on the body, “ngx\_http\_my\_module\_handler” really doesn’t have  
to do anything else after setting the call back but return NGX\_DONE.

Thus, when the function “ngx\_http\_my\_module\_body\_handler” gets called,  
I’ve got both the original headers, and the request body, and it’s here  
that I should place all of the logic to build my response and send it to  
the client.

This implies that the client program connecting to the server must use  
separate “send” calls to send the header first, then the body, and  
cannot put both a header and a body into the same buffer and send them  
at the same time. Is this the case?

> To see the difference between (2) and (3) you may consider using  
> some big POST (e.g. 10M) slowly uploaded by client. This also  
> should give you some idea about how to handle multiple request  
> body buffers and disk-buffered data.

In this case, what would be the order of function calls? For example,  
would “ngx\_http\_my\_module\_body\_handler” be called for each piece of the  
body (which I would write my own logic to handle) or called after the  
entire (say 10M) body has been read. Is “ngx\_http\_my\_module\_handler”  
still called only once when the header arises?

In the meantime, I have moved my response generating logic into the  
“ngx\_http\_my\_module\_body\_handler” which is working much better (for  
single chunk bodies at least). Also, when I was telnet-ing, I was trying  
to send headers and body in one chunk, which wasn’t working. Instead I  
started sending them in separate chunks and am getting closer to the  
behaviour I expected.

Thanks for your help!

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,31312,31370#msg-31370](http://forum.nginx.org/read.php?2,31312,31370#msg-31370)

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/4 "2009-12-19T15:25:42Z")

</div>

Hello!

On Thu, Dec 17, 2009 at 01:58:00PM -0500, Tronman wrote:

> to be run when the Request Body is read (yes, I did mean Request  
> Body, sorry for my typo). Since the response is dependant on the  
> body, “ngx\_http\_my\_module\_handler” really doesn’t have to do  
> anything else after setting the call back but return NGX\_DONE.

Yes.

But please note that once ngx\_http\_read\_client\_request\_body()  
returns something \>= NGX\_HTTP\_SPECIAL\_RESPONSE you should return  
it from handler instead of NGX\_DONE. Basically it means that your  
body handler was already called (since full body was already  
read from client) and you have to pass it’s return code. So code  
in handler should look like:

```
rc = ngx_http_read_client_request_body(r,
                            ngx_http_my_module_body_handler);

if (rc >= NGX_HTTP_SPECIAL_RESPONSE) {
    return rc;
}

return NGX_DONE;

```

See e.g. proxy module for an example.

> Thus, when the function “ngx\_http\_my\_module\_body\_handler” gets  
> called, I’ve got both the original headers, and the request  
> body, and it’s here that I should place all of the logic to  
> build my response and send it to the client.

Yes.

> This implies that the client program connecting to the server  
> must use separate “send” calls to send the header first, then  
> the body, and cannot put both a header and a body into the same  
> buffer and send them at the same time. Is this the case?

No. As long as http request is correct it doesn’t matter if it  
was send in one packet or not.

> > To see the difference between (2) and (3) you may consider  
> > using some big POST (e.g. 10M) slowly uploaded by client. This  
> > also should give you some idea about how to handle multiple  
> > request body buffers and disk-buffered data.
> 
> In this case, what would be the order of function calls? For  
> example, would “ngx\_http\_my\_module\_body\_handler” be called for  
> each piece of the body (which I would write my own logic to  
> handle) or called after the entire (say 10M) body has been read.

It’s called once entire body has been read.

> Is “ngx\_http\_my\_module\_handler” still called only once when the header arises?

Yes.

> In the meantime, I have moved my response generating logic into  
> the “ngx\_http\_my\_module\_body\_handler” which is working much  
> better (for single chunk bodies at least). Also, when I was  
> telnet-ing, I was trying to send headers and body in one chunk,  
> which wasn’t working. Instead I started sending them in separate  
> chunks and am getting closer to the behaviour I expected.

Most likely you did something wrong, either while telneting or in  
you module.

Maxim D.

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/5 "2009-12-19T15:25:42Z")

</div>

curl is a good way to test URLs without having to re-type everything  
each time. You can POST by using the -d or -f option. Much better  
than telnet for this type of testing, in my experience.

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/6 "2009-12-19T15:25:42Z")

</div>

> This implies that the client program connecting to the server must use separate “send” calls to send the \> header first, then the body, and cannot put both a header and a body into the same buffer and send  
> them at the same time. Is this the case?

Ah, I’m wrong on this. It’s just the screwy way Telnet was sending the  
data chunks.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,31312,31380#msg-31380](http://forum.nginx.org/read.php?2,31312,31380#msg-31380)

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/7 "2009-12-19T15:25:42Z")

</div>

> Most likely you did something wrong, either while  
> telneting or in you module.

Precisely, it was a mistake I was making with my telnet session. I  
misinterpreted “sending” as really just inserting a couple of new lines.  
Also, it worked properly when I wrote a simple form to POST in Firefox.

Great! Your response cleared up nearly everything for me, the only other  
thing I’m not completely clear on is the difference between:

ngx\_chain\_t \*bufs;

and

ngx\_buf\_t \*buf;

in ngx\_http\_request\_body\_t. What is the difference between them? When  
should I use either to access the body?

I seem to be able to access:

r-\>request\_body-\>buf-\>start and  
r-\>request\_body-\>buf-\>pos

the same as I can use:

r-\>request\_body-\>bufs-\>buf-\>start and  
r-\>request\_body-\>bufs-\>buf-\>pos

While r-\>request\_body-\>bufs-\>next is NULL. So obviously the chain is  
some sort of linked list.

My only clue here are the:

client\_body\_in\_single\_buffer and  
client\_body\_buffer\_size

directives in NginxHttpCoreModule

So if I specify “client\_body\_in\_single\_buffer” as “true”, does it store  
the entire client request in r-\>request\_body-\>buf?  
But if not, does it separate the client request body into the chunks of  
whatever client\_body\_buffer\_size is set to, and then you use the chain  
to gain access to the entire thing?

Thanks again!

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,31312,31404#msg-31404](http://forum.nginx.org/read.php?2,31312,31404#msg-31404)

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/8 "2009-12-19T15:25:42Z")

</div>

Hello!

On Thu, Dec 17, 2009 at 03:33:04PM -0500, Tronman wrote:

[…]

> in ngx\_http\_request\_body\_t. What is the difference between them?  
> When should I use either to access the body?

You should use -\>bufs. Here is the comment from  
ngx\_http\_request\_body.c which should be helpful:

/\*

- on completion ngx\_http\_read\_client\_request\_body() adds to
- r-\>request\_body-\>bufs one or two bufs:
- \*) one memory buf that was preread in r-\>header\_in;
- \*) one memory or file buf that contains the rest of the body  
\*/

Simple example of using request\_body may be found in  
ngx\_http\_variables.c, in function  
ngx\_http\_variable\_request\_body().

[…]

> So if I specify “client\_body\_in\_single\_buffer” as “true”, does  
> it store the entire client request in r-\>request\_body-\>buf?

No. It will be in first buffer of r-\>request\_body-\>bufs, either  
in memory if it fits into client\_body\_buffer\_size or in file.

> But if not, does it separate the client request body into the  
> chunks of whatever client\_body\_buffer\_size is set to, and then  
> you use the chain to gain access to the entire thing?

Not exactly. Currently r-\>request\_body-\>bufs will contain up to two  
buffers: one for data preread with headers, and one for data got  
later. If there are more data than client\_body\_buffer\_size - the  
last buffer will be in temporary file.

It’s probably a good idea to spend some time reading  
ngx\_http\_request\_body.c for better clue what happens in various  
cases.

Maxim D.

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [December 19, 2009, 3:25pm UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/9 "2009-12-19T15:25:43Z")

</div>

## Nick P. Wrote:

> curl is a good way to test URLs without having to  
> re-type everything  
> each time. You can POST by using the -d or -f  
> option. Much better  
> than telnet for this type of testing, in my  
> experience.

Yes, I’ve heard of cURL but hadn’t used it a whole lot. I’ve installed  
it and will give it a try, thanks!

## Maxim D. Wrote:

> rest of the body  
> \*/
> 
> Simple example of using request\_body may be found  
> in  
> ngx\_http\_variables.c, in function  
> ngx\_http\_variable\_request\_body().

Ah, I was looking for something like that. Thanks!

> one for data got  
> later. If there are more data than  
> client\_body\_buffer\_size - the  
> last buffer will be in temporary file.
> 
> It’s probably a good idea to spend some time  
> reading  
> ngx\_http\_request\_body.c for better clue what  
> happens in various  
> cases.

> Maxim D.

Gotcha. I think I’m getting a handle on what’s going on. Thanks again  
for all your help!

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,31312,31612#msg-31612](http://forum.nginx.org/read.php?2,31312,31612#msg-31612)

---

<div class="post-metadata">

**Author:** ![Tronman](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/e9c0ed/32.png) [@Tronman](https://www.ruby-forum.com/u/Tronman)\
**Post date:** [February 9, 2011, 12:39am UTC](https://www.ruby-forum.com/t/trouble-getting-the-request-body-of-a-http-post/180463/10 "2011-02-09T00:39:54Z")

</div>

Great discussion even though this was posted a year ago! I was looking  
for information on capturing the POST request body and this cleared  
things up quite a bit on what is going on under the hood. My problem is  
slightly different in that I just want to log the request body.

I have my config setup to handle a bunch of GET requests which render  
pixels that work fine to handle analytics and parse query strings for  
logging. With an additional third party data stream, I need to handle a  
POST request to a given url that has JSON in an expected loggable format  
inside of it’s request body. I don’t want to use a secondary server with  
proxy\_pass and just want to log the whole response into an associated  
log file like what it does with GET requests. A snippet of some code  
that I’m using looks like the following:

GET request (which works great):

```auto
location ^~ /rl.gif {
    set $rl_lcid $arg_lcid;
    if ($http_cookie ~* "lcid=(.*\S)")
    {
      set $rl_lcid $cookie_lcid;
    }

    empty_gif;
    log_format my_tracking '{ "guid" : "$rl_lcid", "data" :
"$arg__rlcdnsegs" }';
    access_log /mnt/logs/nginx/my.access.log my_tracking;
    rewrite ^(.*)$ http://my/url?id=$cookie_lcid? redirect;
  }
```

Here is kinda what I am trying to do:  
POST request (which does not work):

```auto
location /bk {
    log_format bk_tracking $request_body;
    access_log /mnt/logs/nginx/bk.access.log bk_tracking;
  }
```

Curling “curl [http://myurl/bk](http://myurl/bk) -d name=example” gives me a 404 page not  
found.

Then I tried:

```auto
location /bk.gif {
    empty_gif;
    log_format bk_tracking $request_body;
    access_log /mnt/logs/nginx/bk.access.log bk_tracking;
  }
```

Curling “curl [http://myurl/bk.gif](http://myurl/bk.gif) -d name=example” gives me a 405 Not  
Allowed.

My current version is nginx/0.7.62. Any help in the right direction is  
very much appreciated! Thanks!

Posted at Nginx Forum:

> **[Re: Trouble getting the Request Body of a HTTP Post](https://forum.nginx.org/read.php?2,31312,173389#msg-173389)**
>
> Hi there, I recently created a custom Nginx module, and successfully compiled it into the Nginx source and used a location directive to direct traffic to my module (good job me). In my modules handler, I of course have access to the...