Handling large POST data in nginx module

Hi,

I wrote an nginx module that handles POST data and returns back the same
(something similar to echo_read_request_body in echo module) however,
in echo module the input buffer is passed as is to output buffer.
However, in my case, I need to copy the incoming buffer and then process
it and then send it back to client.

What I am seeing in my POST handler routine is, the incoming request
size is 1377344 bytes. However, the file I am sending via curl command
is 1420448 bytes. Where could I be losing the rest of the bytes ??
Moreover the curl command on client hangs for a while and eventually
returns curl error code ‘18’ (CURLE_PARTIAL_FILE)

What could be going wrong ? Thanks in advance!

Here is my POST handler code


static void ngx_my_post_handler(ngx_http_request_t *r)
{
ngx_chain_t *cl, *in = r->request_body->bufs;
ngx_buf_t *b;

for (cl = in; cl; count++, cl=cl->next)
{
    b = cl->buf;

if (!ngx_buf_in_memory(b)) {
if (b->in_file) {
ngx_log_debug0(NGX_LOG_DEBUG_HTTP, r->connection->log,
0,
" What to do here ???");
return;
}

    } else {
    in_count++;
        in_sz += b->last - b->pos;
    }
}

u_char *out_data = (unsigned char *)ngx_pcalloc(r->pool, in_sz);

// Copy incoming data to outgoing data buffer
for (cl = in; cl; count++, cl = cl->next) {
    b = cl->buf;

    if (ngx_buf_in_memory(b)) {
        out_data = ngx_copy(out_data, b->pos, b->last - b->pos);
    }
}

// Allocate response buffer and fill it out
b = (ngx_buf_t *) ngx_pcalloc(r->pool, sizeof (ngx_buf_t));
if (b == NULL)
    return;

b->pos = out_data;
b->last = out_data + out_len;
b->memory = 1;
b->last_buf = 1;
b->last_in_chain = 1;

out_chain.buf = b;
out_chain.next = NULL;

// Setup HTTP response headers
r->headers_out.status = NGX_HTTP_OK;
r->headers_out.content_length_n = out_len;
r->headers_out.content_type.len = sizeof ("text/html") - 1;
r->headers_out.content_type.data = (u_char *) "text/html";

ngx_http_send_header(r);
ngx_http_output_filter(r, &out_chain);

}

Hello.

What I am seeing in my POST handler routine is, the incoming request
size is 1377344 bytes. However, the file I am sending via curl command
is 1420448 bytes. Where could I be losing the rest of the bytes ??

The incoming request buffer is not always completed reading when nginx’s
handler is called in some module.

You may look the function ngx_http_image_read in
nginx/src/http/modules/ngx_http_image_filter_module.c as reference.

http://hg.nginx.org/nginx/file/a24f88eff684/src/http/modules/ngx_http_image_filter_module.c#l451

The key point is using NGX_AGAIN.

Hello!

On Sun, Mar 30, 2014 at 12:20:30PM +0200, Mapper Uno wrote:

is 1420448 bytes. Where could I be losing the rest of the bytes.
Moreover the curl command on client hangs for a while and eventually
returns 18 error code.

First of all, make sure to use curl properly.

In particular, curl --data is identical to --data-ascii, while for
binary data --data-binary must be used.


Maxim D.
http://nginx.org/