Implementing HTTP/1.1 request chunking

Server NGINX
1

Hi all,

I want to get some input with regards to having HTTP/1.1 request
chunking in
Nginx. I¹ve done some digging, and I¹ve found a few patches and the
module
developed by agentzh.

So, my question is this –
What, as a developer, would I need to do so that I could create a patch
for
this that would be accepted as part of the Nginx distribution?

I know that some of the items on this list so far are:

  1. That the patch works.
  2. Error checking – check the size of the chunks & that the series of
    chunks are terminated properly.
  3. Don’t use Ragel. :slight_smile: Or, for that matter, any tool not already in use
    by
    the Nginx build process.

Thank you -

Ian H. Greenhoe
MANDIANT
Senior Software Developer
675 North Washington Street
Suite 210
Alexandria, VA 22314
703.224.2905 t
703.683.2891 f
512.422.9564 m
[email protected]

2

On Fri, Jan 15, 2010 at 3:19 PM, Ian G.
[email protected] wrote:

So, my question is this –
What, as a developer, would I need to do so that I could create a patch for
this that would be accepted as part of the Nginx distribution?

I started writing the ngx_chunkin module just because it’s
particularly hard (for me) to create a patch (of this size) that can
be accepted by Igor S… Patching the core would be much much
easier than doing chunked reading from outside :slight_smile:

I know that some of the items on this list so far are:

  1. That the patch works.

ngx_chunkin works :slight_smile: AFAIK, some people have already put it into
production :slight_smile:

  1. Error checking – check the size of the chunks & that the series of
    chunks are terminated properly.

ngx_chunkin honors the http core module’s max_client_body_size
directive settings, as well as the client_body_buffer_size and
client_body_timeout settings.

  1. Don’t use Ragel. :slight_smile: Or, for that matter, any tool not already in use by
    the Nginx build process.

Okay, ngx_chunkin uses Ragel just to eliminate a lot of boring code
for truly streaming parsing. Fortunately we can generate C source
before we make a release such that the end user don’t need to know
anything about Ragel :slight_smile:

I’m not meant to discourage your efforts. If your patch gets accepted
by the core, I’d happily mark ngx_chunkin as obsolete and just
encourage people to use it if they have to stick with an earlier nginx
version for some reasons.

Cheers,
-agentzh

3

On Fri, Jan 15, 2010 at 3:19 PM, Ian G.
[email protected] wrote:

I know that some of the items on this list so far are:

Oh, forgot to mention we should also take into account HTTP 1.1
keepalive and pipelining support.

ngx_chunkin currently has a working non-pipelined keepalive support,
but the pipelining support is untested so far. (Even though POST +
pipelining is rare in real world.)

Cheers,
-agentzh

4

On 1/15/10 5:04 AM, “agentzh” [email protected] wrote:

On Fri, Jan 15, 2010 at 3:19 PM, Ian G. [email protected]
wrote:

So, my question is this –
What, as a developer, would I need to do so that I could create a patch for
this that would be accepted as part of the Nginx distribution?

I started writing the ngx_chunkin module just because it’s
particularly hard (for me) to create a patch (of this size) that can
be accepted by Igor S… Patching the core would be much much
easier than doing chunked reading from outside :slight_smile:

Ooh, yeah. It would definitely be easier to patch the core… Which is
why I
was asking.

I know that some of the items on this list so far are:

  1. That the patch works.

ngx_chunkin works :slight_smile: AFAIK, some people have already put it into production :slight_smile:

Didn’t mean to imply that it didn’t – I’m just listing all of the
requirements. :slight_smile:

  1. Error checking – check the size of the chunks & that the series of
    chunks are terminated properly.

ngx_chunkin honors the http core module’s max_client_body_size
directive settings, as well as the client_body_buffer_size and
client_body_timeout settings.

This is exactly the sort of thing that I need to know.

  1. Don’t use Ragel. :slight_smile: Or, for that matter, any tool not already in use by
    the Nginx build process.

Okay, ngx_chunkin uses Ragel just to eliminate a lot of boring code
for truly streaming parsing. Fortunately we can generate C source
before we make a release such that the end user don’t need to know
anything about Ragel :slight_smile:

I understand – I’ve both used parser generators and coded parsers by
hand.
Using parser generators is much preferred.

I’m not meant to discourage your efforts. If your patch gets accepted
by the core, I’d happily mark ngx_chunkin as obsolete and just
encourage people to use it if they have to stick with an earlier nginx
version for some reasons.

Here’s to hoping that I can obsolete your code. :slight_smile:

-Ian