Is it possible that nginx will not buffer the client body?


#1

Hello!

is it possible that nginx will not buffer the client body before handle
the
request to upstream?

we want to use nginx as a reverse proxy to upload very very big file to
the
upstream, but the default behavior of nginx is to save the whole request
to
the local disk first before handle it to the upstream, which make the
upstream impossible to process the file on the fly when the file is
uploading, results in much high request latency and server-side resource
consumption.

Thanks!


#2

I know nginx team are working on it. You can wait for it.

If you are eager for this feature, you could try my patch:
https://github.com/taobao/tengine/pull/91. This patch has been running
in
our production servers.

2013/1/11 li zJay removed_email_address@domain.invalid


#3

lm011111 Wrote:

the local disk first before handle it to the upstream, which make the
upstream impossible to process the file on the fly when the file is
uploading, results in much high request latency and server-side
resource
consumption.

Thanks!


You could use the nginx upload module to do this.

http://www.grid.net.ru/nginx/upload.en.html

I currently am using this to support a video upload application built
with
Django and has full upload resume functionality as well. The only
caveat is
that I cannot support resumes across different sessions, but within the
same
session it works (uses the X-UPLOAD-SESSION-ID header or something
similar).

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,234926,234964#msg-234964


#4

This patch should work between nginx-1.2.6 and nginx-1.3.8.

The documentation is here:

client_body_postpone_sending

Syntax: client_body_postpone_sending size

Default: 64k

Context: http, server, location

If you specify the proxy_request_buffering or
fastcgi_request_buffering
to be off, Nginx will send the body to backend when it receives more
than
size data or the whole request body has been received. It could save
the
connection and reduce the IO number with backend.

proxy_request_buffering

Syntax: proxy_request_buffering on | off

Default: on

Context: http, server, location

Specify the request body will be buffered to the disk or not. If it’s
off,
the request body will be stored in memory and sent to backend after
Nginx
receives more than client_body_postpone_sending data. It could save
the
disk IO with large request body.

 Note that, if you specify it to be off, the nginx retry mechanism 

with
unsuccessful response will be broken after you sent part of the request
to
backend. It will just return 500 when it encounters such unsuccessful
response. This directive also breaks these variables: $request_body,
$request_body_file. You should not use these variables any more while
their
values are undefined.

fastcgi_request_buffering

Syntax: fastcgi_request_buffering on | off

Default: on

Context: http, server, location

The same as proxy_request_buffering.

2013/1/13 li zJay removed_email_address@domain.invalid


#5

Hello!

@yaoweibin

If you are eager for this feature, you could try my patch:
https://github.com/taobao/tengine/pull/91. This patch has been running in
our production servers.

what’s the nginx version your patch based on?

Thanks!


#6

This patch should work between nginx-1.2.6 and nginx-1.3.8.

The documentation is here:

client_body_postpone_sending

Syntax: client_body_postpone_sending size

Default: 64k

Context: http, server, location

If you specify the proxy_request_buffering or
fastcgi_request_buffering
to be off, Nginx will send the body to backend when it receives more
than
size data or the whole request body has been received. It could save
the
connection and reduce the IO number with backend.

proxy_request_buffering

Syntax: proxy_request_buffering on | off

Default: on

Context: http, server, location

Specify the request body will be buffered to the disk or not. If it’s
off,
the request body will be stored in memory and sent to backend after
Nginx
receives more than client_body_postpone_sending data. It could save
the
disk IO with large request body.

 Note that, if you specify it to be off, the nginx retry mechanism 

with
unsuccessful response will be broken after you sent part of the request
to
backend. It will just return 500 when it encounters such unsuccessful
response. This directive also breaks these variables: $request_body,
$request_body_file. You should not use these variables any more while
their
values are undefined.

fastcgi_request_buffering

Syntax: fastcgi_request_buffering on | off

Default: on

Context: http, server, location

The same as proxy_request_buffering.

2013/1/13 li zJay removed_email_address@domain.invalid


#7

Yes. It should work for any request method.

2013/1/16 Pasi Kärkkäinen removed_email_address@domain.invalid


#8

On Sun, Jan 13, 2013 at 08:22:17PM +0800, ??? wrote:

This patch should work between nginx-1.2.6 and nginx-1.3.8.
The documentation is here:

proxy_request_buffering

with unsuccessful response will be broken after you sent part of the
request to backend. It will just return 500 when it encounters such
unsuccessful response. This directive also breaks these variables:
$request_body, $request_body_file. You should not use these variables any
more while their values are undefined.

Hello,

This patch sounds exactly like what I need aswell!
I assume it works for both POST and PUT requests?

Thanks,

– Pasi

     we want to use nginx as a reverse proxy to upload very very big file

 [10]removed_email_address@domain.invalid
  1. https://github.com/taobao/tengine/pull/91
  2. mailto:removed_email_address@domain.invalid
  3. https://github.com/taobao/tengine/pull/91
  4. mailto:removed_email_address@domain.invalid
  5. mailto:removed_email_address@domain.invalid
  6. http://mailman.nginx.org/mailman/listinfo/nginx
  7. mailto:removed_email_address@domain.invalid
  8. http://mailman.nginx.org/mailman/listinfo/nginx
  9. mailto:removed_email_address@domain.invalid
  10. http://mailman.nginx.org/mailman/listinfo/nginx

#9

On Thu, Jan 17, 2013 at 11:15:58AM +0800, ??? wrote:

Yes. It should work for any request method.

Great, thanks, I’ll let you know how it works for me. Probably in two
weeks or so.

– Pasi

 > Â  Â If you specify the `proxy_request_buffering` or
 > Â  Â Default: `on`
 > Â  Â  Â  Â  Â  Â  Note that, if you specify it to be off, the nginx
 >
 > Â  Â  Â Hello!
 <[3][3]removed_email_address@domain.invalid> wrote:
 > Â  Â  Â  Â  Â is it possible that nginx will not buffer the client
 > Â  Â  Â  Â  Â the fly when the file is uploading, results in much high
 > Â  Â  Â  Â Developer @ Server Platform Team of Taobao
 > Â  Â --
 > Â  Â 5. mailto:[16]removed_email_address@domain.invalid
 > [24]http://mailman.nginx.org/mailman/listinfo/nginx

References
9. http://mailman.nginx.org/mailman/listinfo/nginx
20. http://mailman.nginx.org/mailman/listinfo/nginx
21. mailto:removed_email_address@domain.invalid
22. http://mailman.nginx.org/mailman/listinfo/nginx
23. mailto:removed_email_address@domain.invalid
24. http://mailman.nginx.org/mailman/listinfo/nginx
25. mailto:removed_email_address@domain.invalid
26. http://mailman.nginx.org/mailman/listinfo/nginx


#10

Use the patch I attached in this mail thread instead, don’t use the pull
request patch which is for tengine.

Thanks.

2013/2/22 Pasi Kärkkäinen removed_email_address@domain.invalid


#11

On Fri, Jan 18, 2013 at 10:38:21AM +0200, Pasi Kärkkäinen wrote:

On Thu, Jan 17, 2013 at 11:15:58AM +0800, ??? wrote:

Yes. It should work for any request method.

Great, thanks, I’ll let you know how it works for me. Probably in two weeks or
so.

Hi,

Adding the tengine pull request 91 on top of nginx 1.2.7 doesn’t work:

cc1: warnings being treated as errors
src/http/ngx_http_request_body.c: In function
‘ngx_http_read_non_buffered_client_request_body’:
src/http/ngx_http_request_body.c:506: error: implicit declaration of
function ‘ngx_http_top_input_body_filter’
make[1]: *** [objs/src/http/ngx_http_request_body.o] Error 1
make[1]: Leaving directory `/root/src/nginx/nginx-1.2.7’
make: *** [build] Error 2

ngx_http_top_input_body_filter() cannot be found from any .c/.h files…
Which other patches should I apply?

Perhaps this?
https://github.com/cfsego/limit_upload_rate/blob/master/for-nginx.patch

Thanks,

– Pasi


#12

On Fri, Feb 22, 2013 at 11:25:24AM +0200, Pasi Kärkkäinen wrote:

On Fri, Feb 22, 2013 at 10:06:11AM +0800, Weibin Y. wrote:

Use the patch I attached in this mail thread instead, don’t use the pull
request patch which is for tengine.Â
Thanks.

Oh sorry I missed that attachment. It seems to apply and build OK.
I’ll start testing it.

I added the patch on top of nginx 1.2.7 and enabled the following
options:

client_body_postpone_sending 64k;
proxy_request_buffering off;

after that connections through the nginx reverse proxy started failing
with errors like this:

[error] 29087#0: *49 upstream prematurely closed connection while
reading response header from upstream
[error] 29087#0: *60 upstream sent invalid header while reading response
header from upstream

And the services are unusable.

Commenting out the two config options above makes nginx happy again.
Any idea what causes that? Any tips how to troubleshoot it?

Thanks!

– Pasi


#13

Can you show me your configure? It works for me with nginx-1.2.7.

Thanks.

2013/2/22 Pasi Kärkkäinen removed_email_address@domain.invalid


#14

On Fri, Feb 22, 2013 at 10:06:11AM +0800, Weibin Y. wrote:

Use the patch I attached in this mail thread instead, don’t use the pull
request patch which is for tengine.Â
Thanks.

Oh sorry I missed that attachment. It seems to apply and build OK.
I’ll start testing it.

Thanks!

– Pasi

 make[1]: Leaving directory `/root/src/nginx/nginx-1.2.7'

 > > Â  Â  Â > Ã* Â Ã* Default: 64k
 > > Â  Â  Â with
 to backend
 > > Â  Â  Â > Ã* Â Ã* with unsuccessful response will be broken after
 > > Â  Â  Â variables any
 > > Â  Â  Â -- Pasi
 > > Â  Â  Â >
 > > Â  Â  Â > Ã* Â Ã* Â Ã* Â Ã* patch:
 > > Â  Â  Â body before
 > > Â  Â  Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* upstream, which make the upstream
 > > Â  Â  Â > Ã* Â Ã* Â Ã* Â Ã* Â Ã* nginx mailing list
 > > Â  Â  Â > Ã* Â Ã* Â Ã* Â Ã* nginx mailing list
 > > Â  Â  Â >
 > > Â  Â  Â > Ã* Â Ã* 3. mailto:[14][16]removed_email_address@domain.invalid
 > > Â  Â  Â > Ã* Â 11.
 > > Â  Â  Â [26][28]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  Â 3. mailto:[31]removed_email_address@domain.invalid
 > > Â  14. mailto:[42]removed_email_address@domain.invalid
 > > Â  25. mailto:[53]removed_email_address@domain.invalid
 > [58]http://mailman.nginx.org/mailman/listinfo/nginx

References
9. http://mailman.nginx.org/mailman/listinfo/nginx
20. http://mailman.nginx.org/mailman/listinfo/nginx
31. mailto:removed_email_address@domain.invalid
42. mailto:removed_email_address@domain.invalid
53. mailto:removed_email_address@domain.invalid
54. http://mailman.nginx.org/mailman/listinfo/nginx
55. mailto:removed_email_address@domain.invalid
56. http://mailman.nginx.org/mailman/listinfo/nginx
57. mailto:removed_email_address@domain.invalid
58. http://mailman.nginx.org/mailman/listinfo/nginx
59. mailto:removed_email_address@domain.invalid
60. http://mailman.nginx.org/mailman/listinfo/nginx


#15

nginx mailing list
removed_email_address@domain.invalid
http://mailman.nginx.org/mailman/listinfo/nginx


#16

This would be an excellent feature!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,234926,236640#msg-236640


#17

I know nginx team are working on it. You can wait for it.

Hopefully they will find a solution!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,234926,236723#msg-236723


#18

On Mon, Feb 25, 2013 at 10:13:42AM +0800, Weibin Y. wrote:

Can you show me your configure? It works for me with nginx-1.2.7.
Thanks.

Hi,

I’m using the nginx 1.2.7 el6 src.rpm rebuilt with “headers more” module
added,
and your patch.

I’m using the following configuration:

server {
listen public_ip:443 ssl;
server_name service.domain.tld;

    ssl                     on;
    keepalive_timeout       70;

    access_log              /var/log/nginx/access-service.log;
    access_log              /var/log/nginx/access-service-full.log 

full;
error_log /var/log/nginx/error-service.log;

    client_header_buffer_size 64k;
    client_header_timeout   120;

    proxy_next_upstream error timeout invalid_header http_500 

http_502 http_503;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_redirect off;
proxy_buffering off;
proxy_cache off;

add_header Last-Modified “”;
if_modified_since off;

    client_max_body_size    262144M;
    client_body_buffer_size 1024k;
    client_body_timeout     240;

    chunked_transfer_encoding off;

client_body_postpone_sending 64k;

proxy_request_buffering off;

location / {

proxy_pass      https://service-backend;

}
}

Thanks!

– Pasi

 > Oh sorry I missed that attachment. It seems to apply and build OK.
 with errors like this:
 Thanks!
 > > Â  Â  Â > >
 > >
 > > Â  Â  Â ngx_http_top_input_body_filter() cannot be found from any
 > > Â  Â  Â >
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** ##
 > > Â  Â  Â send the body
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* >
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** the request body will be
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* >
 > > Â  Â  Â it encounters
 undefined.
 > > Â  Â  Â > >
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* This patch has
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* >
 our production servers.
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Ã* Ã** Ã* Ã** Ã* Ã** handle
 request to the local disk
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Ã* Ã** Ã* Ã** Ã* Ã** latency
 > > Â  Â  Â [7][7][9][10]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Ã* Ã** Ã* Ã**
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Ã* Ã**
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Visible links
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** 6.
 > > Â  Â  Â [22][24][25]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  Â  Â > > Ã* Â Ã* Â Ã* nginx mailing list
 > > Â  Â  Â > > Ã* Â Ã* Visible links
 > > Â  Â  Â > > Ã* Â Ã* 8. mailto:[36][37]removed_email_address@domain.invalid
 [43][44]https://github.com/taobao/tengine/pull/91
 > > Â  Â  Â > > Ã* Â 23. mailto:[51][52]removed_email_address@domain.invalid
 > > Â  Â  Â >
 > > Â  Â --
 > > Â  Â 4. [65]https://github.com/taobao/tengine/pull/91
 > > Â  15. [76]https://github.com/taobao/tengine/pull/91
 > > Â  26. [87]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  37. [98]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  48. [109]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  59. mailto:[120]removed_email_address@domain.invalid
 > [125]http://mailman.nginx.org/mailman/listinfo/nginx

References
9. mailto:removed_email_address@domain.invalid
20. mailto:removed_email_address@domain.invalid
31. https://github.com/taobao/tengine/pull/91
42. https://github.com/taobao/tengine/pull/91
53. http://mailman.nginx.org/mailman/listinfo/nginx
64. mailto:removed_email_address@domain.invalid
75. mailto:removed_email_address@domain.invalid
86. mailto:removed_email_address@domain.invalid
97. mailto:removed_email_address@domain.invalid
108. mailto:removed_email_address@domain.invalid
119. http://mailman.nginx.org/mailman/listinfo/nginx
120. mailto:removed_email_address@domain.invalid
121. http://mailman.nginx.org/mailman/listinfo/nginx
122. mailto:removed_email_address@domain.invalid
123. http://mailman.nginx.org/mailman/listinfo/nginx
124. mailto:removed_email_address@domain.invalid
125. http://mailman.nginx.org/mailman/listinfo/nginx
126. mailto:removed_email_address@domain.invalid
127. http://mailman.nginx.org/mailman/listinfo/nginx


#19

Hello!

On Thu, Feb 28, 2013 at 05:36:23PM +0000, André Cruz wrote:

I’m also very interested in being able to configure nginx to NOT
proxy the entire request.

Regarding this patch,
https://github.com/alibaba/tengine/pull/91, is anything
fundamentally wrong with it? I don’t understand Chinese so I’m
at a loss here…

As a non-default mode of operation the aproach taken is likely
good enough (not looked into details), but the patch won’t work
with current nginx versions - at least it needs (likely major)
adjustments to cope with changes introduced during work on chunked
request body support as available in nginx 1.3.9+.


Maxim D.
http://nginx.org/en/donation.html


#20

On Thursday 28 February 2013 21:36:23 Andr Cruz wrote:

I’m also very interested in being able to configure nginx to NOT proxy the
entire request.

[…]

Actually, you can.

http://nginx.org/r/proxy_set_body
http://nginx.org/r/proxy_pass_request_body

wbr, Valentin V. Bartenev