Is it possible that nginx will not buffer the client body?

Server NGINX
21

On Feb 28, 2013, at 6:51 PM, Valentin V. Bartenev [email protected]
wrote:

On Thursday 28 February 2013 21:36:23 Andr Cruz wrote:

I’m also very interested in being able to configure nginx to NOT proxy the
entire request.

[…]

Actually, you can.

Module ngx_http_proxy_module
Module ngx_http_proxy_module

I’ve probably explained myself wrong. What I want is for nginx to buffer
only chunks of the request body and pass these chunks to the upstream
server as they arrive.

Andr

22

I’m also very interested in being able to configure nginx to NOT proxy
the entire request.

Regarding this patch, added the feature of request_no_buffering for http and fastcgi proxy by yaoweibin · Pull Request #91 · alibaba/tengine · GitHub, is
anything fundamentally wrong with it? I don’t understand Chinese so I’m
at a loss here…

Best regards,
Andr Cruz

23

Pasi Kärkkäinen
People want nginx to not save the http PUT/POST request to disk
when using nginx as a reverse proxy.

Have the same problem. I use “haproxy” in front of “nginx” - but this
more a bad hack than a workaround.

Posted at Nginx Forum:

24

On Thu, Feb 28, 2013 at 07:02:46PM +0000, Andr Cruz wrote:

Module ngx_http_proxy_module
Module ngx_http_proxy_module

I’ve probably explained myself wrong. What I want is for nginx to buffer only
chunks of the request body and pass these chunks to the upstream server as they
arrive.

Yes, the problem is nginx saves the request body to the local disk on
the nginx server,
filling the disk and making the upload slower.

People want nginx to not save the http PUT/POST request to disk when
using nginx as a reverse proxy.

– Pasi

25

I keep getting the “upstream sent invalid header while reading response
header from upstream”
error when using the no_buffer patch…

The patch does not work for you?
Thanks
Markus

Posted at Nginx Forum:

26

Are there any plans to integrate this feature into NGINX?
It would be very great.

Maxim D.:
As a non-default mode of operation the aproach taken is likely
good enough (not looked into details), but the patch won’t work
with current nginx versions - at least it needs (likely major)
adjustments to cope with changes introduced during work on chunked
request body support as available in nginx 1.3.9+.

Posted at Nginx Forum:

27

On Tue, Mar 05, 2013 at 03:17:41PM +0200, Pasi Kärkkäinen wrote:

I keep getting the “upstream sent invalid header while reading response header
from upstream”
error when using the no_buffer patch…

Is there something you’d want me to try? Adjusting some config options?
Did you find anything weird in the log?

Thanks!

– Pasi

28

On Thu, Mar 07, 2013 at 12:25:43PM -0500, double wrote:

I keep getting the “upstream sent invalid header while reading response
header from upstream”
error when using the no_buffer patch…

The patch does not work for you?
Thanks
Markus

Yep, the patch doesn’t work for me.

– Pasi

29

Try the new patch, It could solve your problem.

Thanks for your test effort.

2013/3/8 Pasi Kärkkäinen [email protected]

30

On Thu, Mar 07, 2013 at 07:48:37PM +0200, Pasi K. wrote:

Yep, the patch doesn’t work for me.

the latest no_buffer_v5.patch seems to work OK for me with nginx 1.2.7.

– Pasi

31

2013/3/14 Pasi Kärkkäinen [email protected]

Weibin: Have you thought of upstreaming the no_buffer patch to nginx 1.3.x
so it could become part of next nginx stable version 1.4 ?

You can see my first email in this thread, The nginx team are working
on
it. But I don’t know when they finish it. You can wait for their
version.

Thanks.

32

On Thu, Feb 28, 2013 at 10:12:47PM +0400, Maxim D. wrote:

Hello!

Hello,

As a non-default mode of operation the aproach taken is likely
good enough (not looked into details), but the patch won’t work
with current nginx versions - at least it needs (likely major)
adjustments to cope with changes introduced during work on chunked
request body support as available in nginx 1.3.9+.

Weibin: Have you thought of upstreaming the no_buffer patch to nginx
1.3.x
so it could become part of next nginx stable version 1.4 ?

It’d be really nice to have the no_buffer functionality in stock nginx!
(the current no_buffer_v5.patch seems to work OK for me on nginx 1.2.7)

– Pasi

33

Weibin: Have you thought of upstreaming the no_buffer patch to nginx
1.3.x
so it could become part of next nginx stable version 1.4 ?
It’d be really nice to have the no_buffer functionality in stock nginx!

I agree. Nginx is such an excellent product - but this is a missing
feature.
Thanks
Markus

Posted at Nginx Forum:

34

Hmm, this patch is still not in nginx-1.3.16.
It would be great to have this feature in vanilla nginx.

Posted at Nginx Forum:

35

On Thu, Mar 14, 2013 at 10:39:12AM +0200, Pasi Kärkkäinen wrote:

On Thu, Mar 14, 2013 at 01:17:16PM +0800, Weibin Y. wrote:

Try the new patch, It could solve your problem.
Thanks for your test effort.

Thanks a lot!

I can confirm the “no_buffer_v5.patch” with nginx 1.2.7 fixes the problem for
me,
and both HTTP POST and HTTP PUT requests work OK now without buffering to disk.

For the archives: v6 of the patch fixes some remaining problems:
http://yaoweibin.cn/patches/nginx-1.2.7-no_buffer-v6.patch

– Pasi

36

On Tue, Feb 26, 2013 at 10:13:11PM +0800, Weibin Y. wrote:

It still worked in my box. Can you show me the debug.log
([1]Debugging | NGINX)? You need recompile  with
–with-debug configure argument and set debug level in error_log
directive.

Ok so I’ve sent you the debug log.
Can you see anything obvious in it?

I keep getting the “upstream sent invalid header while reading response
header from upstream”
error when using the no_buffer patch…

Thanks!

– Pasi

 I'm using the nginx 1.2.7 el6 src.rpm rebuilt with "headers more" module
         keepalive_timeout       70;

         if_modified_since  off;
 Â  Â  Â  Â  location / {
 >
 > Â  Â  Â > Oh sorry I missed that attachment. It seems to apply and
 >
 >
 >      > > Ã*  Ã* 2013/2/22 Pasi KÃ**â*¬rkkÃ**â*¬inen
 > Â  Â  Â > > Ã* Â Ã* Â Ã* >
 > Â  Â  Â doesn't work:
 [objs/src/http/ngx_http_request_body.o] Error 1
 > Â  Â  Â > > Ã* Â Ã*
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > >
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > Ã*** Ã** Ã*** ##
 > Â  Â  Â `proxy_request_buffering` or
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > Ã*** Ã** Ã***
 > Â  Â  Â **proxy_request_buffering** `on | off`
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > Ã*** Ã** Ã*** the
 with large request
 > Â  Â  Â will be broken after
 > Â  Â  Â > > Ã* Â Ã* Â Ã* these
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > >
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > >
 > Â  Â  Â > > Ã* Â Ã* Â Ã* could try my
 Ã*** what's the nginx
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > Ã*** Ã** Ã*** Ã**
 [4][4][6][7][10]https://github.com/taobao/tengine/pull/91.
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > Ã*** Ã** Ã*** Ã**
 Ã*** Ã** Ã*** Ã** Ã*** we want
 > Â  Â  Â request to the local disk
 > Â  Â  Â > > Ã* Â Ã* Â Ã* results in much high
 _______________________________________________
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** >
 > Â  Â  Â > > Ã* Â Ã* Â Ã*
 > Â  Â  Â Ã*
 > Â  Â  Â [10][10][12][13][16][email protected]
 Developer @ Server Platform
 [13][15][16][19]https://github.com/taobao/tengine/pull/91
 > Â  Â  Â > > Ã* Â Ã* Â Ã*
 [22][24][25][28]http://mailman.nginx.org/mailman/listinfo/nginx
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã**
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Developer @ Server Platform
 mailto:[31][32][35][email protected]
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** 9.
 mailto:[42][43][46][email protected]
 > Â  Â  Â > > Ã* Â Ã* Â Ã* > > Ã** Ã* 20.
 mailto:[53][54][57][email protected]
 > Â  Â  Â > > Ã* Â Ã* Â Ã* >
 > Â  Â  Â > > Ã* Â Ã* Â Ã*
 > Â  Â  Â > > Ã* Â Ã* 2.
 > Â  Â  Â > > Ã* Â Ã* 9.
 > Â  Â  Â > > Ã* Â 17. [78][81]https://github.com/taobao/tengine/pull/91
 > Â  Â  Â > > Ã* Â 25. mailto:[86][89][email protected]
 > Â  Â  Â > > Ã* Â 34. mailto:[95][98][email protected]
 [102][105]https://github.com/taobao/tengine/pull/91
 > Â  Â  Â > > Ã* Â 49. mailto:[110][113][email protected]
 [117][120]http://mailman.nginx.org/mailman/listinfo/nginx
 > Â  Â  Â > > [123][126]http://mailman.nginx.org/mailman/listinfo/nginx
 >

[133]limit_upload_rate/for-nginx.patch at master · cfsego/limit_upload_rate · GitHub

 > Â  14. [144]http://mailman.nginx.org/mailman/listinfo/nginx
 > Â  25. [155]http://mailman.nginx.org/mailman/listinfo/nginx
 > Â  36. [166]http://mailman.nginx.org/mailman/listinfo/nginx
 > Â  47. [177]http://mailman.nginx.org/mailman/listinfo/nginx
 > Â  58. mailto:[188][email protected]
 > Â  68. mailto:[198][email protected]
 > Â  79. mailto:[209][email protected]
 > Â  90. mailto:[220][email protected]
 > Â 101. mailto:[231][email protected]
 > Â 112. mailto:[242][email protected]
 > Â 123. [253]http://mailman.nginx.org/mailman/listinfo/nginx
 nginx mailing list
  1. Debugging | NGINX
  2. mailto:[email protected]
  3. mailto:[email protected]
  4. https://github.com/taobao/tengine/pull/91
  5. https://github.com/taobao/tengine/pull/91
  6. nginx Info Page
  7. mailto:[email protected]
  8. mailto:[email protected]
  9. mailto:[email protected]
  10. mailto:[email protected]
  11. mailto:[email protected]
  12. nginx Info Page
  13. limit_upload_rate/for-nginx.patch at master · cfsego/limit_upload_rate · GitHub
  14. nginx Info Page
  15. nginx Info Page
  16. nginx Info Page
  17. nginx Info Page
  18. mailto:[email protected]
  19. mailto:[email protected]
  20. mailto:[email protected]
  21. https://github.com/taobao/tengine/pull/91
  22. https://github.com/taobao/tengine/pull/91
  23. nginx Info Page
  24. mailto:[email protected]
  25. nginx Info Page
  26. mailto:[email protected]
  27. nginx Info Page
  28. mailto:[email protected]
  29. nginx Info Page
  30. mailto:[email protected]
  31. nginx Info Page
37

On Thu, Mar 14, 2013 at 01:17:16PM +0800, Weibin Y. wrote:

Try the new patch, It could solve your problem.
Thanks for your test effort.

Thanks a lot!

I can confirm the “no_buffer_v5.patch” with nginx 1.2.7 fixes the
problem for me,
and both HTTP POST and HTTP PUT requests work OK now without buffering
to disk.

– Pasi

 > >
 Did you find anything weird in the log?
 with nginx-1.2.7.
 > > Â  Â  Â I'm using the following configuration:
 > >
 > >
 > >
 > > Â  Â  Â # Ã* Â Ã* Â Ã* Â proxy_request_buffering Ã* Â Ã* Â Ã* Â Ã*
 > >
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Use the patch I attached in
 > > Â  Â  Â build OK.
 Ã** Ã* off;
 > > Â  Â  Â > Ã* Â Ã* Â Ã* [error] 29087#0: *60 upstream sent invalid
 troubleshoot it?
 10:38:21AM +0200,
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > Great, thanks, I'll
 > > Â  Â  Â nginx 1.2.7
 > > Â  Â  Â implicit
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > >
 > > Â  Â
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > >
 Ã*** > Ã**** Ã*** Ã**** The
 > > Â  Â  Â Default: 64k
 > > Â  Â  Â `fastcgi_request_buffering` to
 Ã*** has been
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** data. It could
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** you sent part of
 Ã*** > Ã**** Ã*** Ã****
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > >
 Ã*** Thanks,
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 > > Â  Â  Â [2][2][4][5][8][9]https://github.com/taobao/tengine/pull/91.
 > > Â  Â  Â Ã**** what's the nginx
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 Ã*** > Ã**** Ã*** Ã**** Ã***
 Ã*** This patch has
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** buffer the client
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** proxy to upload very
 Ã*** > Ã**** Ã*** Ã**** Ã***
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 > > Â  Â  Â Ã**** Ã*** Ã**** Ã*** Ã**** latency
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 Ã*** > Ã**** Ã*** Ã**** Ã***
 > > Â  Â  Â > Ã* Â Ã* Â Ã* Ã**
 Ã*** > Ã**** Ã*** Ã**** Ã***
 > > Â  Â
 > > Â  Â  Â Developer @ Server Platform
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 mailto:[14][16][17][20][21][email protected]
 > > Â  Â  Â > Ã* Â Ã* Â Ã* mailto:[17][19][20][23][24][email protected]
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã**
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > >
 Ã*** >
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** Ã**
 Weibin Y.
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** Ã*** 2.
 > > Â  Â  Â mailto:[34][35][38][39][email protected]
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** 11.
 > > Â  Â  Â > Ã* Â Ã* Â Ã*
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > > Ã*** Ã** 20.
 > > Â  Â  Â > Ã* Â Ã* Â Ã*
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** > >
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** Ã* Ã** >
 Â [60][61][64][65]http://mailman.nginx.org/mailman/listinfo/nginx
 mailto:[62][65][66][email protected]
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* Ã** 6.
 > > Â  Â  Â [72][75][76]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* 17.
 > > Â  Â  Â [83][86][87]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* 28.
 mailto:[94][97][98][email protected]
 mailto:[99][102][103][email protected]
 > > Â  Â  Â [104][107][108]https://github.com/taobao/tengine/pull/91
 > > Â  Â
 Â [113][116][117]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > > Ã** Ã* 57.
 > > Â  Â  Â > Ã* Â Ã* Â Ã* > >
 > > Â  Â  Â > Ã* Â Ã* Â Ã* >
 > > Â  Â  Â > Ã* Â Ã* Weibin Y.
 > > Â  Â  Â > Ã* Â Ã* 4. mailto:[134][135][email protected]
 > > Â  Â  Â > Ã* Â 12.
 > > Â  Â  Â > Ã* Â 19. mailto:[149][150][email protected]
 > > Â  Â  Â > Ã* Â 27.
 > > Â  Â  Â > Ã* Â 34. mailto:[164][165][email protected]
 > > Â  Â  Â > Ã* Â 42.
 [179][180]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  Â  Â > Ã* Â 57.
 Â

[193][194]limit_upload_rate/for-nginx.patch at master · cfsego/limit_upload_rate · GitHub

 > > Â  Â  Â > Ã* Â 71. mailto:[201][202][email protected]
 [208][209]https://github.com/taobao/tengine/pull/91
 > > Â  Â  Â > Ã* Â 86. mailto:[216][217][email protected]
 [223][224]https://github.com/taobao/tengine/pull/91
 > > Â  Â  Â > Ã* 101. mailto:[231][232][email protected]
 > > Â  Â  Â > Ã* 109.
 > > Â  Â  Â > Ã* 116. mailto:[246][247][email protected]
 [253][254]http://mailman.nginx.org/mailman/listinfo/nginx
 > >
 > >
 > > Â  Â 9. mailto:[271][email protected]
 > > Â  20. mailto:[282][email protected]
 > > Â  31. mailto:[293][email protected]
 > > Â  42. mailto:[304][email protected]
 > > Â  53. mailto:[315][email protected]
 > > Â  64. [326]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â  74. mailto:[336][email protected]
 > > Â  85. mailto:[347][email protected]
 > > Â  96. [358]https://github.com/taobao/tengine/pull/91
 > > Â 107. [369]https://github.com/taobao/tengine/pull/91
 > > Â 118. [380]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â 129. mailto:[391][email protected]
 > > Â 139. mailto:[401][email protected]
 > > Â 150. mailto:[412][email protected]
 > > Â 161. [423]https://github.com/taobao/tengine/pull/91
 > > Â 172. [434]https://github.com/taobao/tengine/pull/91
 > > Â 183. [445]http://mailman.nginx.org/mailman/listinfo/nginx

[455]limit_upload_rate/for-nginx.patch at master · cfsego/limit_upload_rate · GitHub

 > > Â 204. [466]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â 215. [477]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â 226. [488]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â 237. [499]http://mailman.nginx.org/mailman/listinfo/nginx
 > > Â 248. mailto:[510][email protected]
 > > Â 259. [521]http://mailman.nginx.org/mailman/listinfo/nginx
 > _______________________________________________

Weibin Y.
6. mailto:[email protected]
17. mailto:[email protected]
28. mailto:[email protected]
39. mailto:[email protected]
50. mailto:[email protected]
61. nginx Info Page
72. mailto:[email protected]
83. mailto:[email protected]
94. mailto:[email protected]
105. mailto:[email protected]
116. mailto:[email protected]
127. nginx Info Page
138. https://github.com/taobao/tengine/pull/91
149. https://github.com/taobao/tengine/pull/91
160. nginx Info Page
171. nginx Info Page
182. nginx Info Page
193. mailto:[email protected]
204. mailto:[email protected]
215. mailto:[email protected]
226. mailto:[email protected]
237. mailto:[email protected]
248. nginx Info Page
259. mailto:[email protected]
270. https://github.com/taobao/tengine/pull/91
281. https://github.com/taobao/tengine/pull/91
292. nginx Info Page
303. nginx Info Page
314. nginx Info Page
325. mailto:[email protected]
336. mailto:[email protected]
347. mailto:[email protected]
358. https://github.com/taobao/tengine/pull/91
369. https://github.com/taobao/tengine/pull/91
380. nginx Info Page
391. mailto:[email protected]
402. nginx Info Page
413. nginx Info Page
424. mailto:[email protected]
435. mailto:[email protected]
446. mailto:[email protected]
457. https://github.com/taobao/tengine/pull/91
468. https://github.com/taobao/tengine/pull/91
479. nginx Info Page
490. nginx Info Page
501. nginx Info Page
512. mailto:[email protected]
523. nginx Info Page
524. mailto:[email protected]
525. nginx Info Page
526. mailto:[email protected]
527. nginx Info Page
528. mailto:[email protected]
529. nginx Info Page