Hello! is it possible that nginx will not buffer the client body before handle the request to upstream? we want to use nginx as a reverse proxy to upload very very big file to the upstream, but the default behavior of nginx is to save the whole request to the local disk first before handle it to the upstream, which make the upstream impossible to process the file on the fly when the file is uploading, results in much high request latency and server-side resource consumption. Thanks!
on 2013-01-11 08:21
on 2013-01-11 10:18
I know nginx team are working on it. You can wait for it. If you are eager for this feature, you could try my patch: https://github.com/taobao/tengine/pull/91. This patch has been running in our production servers. 2013/1/11 li zJay <zjay1987@gmail.com>
on 2013-01-11 20:31
lm011111 Wrote: ------------------------------------------------------- > the local disk first before handle it to the upstream, which make the > upstream impossible to process the file on the fly when the file is > uploading, results in much high request latency and server-side > resource > consumption. > > Thanks! > _______________________________________________ You could use the nginx upload module to do this. http://www.grid.net.ru/nginx/upload.en.html I currently am using this to support a video upload application built with Django and has full upload resume functionality as well. The only caveat is that I cannot support resumes across different sessions, but within the same session it works (uses the X-UPLOAD-SESSION-ID header or something similar). Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234926,234964#msg-234964
on 2013-01-13 10:02
Hello! @yaoweibin > If you are eager for this feature, you could try my patch: > https://github.com/taobao/tengine/pull/91. This patch has been running in > our production servers. what's the nginx version your patch based on? Thanks!
on 2013-01-13 13:23
This patch should work between nginx-1.2.6 and nginx-1.3.8.
The documentation is here:
## client_body_postpone_sending ##
Syntax: **client_body_postpone_sending** `size`
Default: 64k
Context: `http, server, location`
If you specify the `proxy_request_buffering` or
`fastcgi_request_buffering`
to be off, Nginx will send the body to backend when it receives more
than
`size` data or the whole request body has been received. It could save
the
connection and reduce the IO number with backend.
## proxy_request_buffering ##
Syntax: **proxy_request_buffering** `on | off`
Default: `on`
Context: `http, server, location`
Specify the request body will be buffered to the disk or not. If it's
off,
the request body will be stored in memory and sent to backend after
Nginx
receives more than `client_body_postpone_sending` data. It could save
the
disk IO with large request body.
Note that, if you specify it to be off, the nginx retry mechanism
with
unsuccessful response will be broken after you sent part of the request
to
backend. It will just return 500 when it encounters such unsuccessful
response. This directive also breaks these variables: $request_body,
$request_body_file. You should not use these variables any more while
their
values are undefined.
## fastcgi_request_buffering ##
Syntax: **fastcgi_request_buffering** `on | off`
Default: `on`
Context: `http, server, location`
The same as `proxy_request_buffering`.
2013/1/13 li zJay <zjay1987@gmail.com>
on 2013-01-13 13:26
This patch should work between nginx-1.2.6 and nginx-1.3.8.
The documentation is here:
## client_body_postpone_sending ##
Syntax: **client_body_postpone_sending** `size`
Default: 64k
Context: `http, server, location`
If you specify the `proxy_request_buffering` or
`fastcgi_request_buffering`
to be off, Nginx will send the body to backend when it receives more
than
`size` data or the whole request body has been received. It could save
the
connection and reduce the IO number with backend.
## proxy_request_buffering ##
Syntax: **proxy_request_buffering** `on | off`
Default: `on`
Context: `http, server, location`
Specify the request body will be buffered to the disk or not. If it's
off,
the request body will be stored in memory and sent to backend after
Nginx
receives more than `client_body_postpone_sending` data. It could save
the
disk IO with large request body.
Note that, if you specify it to be off, the nginx retry mechanism
with
unsuccessful response will be broken after you sent part of the request
to
backend. It will just return 500 when it encounters such unsuccessful
response. This directive also breaks these variables: $request_body,
$request_body_file. You should not use these variables any more while
their
values are undefined.
## fastcgi_request_buffering ##
Syntax: **fastcgi_request_buffering** `on | off`
Default: `on`
Context: `http, server, location`
The same as `proxy_request_buffering`.
2013/1/13 li zJay <zjay1987@gmail.com>
on 2013-01-16 16:15
On Sun, Jan 13, 2013 at 08:22:17PM +0800, ?????? wrote: > This patch should work between nginx-1.2.6 and nginx-1.3.8. > The documentation is here: > ## proxy_request_buffering ## > with unsuccessful response will be broken after you sent part of the > request to backend. It will just return 500 when it encounters such > unsuccessful response. This directive also breaks these variables: > $request_body, $request_body_file. You should not use these variables any > more while their values are undefined. > Hello, This patch sounds exactly like what I need aswell! I assume it works for both POST and PUT requests? Thanks, -- Pasi > > we want to use nginx as a reverse proxy to upload very very big file > > [10]nginx@nginx.org > 2. https://github.com/taobao/tengine/pull/91 > 3. mailto:yaoweibin@gmail.com > 4. https://github.com/taobao/tengine/pull/91 > 5. mailto:zjay1987@gmail.com > 6. mailto:nginx@nginx.org > 7. http://mailman.nginx.org/mailman/listinfo/nginx > 8. mailto:nginx@nginx.org > 9. http://mailman.nginx.org/mailman/listinfo/nginx > 10. mailto:nginx@nginx.org > 11. http://mailman.nginx.org/mailman/listinfo/nginx
on 2013-01-17 04:16
Yes. It should work for any request method. 2013/1/16 Pasi Kärkkäinen <pasik@iki.fi>
on 2013-01-18 09:39
On Thu, Jan 17, 2013 at 11:15:58AM +0800, ?????? wrote: > Yes. It should work for any request method. > Great, thanks, I'll let you know how it works for me. Probably in two weeks or so. -- Pasi > > Â Â If you specify the `proxy_request_buffering` or > > Â Â Default: `on` > > Â Â Â Â Â Â Note that, if you specify it to be off, the nginx > > > > Â Â Â Hello! > <[3][3]yaoweibin@gmail.com> wrote: > > Â Â Â Â Â is it possible that nginx will not buffer the client > > Â Â Â Â Â the fly when the file is uploading, results in much high > > Â Â Â Â Developer @ Server Platform Team of Taobao > > Â Â -- > > Â Â 5. mailto:[16]zjay1987@gmail.com > > [24]http://mailman.nginx.org/mailman/listinfo/nginx > References > 9. http://mailman.nginx.org/mailman/listinfo/nginx > 20. http://mailman.nginx.org/mailman/listinfo/nginx > 21. mailto:nginx@nginx.org > 22. http://mailman.nginx.org/mailman/listinfo/nginx > 23. mailto:nginx@nginx.org > 24. http://mailman.nginx.org/mailman/listinfo/nginx > 25. mailto:nginx@nginx.org > 26. http://mailman.nginx.org/mailman/listinfo/nginx
on 2013-02-21 21:08
On Fri, Jan 18, 2013 at 10:38:21AM +0200, Pasi Kärkkäinen wrote: > On Thu, Jan 17, 2013 at 11:15:58AM +0800, ?????? wrote: > > Yes. It should work for any request method. > > > > Great, thanks, I'll let you know how it works for me. Probably in two weeks or so. > Hi, Adding the tengine pull request 91 on top of nginx 1.2.7 doesn't work: cc1: warnings being treated as errors src/http/ngx_http_request_body.c: In function 'ngx_http_read_non_buffered_client_request_body': src/http/ngx_http_request_body.c:506: error: implicit declaration of function 'ngx_http_top_input_body_filter' make[1]: *** [objs/src/http/ngx_http_request_body.o] Error 1 make[1]: Leaving directory `/root/src/nginx/nginx-1.2.7' make: *** [build] Error 2 ngx_http_top_input_body_filter() cannot be found from any .c/.h files.. Which other patches should I apply? Perhaps this? https://github.com/cfsego/limit_upload_rate/blob/m... Thanks, -- Pasi
on 2013-02-22 03:06
Use the patch I attached in this mail thread instead, don't use the pull request patch which is for tengine. Thanks. 2013/2/22 Pasi Kärkkäinen <pasik@iki.fi>
on 2013-02-22 10:25
On Fri, Feb 22, 2013 at 10:06:11AM +0800, Weibin Yao wrote: > Use the patch I attached in this mail thread instead, don't use the pull > request patch which is for tengine. > Thanks. > Oh sorry I missed that attachment. It seems to apply and build OK. I'll start testing it. Thanks! -- Pasi > > make[1]: Leaving directory `/root/src/nginx/nginx-1.2.7' > > > >    > Ã*  Ã* Default: 64k > > >    with > to backend > > >    > Ã*  Ã* with unsuccessful response will be broken after > > >    variables any > > >    -- Pasi > > >    > > > >    > Ã*  Ã*  Ã*  Ã* patch: > > >    body before > > >    > Ã*  Ã*  Ã*  Ã*  Ã* upstream, which make the upstream > > >    > Ã*  Ã*  Ã*  Ã*  Ã* nginx mailing list > > >    > Ã*  Ã*  Ã*  Ã* nginx mailing list > > >    > > > >    > Ã*  Ã* 3. mailto:[14][16]yaoweibin@gmail.com > > >    > Ã*  11. > > >    [26][28]http://mailman.nginx.org/mailman/listinfo/nginx > > >   3. mailto:[31]yaoweibin@gmail.com > > >  14. mailto:[42]yaoweibin@gmail.com > > >  25. mailto:[53]nginx@nginx.org > > [58]http://mailman.nginx.org/mailman/listinfo/nginx > References > 9. http://mailman.nginx.org/mailman/listinfo/nginx > 20. http://mailman.nginx.org/mailman/listinfo/nginx > 31. mailto:yaoweibin@gmail.com > 42. mailto:yaoweibin@gmail.com > 53. mailto:nginx@nginx.org > 54. http://mailman.nginx.org/mailman/listinfo/nginx > 55. mailto:nginx@nginx.org > 56. http://mailman.nginx.org/mailman/listinfo/nginx > 57. mailto:nginx@nginx.org > 58. http://mailman.nginx.org/mailman/listinfo/nginx > 59. mailto:nginx@nginx.org > 60. http://mailman.nginx.org/mailman/listinfo/nginx
on 2013-02-22 11:51
On Fri, Feb 22, 2013 at 11:25:24AM +0200, Pasi Kärkkäinen wrote: > On Fri, Feb 22, 2013 at 10:06:11AM +0800, Weibin Yao wrote: > > Use the patch I attached in this mail thread instead, don't use the pull > > request patch which is for tengine. > > Thanks. > > > > Oh sorry I missed that attachment. It seems to apply and build OK. > I'll start testing it. > I added the patch on top of nginx 1.2.7 and enabled the following options: client_body_postpone_sending 64k; proxy_request_buffering off; after that connections through the nginx reverse proxy started failing with errors like this: [error] 29087#0: *49 upstream prematurely closed connection while reading response header from upstream [error] 29087#0: *60 upstream sent invalid header while reading response header from upstream And the services are unusable. Commenting out the two config options above makes nginx happy again. Any idea what causes that? Any tips how to troubleshoot it? Thanks! -- Pasi
on 2013-02-25 03:14
Can you show me your configure? It works for me with nginx-1.2.7. Thanks. 2013/2/22 Pasi Kärkkäinen <pasik@iki.fi>
on 2013-02-25 11:13
On Mon, Feb 25, 2013 at 10:13:42AM +0800, Weibin Yao wrote: > Can you show me your configure? It works for me with nginx-1.2.7. > Thanks. > Hi, I'm using the nginx 1.2.7 el6 src.rpm rebuilt with "headers more" module added, and your patch. I'm using the following configuration: server { listen public_ip:443 ssl; server_name service.domain.tld; ssl on; keepalive_timeout 70; access_log /var/log/nginx/access-service.log; access_log /var/log/nginx/access-service-full.log full; error_log /var/log/nginx/error-service.log; client_header_buffer_size 64k; client_header_timeout 120; proxy_next_upstream error timeout invalid_header http_500 http_502 http_503; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_redirect off; proxy_buffering off; proxy_cache off; add_header Last-Modified ""; if_modified_since off; client_max_body_size 262144M; client_body_buffer_size 1024k; client_body_timeout 240; chunked_transfer_encoding off; # client_body_postpone_sending 64k; # proxy_request_buffering off; location / { proxy_pass https://service-backend; } } Thanks! -- Pasi > > Oh sorry I missed that attachment. It seems to apply and build OK. > with errors like this: > Thanks! > > > Â Â Â > > > > > > > > Â Â Â ngx_http_top_input_body_filter() cannot be found from any > > > Â Â Â > > > > Â Â Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** ## > > > Â Â Â send the body > > > Â Â Â > > Ã* Â Ã* Â Ã* > > > > Â Â Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** the request body will be > > > Â Â Â > > Ã* Â Ã* Â Ã* > > > > Â Â Â it encounters > undefined. > > > Â Â Â > > > > > Â Â Â > > Ã* Â Ã* Â Ã* This patch has > > > Â Â Â > > Ã* Â Ã* Â Ã* > > our production servers. > > > Â Â Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Ã* Ã** Ã* Ã** Ã* Ã** handle > request to the local disk > > > Â Â Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Ã* Ã** Ã* Ã** Ã* Ã** latency > > > Â Â Â [7][7][9][10]http://mailman.nginx.org/mailman/listinfo/nginx > > > Â Â Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Ã* Ã** Ã* Ã** > > > Â Â Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Ã* Ã** > > > Â Â Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** Visible links > > > Â Â Â > > Ã* Â Ã* Â Ã* > Ã** Ã* Ã** 6. > > > Â Â Â [22][24][25]http://mailman.nginx.org/mailman/listinfo/nginx > > > Â Â Â > > Ã* Â Ã* Â Ã* nginx mailing list > > > Â Â Â > > Ã* Â Ã* Visible links > > > Â Â Â > > Ã* Â Ã* 8. mailto:[36][37]nginx@nginx.org > [43][44]https://github.com/taobao/tengine/pull/91 > > > Â Â Â > > Ã* Â 23. mailto:[51][52]nginx@nginx.org > > > Â Â Â > > > > Â Â -- > > > Â Â 4. [65]https://github.com/taobao/tengine/pull/91 > > > Â 15. [76]https://github.com/taobao/tengine/pull/91 > > > Â 26. [87]http://mailman.nginx.org/mailman/listinfo/nginx > > > Â 37. [98]http://mailman.nginx.org/mailman/listinfo/nginx > > > Â 48. [109]http://mailman.nginx.org/mailman/listinfo/nginx > > > Â 59. mailto:[120]nginx@nginx.org > > [125]http://mailman.nginx.org/mailman/listinfo/nginx > References > 9. mailto:nginx@nginx.org > 20. mailto:nginx@nginx.org > 31. https://github.com/taobao/tengine/pull/91 > 42. https://github.com/taobao/tengine/pull/91 > 53. http://mailman.nginx.org/mailman/listinfo/nginx > 64. mailto:pasik@iki.fi > 75. mailto:zjay1987@gmail.com > 86. mailto:nginx@nginx.org > 97. mailto:nginx@nginx.org > 108. mailto:nginx@nginx.org > 119. http://mailman.nginx.org/mailman/listinfo/nginx > 120. mailto:nginx@nginx.org > 121. http://mailman.nginx.org/mailman/listinfo/nginx > 122. mailto:nginx@nginx.org > 123. http://mailman.nginx.org/mailman/listinfo/nginx > 124. mailto:nginx@nginx.org > 125. http://mailman.nginx.org/mailman/listinfo/nginx > 126. mailto:nginx@nginx.org > 127. http://mailman.nginx.org/mailman/listinfo/nginx
_______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
on 2013-02-26 22:11
This would be an excellent feature! Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234926,236640#msg-236640
on 2013-02-28 16:17
> I know nginx team are working on it. You can wait for it. Hopefully they will find a solution! Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234926,236723#msg-236723
on 2013-02-28 18:37
I'm also very interested in being able to configure nginx to NOT proxy the entire request. Regarding this patch, https://github.com/alibaba/tengine/pull/91, is anything fundamentally wrong with it? I don't understand Chinese so I'm at a loss here... Best regards, Andr Cruz
on 2013-02-28 19:13
Hello! On Thu, Feb 28, 2013 at 05:36:23PM +0000, André Cruz wrote: > I'm also very interested in being able to configure nginx to NOT > proxy the entire request. > > Regarding this patch, > https://github.com/alibaba/tengine/pull/91, is anything > fundamentally wrong with it? I don't understand Chinese so I'm > at a loss here... As a non-default mode of operation the aproach taken is likely good enough (not looked into details), but the patch won't work with current nginx versions - at least it needs (likely major) adjustments to cope with changes introduced during work on chunked request body support as available in nginx 1.3.9+. -- Maxim Dounin http://nginx.org/en/donation.html
on 2013-02-28 19:51
On Thursday 28 February 2013 21:36:23 Andr Cruz wrote: > I'm also very interested in being able to configure nginx to NOT proxy the > entire request. > [...] Actually, you can. http://nginx.org/r/proxy_set_body http://nginx.org/r/proxy_pass_request_body wbr, Valentin V. Bartenev
on 2013-02-28 20:03
On Feb 28, 2013, at 6:51 PM, Valentin V. Bartenev <vbart@nginx.com> wrote: > On Thursday 28 February 2013 21:36:23 Andr Cruz wrote: >> I'm also very interested in being able to configure nginx to NOT proxy the >> entire request. >> > [...] > > Actually, you can. > > http://nginx.org/r/proxy_set_body > http://nginx.org/r/proxy_pass_request_body I've probably explained myself wrong. What I want is for nginx to buffer only chunks of the request body and pass these chunks to the upstream server as they arrive. Andr
on 2013-03-01 14:22
On Thu, Feb 28, 2013 at 07:02:46PM +0000, Andr Cruz wrote: > > http://nginx.org/r/proxy_set_body > > http://nginx.org/r/proxy_pass_request_body > > I've probably explained myself wrong. What I want is for nginx to buffer only chunks of the request body and pass these chunks to the upstream server as they arrive. > Yes, the problem is nginx saves the request body to the local disk on the nginx server, filling the disk and making the upload slower. People want nginx to *not* save the http PUT/POST request to disk when using nginx as a reverse proxy. -- Pasi
on 2013-03-01 16:18
> Pasi Kärkkäinen > People want nginx to *not* save the http PUT/POST request to disk > when using nginx as a reverse proxy. Have the same problem. I use "haproxy" in front of "nginx" - but this more a bad hack than a workaround. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234926,236793#msg-236793
on 2013-03-05 12:25
Are there any plans to integrate this feature into NGINX? It would be very great. > Maxim Dounin: > As a non-default mode of operation the aproach taken is likely > good enough (not looked into details), but the patch won't work > with current nginx versions - at least it needs (likely major) > adjustments to cope with changes introduced during work on chunked > request body support as available in nginx 1.3.9+. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234926,236948#msg-236948
on 2013-03-08 11:27
> I keep getting the "upstream sent invalid header while reading response header from upstream" > error when using the no_buffer patch.. The patch does not work for you? Thanks Markus Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234926,237090#msg-237090
on 2013-03-08 11:51
On Thu, Mar 07, 2013 at 12:25:43PM -0500, double wrote: > > I keep getting the "upstream sent invalid header while reading response > header from upstream" > > error when using the no_buffer patch.. > > The patch does not work for you? > Thanks > Markus > Yep, the patch doesn't work for me. -- Pasi
on 2013-03-09 07:39
On Tue, Mar 05, 2013 at 03:17:41PM +0200, Pasi Kärkkäinen wrote: > I keep getting the "upstream sent invalid header while reading response header from upstream" > error when using the no_buffer patch.. > Is there something you'd want me to try? Adjusting some config options? Did you find anything weird in the log? Thanks! -- Pasi
on 2013-03-14 06:17
Try the new patch, It could solve your problem. Thanks for your test effort. 2013/3/8 Pasi Kärkkäinen <pasik@iki.fi>
on 2013-03-14 09:44
On Thu, Mar 07, 2013 at 07:48:37PM +0200, Pasi Krkkinen wrote: > Yep, the patch doesn't work for me. > the latest no_buffer_v5.patch seems to work OK for me with nginx 1.2.7. -- Pasi
on 2013-03-14 15:14
On Thu, Feb 28, 2013 at 10:12:47PM +0400, Maxim Dounin wrote: > Hello! > Hello, > As a non-default mode of operation the aproach taken is likely > good enough (not looked into details), but the patch won't work > with current nginx versions - at least it needs (likely major) > adjustments to cope with changes introduced during work on chunked > request body support as available in nginx 1.3.9+. > Weibin: Have you thought of upstreaming the no_buffer patch to nginx 1.3.x so it could become part of next nginx stable version 1.4 ? It'd be really nice to have the no_buffer functionality in stock nginx! (the current no_buffer_v5.patch seems to work OK for me on nginx 1.2.7) -- Pasi
on 2013-03-18 05:22
2013/3/14 Pasi Kärkkäinen <pasik@iki.fi> > > > > > > > Weibin: Have you thought of upstreaming the no_buffer patch to nginx 1.3.x > so it could become part of next nginx stable version 1.4 ? > You can see my first email in this thread, The nginx team are working on it. But I don't know when they finish it. You can wait for their version. Thanks.
on 2013-03-21 11:06
On Thu, Mar 14, 2013 at 10:39:12AM +0200, Pasi Kärkkäinen wrote: > On Thu, Mar 14, 2013 at 01:17:16PM +0800, Weibin Yao wrote: > > Try the new patch, It could solve your problem. > > Thanks for your test effort. > > > > Thanks a lot! > > I can confirm the "no_buffer_v5.patch" with nginx 1.2.7 fixes the problem for me, > and both HTTP POST and HTTP PUT requests work OK now without buffering to disk. > For the archives: v6 of the patch fixes some remaining problems: http://yaoweibin.cn/patches/nginx-1.2.7-no_buffer-v6.patch -- Pasi
on 2013-04-04 10:31
> Weibin: Have you thought of upstreaming the no_buffer patch to nginx 1.3.x > so it could become part of next nginx stable version 1.4 ? > It'd be really nice to have the no_buffer functionality in stock nginx! I agree. Nginx is such an excellent product - but this is a missing feature. Thanks Markus Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234926,238080#msg-238080
on 2013-04-18 11:28
Hmm, this patch is still not in nginx-1.3.16. It would be great to have this feature in vanilla nginx. Posted at Nginx Forum: http://forum.nginx.org/read.php?2,234926,238461#msg-238461
Enable email notification
Enable multi-page viewPlease log in before posting. Registration is free and takes only a minute.
Existing account
(Switch to SSL-encrypted connection)
NEW: Do you have a Google/GoogleMail or Yahoo account? No registration required!
Log in with Google account | Log in with Yahoo account
Log in with Google account | Log in with Yahoo account
No account? Register here.