# Ngx\_echo v0.35: ability to POST/PUT local files as subrequest bodies **URL:** \ **Category:** NGINX\ **Created:** [February 7, 2011, 12:02pm UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838 "2011-02-07T12:02:37Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\ **Post date:** [February 7, 2011, 12:02pm UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/1 "2011-02-07T12:02:37Z")
Hi, folks! I’m happy to announce the v0.35 release of the ngx\_echo module, which features the ability to POST/PUT location files as subrequest bodies via the -f option of the echo\_subrequest and echo\_subrequest\_async directives. You can get the release tarball from the download page below [https://github.com/agentzh/echo-nginx-module/downloads](https://github.com/agentzh/echo-nginx-module/downloads) This new feature was implemented by Bernd Dorn. Many thanks to him 😉 Bernd uses the nginx upload module to upload big files and he wants to issue a PUT subrequest with the uploaded file to store the file somewhere else based on its hash (hash generation is done by the upload module). So it’s nice to have something like a file option to replace the “-b” option in the echo\_subrequest directive like this: echo\_subrequest PUT /some\_upstream/$upload\_file\_md5 -f $upload\_tmp\_path The echo module wraps lots of Nginx internal APIs for streaming input and output, parallel/sequential subrequests, timers and sleeping, as well as various meta data accessing. You can read the full documentation on the following wiki page: [HTTP Echo Module | NGINX](http://wiki.nginx.org/HttpEchoModule) and you can always get the latest source code from GitHub: [GitHub - openresty/echo-nginx-module: An Nginx module for bringing the power of "echo", "sleep", "time" and more to Nginx's config file](http://github.com/agentzh/echo-nginx-module) Happy echoing! Best regards, -agentzh ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [February 7, 2011, 8:44pm UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/2 "2011-02-07T20:44:09Z")
agentzh wrote: > This new feature was implemented by Bernd Dorn. Many thanks to him 😉 > > Bernd uses the nginx upload module to upload big files and he wants to > issue a PUT subrequest with the uploaded file to store the file > somewhere else based on its hash (hash generation is done by the > upload module). So it’s nice to have something like a file option to > replace the “-b” option in the echo\_subrequest directive like this: > > ``` > echo_subrequest PUT /some_upstream/$upload_file_md5 -f $upload_tmp_path > > ``` That’s bizarre! Combined with mogilefs module this gives an ultimately scalable storage solution out of the box for free! Congrats agentzh! > The echo module wraps lots of Nginx internal APIs for streaming input > and output, parallel/sequential subrequests, timers and sleeping, as > well as various meta data accessing. You can read the full > documentation on the following wiki page: > > ``` > http://wiki.nginx.org/HttpEchoModule > > ``` > > and you can always get the latest source code from GitHub: > > ``` > http://github.com/agentzh/echo-nginx-module > > ``` – Best regards, Valery K. ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [February 8, 2011, 9:25am UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/3 "2011-02-08T09:25:06Z")
On Tue, Feb 8, 2011 at 3:41 AM, Valery K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > That’s bizarre! Combined with mogilefs module this gives an ultimately > scalable storage solution out of the box for free! Yup, indeed 🙂 > Congrats agentzh! Thanks! Happy hacking! -agentzh ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [March 11, 2011, 2:21am UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/4 "2011-03-11T02:21:48Z")
Hello, could this be a way to put POST multipart uploads to redis/memcache? I look for a solution to put small Images to redis or memcached. I am right, if i use a fileread with lua in nginx the script is blocking? Any ideas how i can store POST multipart uploads nonblocking? I know there is a great file upload module for nginx, but what happens if i use lua to work on an uploaded file like move / rename / copy uploaded files, this blocks nginx? How can i check if something block nginx? Thanks for your help. Elena Hi, folks! I’m happy to announce the v0.35 release of the ngx\_echo module, which features the ability to POST/PUT location files as subrequest bodies via the -f option of the echo\_subrequest and echo\_subrequest\_async directives. You can get the release tarball from the download page below ``` https://github.com/agentzh/echo-nginx-module/downloads ``` This new feature was implemented by Bernd Dorn. Many thanks to him 😉 Bernd uses the nginx upload module to upload big files and he wants to issue a PUT subrequest with the uploaded file to store the file somewhere else based on its hash (hash generation is done by the upload module). So it’s nice to have something like a file option to replace the “-b” option in the echo\_subrequest directive like this: ``` echo_subrequest PUT /some_upstream/$upload_file_md5 -f ``` $upload\_tmp\_path The echo module wraps lots of Nginx internal APIs for streaming input and output, parallel/sequential subrequests, timers and sleeping, as well as various meta data accessing. You can read the full documentation on the following wiki page: ``` http://wiki.nginx.org/HttpEchoModule ``` and you can always get the latest source code from GitHub: ``` http://github.com/agentzh/echo-nginx-module ``` Happy echoing! Best regards, -agentzh * * * nginx mailing list removed\_email\_address@domain.invalid [http://nginx.org/mailman/listinfo/nginx](http://nginx.org/mailman/listinfo/nginx) – Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! [https://freundschaftswerbung.gmx.de](https://freundschaftswerbung.gmx.de) ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [March 11, 2011, 3:06am UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/5 "2011-03-11T03:06:45Z")
Thanks so much for the detailed information. Storing uploaded data nonblocking sounds not so easy… What do you think about a solution sending data as base64 encoded sting, then i can use your form-input addon and send this data directly to redis? Have a nice day. Elena -------- Original-Nachricht -------- > Datum: Thu, 10 Mar 2011 16:28:54 +0800 > Von: agentzh [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > An: [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > Betreff: Re: [ANN] ngx\_echo v0.35: ability to POST/PUT local files as > subrequest bodies > and check the results? 😉 > Cheers, > -agentzh > > * * * > > nginx mailing list > [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) > [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx) – Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir belohnen Sie mit bis zu 50,- Euro! [https://freundschaftswerbung.gmx.de](https://freundschaftswerbung.gmx.de) ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [March 11, 2011, 3:38am UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/6 "2011-03-11T03:38:09Z")
On Thu, Mar 10, 2011 at 5:01 PM, Elena Z. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Thanks so much for the detailed information. > > Storing uploaded data nonblocking sounds not so easy… What do you think about > a solution sending data as base64 encoded sting, then i can use your form-input > addon and send this data directly to redis? When your data is small enough to be hold in RAM, just use ngx\_lua as the glue. Use ``` lua_need_request_body on in nginx.conf + client_body_in_single_buffer on in nginx.conf + ngx.var.request_body + ngx.decode_base64() + ``` ngx.location.capture() in Lua + ngx\_redis2 in nginx.conf Then it’ll no longer block on disk operations because no disk operations will be involved. Cheers, -agentzh ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [March 11, 2011, 2:34am UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/7 "2011-03-11T02:34:55Z")
On Thu, Mar 10, 2011 at 4:16 PM, Elena Z. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Hello, > > could this be a way to put POST multipart uploads to redis/memcache? Yes 🙂 > I am right, if i use a fileread with lua in nginx the script is blocking? Yup. > Any ideas how i can store POST multipart uploads nonblocking? Use echo\_subrequest POST /foo -f /path/to/file for that (combined with ngx\_upload, maybe). > I know there is a great file upload module for nginx, but what happens if i use > lua to work on an uploaded file like move / rename / copy uploaded files, this > blocks nginx? Yes. Most disk operations will block nginx (or some other processes like php). > How can i check if something block nginx? Maybe configure only 1 nginx worker and try ``` ab -c 100 -n 10000 http://... ``` and check the results? 😉 There will be a “body\_file” option for ngx\_lua’s “ngx.location.capture” and “ngx.location.capture\_multi” Lua functions, similar to echo\_subrequest’s -f option. But nginx may also block even if itself does the disk operations. A work-around is to start a few more nginx worker processes, just like how apache and php-fpm has been doing for years 😉 Cheers, -agentzh ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [March 15, 2011, 11:17pm UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/8 "2011-03-15T23:17:26Z")
Hello, any chance to debug the -f function? I use the debug log, but get no information about the file… /body1 shows me “test” but /body2 shows nothing… no error, no body content… ``` location /body1 { echo_subrequest POST /sub -b 'test'; } location /body2 { echo_subrequest POST /sub -f /tmp/hello.txt; } location /sub { echo "body: $echo_request_body"; } ``` Thanks for your help. Alexander ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [March 16, 2011, 8:24am UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/9 "2011-03-16T08:24:17Z")
Thanks so much for your detailed answer. Works like a charm 🙂 But i see i can’t access this from LUA? > The |ngx.location.capture| and |ngx.location.capture\_multi| Lua > methods cannot > capture locations configured by ngx\_echo module’s |echo\_location|, > |echo\_location\_async|, | > echo\_subrequest|, or |echo\_subrequest\_async| directives. This won’t > be fixed in the future due to technical problems 🙂 but is it possible to get echo\_request\_body directly in nginx LUA? Thanks. Alexander ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [March 16, 2011, 3:29am UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/10 "2011-03-16T03:29:45Z")
On Wed, Mar 16, 2011 at 6:15 AM, Alexander K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Hello, > > any chance to debug the -f function? I use the debug log, but get no > information about the file… /body1 > shows me “test” but /body2 shows nothing… no error, no body content… Because there’s no error in your /body2 request, no log is expected to be printed 😉 > ``` > } > > ``` Why $echo\_request\_body is empty for /body2? Because “echo\_subrequest /sub -f /tmp/hello.txt” produces a request body that consists of _file_ buffers only (such that nginx does not bother reading the whole disk file into RAM at once) and $echo\_request\_body only respects pure _memory_ buffers. The following example comes from the ngx\_echo’s test suite: ``` location /main { echo_subrequest POST /sub -f html/blah.txt; } location /sub { echo "sub method: $echo_request_method"; # we don't need to call echo_read_client_body explicitly here echo_request_body; } ``` where html/blah.txt contains the following contents: ``` Hello, world ``` Then GET /main gives ``` sub method: POST Hello, world ``` Cheers, -agentzh ---
**Author:** ![agentzh](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/4da419/32.png) [@agentzh](https://www.ruby-forum.com/u/agentzh)\ **Post date:** [March 16, 2011, 8:38am UTC](https://www.ruby-forum.com/t/ngx-echo-v0-35-ability-to-post-put-local-files-as-subrequest-bodies/202838/11 "2011-03-16T08:38:43Z")
On Wed, Mar 16, 2011 at 3:22 PM, Alexander K. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote: > Thanks so much for your detailed answer. Works like a charm 🙂 Cool 🙂 > But i see i can’t access this from LUA? No 🙂 > but is it possible to get > > echo\_request\_body > > directly in nginx LUA? It indeed can be ported from ngx\_echo to ngx\_lua. Patches welcome 🙂 For now you can just read your disk files directly from within Lua. See ``` http://www.lua.org/pil/21.2.html ``` It may be less efficient, depending on the size of your disk files. But generally, it can not be too slow. No request body hassle is really needed here, I think 😉 Cheers, -agentzh