Ngx_echo v0.35: ability to POST/PUT local files as subrequest bodies

[luislavena]

Hi, folks!

I’m happy to announce the v0.35 release of the ngx_echo module, which
features the ability to POST/PUT location files as subrequest bodies
via the -f option of the echo_subrequest and echo_subrequest_async
directives.

You can get the release tarball from the download page below

https://github.com/agentzh/echo-nginx-module/downloads

This new feature was implemented by Bernd Dorn. Many thanks to him

Bernd uses the nginx upload module to upload big files and he wants to
issue a PUT subrequest with the uploaded file to store the file
somewhere else based on its hash (hash generation is done by the
upload module). So it’s nice to have something like a file option to
replace the “-b” option in the echo_subrequest directive like this:

echo_subrequest PUT /some_upstream/$upload_file_md5 -f $upload_tmp_path

The echo module wraps lots of Nginx internal APIs for streaming input
and output, parallel/sequential subrequests, timers and sleeping, as
well as various meta data accessing. You can read the full
documentation on the following wiki page:

HTTP Echo Module | NGINX

and you can always get the latest source code from GitHub:

GitHub - openresty/echo-nginx-module: An Nginx module for bringing the power of "echo", "sleep", "time" and more to Nginx's config file

Happy echoing!

Best regards,
-agentzh

[agentzh]

agentzh wrote:

This new feature was implemented by Bernd Dorn. Many thanks to him

Bernd uses the nginx upload module to upload big files and he wants to
issue a PUT subrequest with the uploaded file to store the file
somewhere else based on its hash (hash generation is done by the
upload module). So it’s nice to have something like a file option to
replace the “-b” option in the echo_subrequest directive like this:

 echo_subrequest PUT /some_upstream/$upload_file_md5 -f $upload_tmp_path

That’s bizarre! Combined with mogilefs module this gives an ultimately
scalable storage solution out of the box for free!

Congrats agentzh!

The echo module wraps lots of Nginx internal APIs for streaming input
and output, parallel/sequential subrequests, timers and sleeping, as
well as various meta data accessing. You can read the full
documentation on the following wiki page:

http://wiki.nginx.org/HttpEchoModule

and you can always get the latest source code from GitHub:

http://github.com/agentzh/echo-nginx-module

–
Best regards,
Valery K.

[agentzh]

On Tue, Feb 8, 2011 at 3:41 AM, Valery K.
[email protected] wrote:

That’s bizarre! Combined with mogilefs module this gives an ultimately
scalable storage solution out of the box for free!

Yup, indeed

Congrats agentzh!

Thanks!

Happy hacking!
-agentzh

[agentzh]

Hello,

could this be a way to put POST multipart uploads to redis/memcache? I
look for a solution to put small Images to redis or memcached.

I am right, if i use a fileread with lua in nginx the script is
blocking? Any ideas how i can store POST multipart uploads nonblocking?
I know there is a great file upload module for nginx, but what happens
if i use lua to work on an uploaded file like move / rename / copy
uploaded files, this blocks nginx? How can i check if something block
nginx?

Thanks for your help.

Elena

Hi, folks!

I’m happy to announce the v0.35 release of the ngx_echo module, which
features the ability to POST/PUT location files as subrequest bodies
via the -f option of the echo_subrequest and echo_subrequest_async
directives.

You can get the release tarball from the download page below

https://github.com/agentzh/echo-nginx-module/downloads

This new feature was implemented by Bernd Dorn. Many thanks to him

Bernd uses the nginx upload module to upload big files and he wants to
issue a PUT subrequest with the uploaded file to store the file
somewhere else based on its hash (hash generation is done by the
upload module). So it’s nice to have something like a file option to
replace the “-b” option in the echo_subrequest directive like this:

 echo_subrequest PUT /some_upstream/$upload_file_md5 -f 

$upload_tmp_path

The echo module wraps lots of Nginx internal APIs for streaming input
and output, parallel/sequential subrequests, timers and sleeping, as
well as various meta data accessing. You can read the full
documentation on the following wiki page:

http://wiki.nginx.org/HttpEchoModule

and you can always get the latest source code from GitHub:

http://github.com/agentzh/echo-nginx-module

Happy echoing!

Best regards,
-agentzh

---

nginx mailing list
[email protected]
http://nginx.org/mailman/listinfo/nginx

–
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de

[agentzh]

Thanks so much for the detailed information.

Storing uploaded data nonblocking sounds not so easy… What do you
think about a solution sending data as base64 encoded sting, then i can
use your form-input addon and send this data directly to redis?

Have a nice day.

Elena

-------- Original-Nachricht --------

Datum: Thu, 10 Mar 2011 16:28:54 +0800
Von: agentzh [email protected]
An: [email protected]
Betreff: Re: [ANN] ngx_echo v0.35: ability to POST/PUT local files as
subrequest bodies

and check the results?
Cheers,
-agentzh

---

nginx mailing list
[email protected]
nginx Info Page

–
Empfehlen Sie GMX DSL Ihren Freunden und Bekannten und wir
belohnen Sie mit bis zu 50,- Euro! https://freundschaftswerbung.gmx.de

[agentzh]

On Thu, Mar 10, 2011 at 5:01 PM, Elena Z. [email protected] wrote:

Thanks so much for the detailed information.

Storing uploaded data nonblocking sounds not so easy… What do you think about
a solution sending data as base64 encoded sting, then i can use your form-input
addon and send this data directly to redis?

When your data is small enough to be hold in RAM, just use ngx_lua as
the glue. Use

lua_need_request_body on in nginx.conf
   + client_body_in_single_buffer on in nginx.conf
   + ngx.var.request_body + ngx.decode_base64() +

ngx.location.capture() in Lua
+ ngx_redis2 in nginx.conf

Then it’ll no longer block on disk operations because no disk
operations will be involved.

Cheers,
-agentzh

[agentzh]

On Thu, Mar 10, 2011 at 4:16 PM, Elena Z. [email protected] wrote:

Hello,

could this be a way to put POST multipart uploads to redis/memcache?

Yes

I am right, if i use a fileread with lua in nginx the script is blocking?

Yup.

Any ideas how i can store POST multipart uploads nonblocking?

Use echo_subrequest POST /foo -f /path/to/file for that (combined with
ngx_upload, maybe).

I know there is a great file upload module for nginx, but what happens if i use
lua to work on an uploaded file like move / rename / copy uploaded files, this
blocks nginx?

Yes. Most disk operations will block nginx (or some other processes like
php).

How can i check if something block nginx?

Maybe configure only 1 nginx worker and try

ab -c 100 -n 10000 http://...

and check the results?

There will be a “body_file” option for ngx_lua’s
“ngx.location.capture” and “ngx.location.capture_multi” Lua functions,
similar to echo_subrequest’s -f option.

But nginx may also block even if itself does the disk operations.

A work-around is to start a few more nginx worker processes, just like
how apache and php-fpm has been doing for years

Cheers,
-agentzh

[agentzh]

Hello,

any chance to debug the -f function? I use the debug log, but get no
information about the file… /body1
shows me “test” but /body2 shows nothing… no error, no body content…

   location /body1 {
         echo_subrequest POST /sub -b 'test';
     }

   location /body2 {
         echo_subrequest POST /sub -f /tmp/hello.txt;
     }

     location /sub {
         echo "body: $echo_request_body";
     }

Thanks for your help.

Alexander

[agentzh]

Thanks so much for your detailed answer. Works like a charm

But i see i can’t access this from LUA?

The |ngx.location.capture| and |ngx.location.capture_multi| Lua
methods cannot
capture locations configured by ngx_echo module’s |echo_location|,
|echo_location_async|, |
echo_subrequest|, or |echo_subrequest_async| directives. This won’t
be fixed in the future due to technical problems

but is it possible to get

echo_request_body

directly in nginx LUA?

Thanks.

Alexander

[agentzh]

On Wed, Mar 16, 2011 at 6:15 AM, Alexander K. [email protected]
wrote:

Hello,

any chance to debug the -f function? I use the debug log, but get no
information about the file… /body1
shows me “test” but /body2 shows nothing… no error, no body content…

Because there’s no error in your /body2 request, no log is expected to
be printed

   }

Why $echo_request_body is empty for /body2? Because “echo_subrequest
/sub -f /tmp/hello.txt” produces a request body that consists of
file buffers only (such that nginx does not bother reading the whole
disk file into RAM at once) and $echo_request_body only respects pure
memory buffers.

The following example comes from the ngx_echo’s test suite:

location /main {
    echo_subrequest POST /sub -f html/blah.txt;
}

location /sub {
    echo "sub method: $echo_request_method";
    # we don't need to call echo_read_client_body explicitly here
    echo_request_body;
}

where html/blah.txt contains the following contents:

Hello, world

Then GET /main gives

sub method: POST
Hello, world

Cheers,
-agentzh

[agentzh]

On Wed, Mar 16, 2011 at 3:22 PM, Alexander K. [email protected]
wrote:

Thanks so much for your detailed answer. Works like a charm

Cool

But i see i can’t access this from LUA?

No

but is it possible to get

echo_request_body

directly in nginx LUA?

It indeed can be ported from ngx_echo to ngx_lua. Patches welcome

For now you can just read your disk files directly from within Lua. See

http://www.lua.org/pil/21.2.html

It may be less efficient, depending on the size of your disk files.
But generally, it can not be too slow.

No request body hassle is really needed here, I think

Cheers,
-agentzh