Lua, memc and image_filter, image not resized via lua

Hello,

We attempt to make a simple image resizer and cacher using Nginx en
storing in Memcache and the image_filter. First we attempted to do it
via eval but we quickly ran into an old issue 1 i had, i can either
store a not-existing image in the cache or return it, never both.
Instead we tried using Lua scripting and now we can retrieve items from
the cache and store remote images in the cache and return them at the
same time!

But we have one problem left, the images are not passed through the
image filter at all when we do it via Lua.

I’ll share the config. The location below is called from outside and is
the only location externally called. If it’s in the cache, return it and
it it’s not in the cache, try to fetch it, store it in the cache and
return it.

  location /img/cache {
    set $memc_key $args;
    set $memc_cmd 'get';
    error_page 404 = /img/populate?$args;
    memc_pass 127.0.0.1:11211;
  }

The following location populates the cache and returns the (hopefully
resized) image. It uses a location to fetch the image (and resize it)
and another location to write the fetched and resized image to the
cache. It finally prints it.

  location /img/populate {
    content_by_lua '
        local args = ngx.req.get_uri_args(5)
        local image = ngx.location.capture("/img/crop?u=" .. 

args[“u”] … “&w=” … args[“w”] …“&h=” … args[“h”])
– Use anoter location to store the image in the cache
(cannot use memc_pass in this location it seems)
local res = ngx.location.capture(“/img/put?u=” … args[“u”]
… “&w=” … args[“w”] …“&h=” … args[“h”] … “&m=” … args[“m”], { body
= image[“body”] } )
ngx.print(image[“body”])
';
}

The following 2 simple locations resp. fetch and resize the image and
store it in memcache.

  location /img/crop {
    image_filter crop $arg_w $arg_h;
    proxy_pass $arg_u;
  }
  location /img/put {
    set $memc_key $args;
    set $memc_cmd 'set';
    memc_pass 127.0.0.1:11211;
  }

The problem is, the image is not resized! If we set the error_page in
the first location to /img/crop the image is properly resized so it
doesn’t work if called from Lua. I would be very grateful for any tips
and hints. I am curious to what i’m doing wrong.

Many thanks,
Markus

Hi,

Any thoughts to share on this one? I’m very stuck now and really want to
avoid using some backend :wink:

Thanks,
Markus

-----Original message-----

Hello!

On Fri, Sep 14, 2012 at 3:46 AM, Markus J.
[email protected] wrote:

Any thoughts to share on this one? I’m very stuck now and really want to avoid
using some backend :wink:

I’m going to look into your issue today :slight_smile:

Best regards,
-agentzh

Hello!

On Wed, Sep 12, 2012 at 8:02 AM, Markus J. wrote:

But we have one problem left, the images are not passed through the image filter
at all when we do it via Lua.

This is because the standard ngx_image_filter module’s output filter
always runs after ngx_lua’s subrequest capturing filter and there’s
no easy way to change this order. The response body of your subrequest
is captured by ngx_lua before the ngx_image_filter’s filter gets a
chance to run.

The solution is to do the image processing completely in Lua via
existing Lua libraries like lua-gd:

http://ittner.github.com/lua-gd/

Or you can take advantage of LuaJIT 2.0’s excellent FFI API to call
the libgd C API directly from within Lua:

http://luajit.org/ext_ffi_tutorial.html

Best regards,
-agentzh

Hello!

On Fri, Sep 14, 2012 at 3:22 PM, agentzh [email protected] wrote:

This is because the standard ngx_image_filter module’s output filter
always runs after ngx_lua’s subrequest capturing filter and there’s
no easy way to change this order. The response body of your subrequest
is captured by ngx_lua before the ngx_image_filter’s filter gets a
chance to run.

BTW, I’ve also just written a command-line utility named
ngx-body-filters that can dump out all the output body filters of any
running Nginx worker processes (on Linux) in the order they actually
run.

This tool is part of my Nginx SystemTap Toolkit:

https://github.com/agentzh/nginx-systemtap-toolkit#ngx-body-filters

For example, on my side, the nginx configured with both the ngx_lua
and ngx_image_filter modules will give me the following result:

$ ./ngx-body-filters -p 30132
Tracing 30132 (.../nginx/sbin/nginx)...

ngx_http_range_body_filter
ngx_http_copy_filter
ngx_output_chain
ngx_http_lua_capture_body_filter
ngx_http_image_body_filter
ngx_http_charset_body_filter
ngx_http_ssi_body_filter
ngx_http_postpone_filter
ngx_http_gzip_body_filter
ngx_http_chunked_body_filter
ngx_http_write_filter

113 microseconds elapsed in the probe handler.

where my nginx worker process pid is 30132.

From this output, we can see clearly that
ngx_http_lua_capture_body_filter indeed runs before
ngx_http_image_body_filter.

So this tool is really handy in determining if there is an issue with
the nginx output filter running order, which is often not so obvious
for many users.

Best regards,
-agentzh

Ilja, agentzh,

Thanks for sharing your comments. I was afraid it would be a phase
issue. I’ll check the toolkit and LuaGD.

-----Original message-----

You may consider user lua-gd, http://ittner.github.com/lua-gd/, which
works just fine with LuaNginx
We making crop+resize+masking of user-uploaded images in lua handlers
with no problems

On Sat, Sep 15, 2012 at 10:15 AM, agentzh [email protected] wrote:

and ngx_image_filter modules will give me the following result:
ngx_http_ssi_body_filter
ngx_http_lua_capture_body_filter indeed runs before
nginx mailing list
[email protected]
nginx Info Page


с уважением, Разинков Илья