So is "rewrite_by_lua" also evil?

I have the following simplified setup …

server {

location @proxy {
include /etc/nginx/firewall.default;
proxy_pass http://127.0.0.1:8080;

}
location ~ ^.+.php$ {
content_by_lua ‘ngx.exec("@proxy");’;
}
location / {
try_files $uri $uri/ @proxy;
}
}
Basically, everything that cannot be found by nginx, as well as php
requests, are sent to the proxy

Now, note the filter.default file in the @proxy location. I use this
to run some tests on these requests for security and my logs show them
catching all sorts of exploit attempts.

Anyway, when I have the following (simplified) in firewall.default …

if ($http_user_agent ~* libwww ) {
return 403;
}

… everything is fine. When a php request is made, libwww user agents
are denied and others get the php output.

When I use the following (simplified) rewrite_by_lua equivalent instead

rewrite_by_lua ’
if ngx.var.http_user_agent == “libwww” then
ngx.exit(ngx.HTTP_FORBIDDEN)
end
';

The php file is downloaded. Obviously I don’t have the “libwww” when
testing so I suppose the lua “if” block is skipped at which point the
physical php file is found and sent to the user as is and the
proxy_pass directive is not run.

Looks similar to the sort of unexpected behaviour from the rewrite
module’s “if”.

Any ideas what gives? Why isn’t rewrite_by_lua behaving like the rewrite
module?

Thanks

On Wed, Oct 12, 2011 at 12:03 AM, Nginx U. [email protected]
wrote:

Anyway, when I have the following (simplified) in firewall.default …

if ($http_user_agent ~* libwww ) {
return 403;
}

… everything is fine. When a php request is made, libwww user agents
are denied and others get the php output.

You’re using the “~*” operator here and that means “case insensitive
match”, see http://wiki.nginx.org/HttpCoreModule#location

When I use the following (simplified) rewrite_by_lua equivalent instead …

rewrite_by_lua ’
if ngx.var.http_user_agent == “libwww” then
ngx.exit(ngx.HTTP_FORBIDDEN)
end
';

Note that you’re using “==” here in Lua which is exact string comparison
:slight_smile:

The php file is downloaded. Obviously I don’t have the “libwww” when
testing so I suppose the lua “if” block is skipped at which point the
physical php file is found and sent to the user as is and the
proxy_pass directive is not run.

Which version of ngx_lua are you using? Please show me your “nginx -V”
output? And which OS are you using? I’ve tested your example with
ngx_lua git master HEAD on Slackware Linux x86_64 and do not have any
issues :slight_smile:

Also, enabling --with-debug in your nginx build and show me the
relevant sections of your error.log on the debug error log level will
be helpful too :slight_smile:

Regards,
-agentzh

Hi
On 12 October 2011 03:15, agentzh [email protected] wrote:

Note that you’re using “==” here in Lua which is exact string comparison :slight_smile:

It is a simplified config. I actually run lua’s string.find first and
test for a hit.
I know ngx.re.match with the “i” modifier would be better but it does
not work … maybe I need to update lua module version. I thought I’ll
look into that later.
Anyway, the exact implementation will cause the “if” block to be
skipped.

The php file is downloaded. Obviously I don’t have the “libwww” when
testing so I suppose the lua “if” block is skipped at which point the
physical php file is found and sent to the user as is and the
proxy_pass directive is not run.

Which version of ngx_lua are you using? Please show me your “nginx -V”
output? And which OS are you using? I’ve tested your example with
ngx_lua git master HEAD on Slackware Linux x86_64 and do not have any
issues :slight_smile:

Centos 5.7 i386.
lua module is 0.2.0

nginx: nginx version: nginx/1.0.6
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-50)
nginx: TLS SNI support disabled
nginx: configure arguments: --user=nginx --group=nginx
–prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log
–http-client-body-temp-path=/var/lib/nginx/tmp/client_body
–http-proxy-temp-path=/var/lib/nginx/tmp/proxy
–http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
–pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx
–with-http_ssl_module --with-http_realip_module
–with-http_stub_status_module --with-http_perl_module --with-mail
–with-mail_ssl_module --with-cc-opt=’-O2 -g -pipe -Wall
-Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector
–param=ssp-buffer-size=4 -m32 -march=i386 -mtune=generic
-fasynchronous-unwind-tables’
–add-module=/usr/src/redhat/BUILD/nginx-1.0.6/nginx-upstream-fair
–add-module=/usr/src/redhat/BUILD/nginx-1.0.6/ngx_cache_purge-1.3
–add-module=/usr/src/redhat/BUILD/nginx-1.0.6/ngx-headers-more
–add-module=/usr/src/redhat/BUILD/nginx-1.0.6/ngx-devel-kit
–add-module=/usr/src/redhat/BUILD/nginx-1.0.6/ngx-echo-module
–add-module=/usr/src/redhat/BUILD/nginx-1.0.6/ngx-lua-module
–add-module=/usr/src/redhat/BUILD/nginx-1.0.6/ngx_auth_request-module
–without-http_autoindex_module --without-http_empty_gif_module
–without-http_memcached_module --without-http_scgi_module
–without-http_split_clients_module --without-http_ssi_module
–without-http_upstream_ip_hash_module --without-http_uwsgi_module

Also, enabling --with-debug in your nginx build and show me the
relevant sections of your error.log on the debug error log level will
be helpful too :slight_smile:

I’ll get this later.

Thanks

On Wed, Oct 12, 2011 at 12:12 PM, Nginx U. [email protected]
wrote:

It is a simplified config. I actually run lua’s string.find first and
test for a hit.
I know ngx.re.match with the “i” modifier would be better but it does
not work

The ngx.re API was first introduced in the v0.2.1rc11 release. There’s
no wonder that it does not work for you because you’re using v0.2.0 :slight_smile:

… maybe I need to update lua module version. I thought I’ll
look into that later.

Surely you need. Please do that before other attempts :slight_smile:

Which version of ngx_lua are you using? Please show me your “nginx -V”
output? And which OS are you using? I’ve tested your example with
ngx_lua git master HEAD on Slackware Linux x86_64 and do not have any
issues :slight_smile:

Centos 5.7 i386.
lua module is 0.2.0

Sigh, your ngx_lua module is way too old :slight_smile:

There’s a bunch of important fixes regarding ngx.exec() and ngx.exit()
since the ngx_lua v0.2.0 release.

The latest version is v0.3.1rc9 and please try it out.

Also, enabling --with-debug in your nginx build and show me the
relevant sections of your error.log on the debug error log level will
be helpful too :slight_smile:

I’ll get this later.

Please upgrade your ngx_lua module to the latest version before doing
this :slight_smile:

Regards,
-agentzh

On 12 October 2011 07:20, agentzh [email protected] wrote:

Sigh, your ngx_lua module is way too old :slight_smile:

There’s a bunch of important fixes regarding ngx.exec() and ngx.exit()
since the ngx_lua v0.2.0 release.

The latest version is v0.3.1rc9 and please try it out.
OK thanks. I need to do a build for Nginx 1.0.8 and update my 3rd
party modules with new nginx builds.
I don’t install the rc versions of 3rd party modules though. I only
use full releases which means 0.3.0 in this case … expect if 0.3.1
is released in the interim of course.

Anyway, nice to know ngx_lua is not “evil” (as long as it is 0.2.1+!)

Thanks

try open this link…i hope is usefull to you
www.anilcetin.com/convert-apache-htaccess-to-nginx/

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,216525,216581#msg-216581

On 12 October 2011 13:23, MyName [email protected] wrote:

try open this link…i hope is usefull to you
www.anilcetin.com/convert-apache-htaccess-to-nginx/
I don’t need to convert Apache htaccess to Nginx but thanks for the link
anyway.

Cheers.

On 12 October 2011 21:35, Nginx U. [email protected] wrote:

if method == nil then
See debug output: http://pastebin.com/A7LNeCf0
Some additional info.

I had thought the php files were being downloaded as text files but
this is not strictly true. Files are downloaded whenever php is
called but they are always empty with a “0” status code.

The associated access log output for the request in the debug log in
the paste bin is: xx.xxx.xx.xx - - [12/Oct/2011:18:20:50 +0000] “GET
/help/ HTTP/1.1” 0 0 “http://testsite/index.php” “Mozilla/5.0
(Macintosh; Intel Mac OS X 10_5_8) AppleWebKit/534.50.2 (KHTML, like
Gecko) Version/5.0.6 Safari/533.22.3”

On Thu, Oct 13, 2011 at 2:35 AM, Nginx U. [email protected] wrote:

   if method == nil then
       method = ngx.re.match(ngx.var.request_method, "HEAD", "i")
       if method == nil then
           ngx.exit(ngx.HTTP_BAD_REQUEST)
       end
   end

end
';

BTW, you could have written the Lua code like this:

local m = ngx.re.match(ngx.var.request_method, “GET|POST|HEAD”, “io”)
if not m then
ngx.exit(400)
end

The ngx.re.match uses the same PCRE engine as ngx_rewrite’s “if”
anyway. Also, because the regex argument is a constant here, the “o”
flag can be specified to cache the compiled regex object on the global
level.

See debug output: http://pastebin.com/A7LNeCf0

The debug log data you’ve provided contains a lot of references to the
location @pretty_urls which is missing from your original nginx.conf
snippet. Could you please provide a minimized but complete nginx.conf
that can reproduce this problem as well as the corresponding debug.log
snippet?

Thanks!
-agentzh

On Thu, Oct 13, 2011 at 5:34 AM, Nginx U. [email protected] wrote:

See debug output: http://pastebin.com/A7LNeCf0

Some additional info.

I had thought the php files were being downloaded as text files but
this is not strictly true. Files are downloaded whenever php is
called but they are always empty with a “0” status code.

This information is invaluable :slight_smile: I’ve already reproduced this bug of
ngx_lua on my side and fixed it in my local tree.

Basically, calling ngx.exec to jump to a named location is buggy in
that ngx_http_named_location() in the nginx core does not clear module
ctx structures as ngx_http_internal_redirect() so ngx_lua should have
cleared its ctx itself.

I’ll tag another rc release of ngx_lua for you to try when I finish
running the whole test suite :slight_smile:

Thank you very much for the bug report! Hopefully it’ll be no longer
evil :wink:

Thanks!
-agentzh

On Thu, Oct 13, 2011 at 10:06 AM, agentzh [email protected] wrote:

Please try out the latest ngx_lua v0.3.1rc10 release:

https://github.com/chaoslawful/lua-nginx-module/tags

Thanks!
-agentzh

Hi.

Updated Nginx to 1.0.8 and did my 3rd party modules as well and even
went for rc versions

nginx: nginx version: nginx/1.0.8
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-51)
nginx: configure arguments: --user=nginx --group=nginx
–prefix=/usr/share/nginx --sbin-path=/usr/sbin/nginx
–conf-path=/etc/nginx/nginx.conf
–error-log-path=/var/log/nginx/error.log
–http-log-path=/var/log/nginx/access.log
–http-client-body-temp-path=/var/lib/nginx/tmp/client_body
–http-proxy-temp-path=/var/lib/nginx/tmp/proxy
–http-fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi
–pid-path=/var/run/nginx.pid --lock-path=/var/lock/subsys/nginx
–with-http_stub_status_module --with-http_perl_module --with-debug
–without-http_memcached_module --without-http_scgi_module
–without-http_split_clients_module --without-http_ssi_module
–without-http_upstream_ip_hash_module --without-http_uwsgi_module
–add-module=/usr/src/redhat/BUILD/nginx-1.0.8/nginx-upstream-fair
–add-module=/usr/src/redhat/BUILD/nginx-1.0.8/ngx_cache_purge-1.4
–add-module=/usr/src/redhat/BUILD/nginx-1.0.8/ngx-headers-more-0.16rc3-0
–add-module=/usr/src/redhat/BUILD/nginx-1.0.8/ngx-devel-kit-0.2.17-4
–add-module=/usr/src/redhat/BUILD/nginx-1.0.8/ngx-echo-module-0.37rc5-0
–add-module=/usr/src/redhat/BUILD/nginx-1.0.8/ngx-lua-module-0.31rc9-0
–add-module=/usr/src/redhat/BUILD/nginx-1.0.8/ngx_auth_request-module
–with-cc-opt=’-O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector --param=ssp-buffer-size=4 -m32 -march=i386
-mtune=generic -fasynchronous-unwind-tables’

Still seeing some evil type behaviour though.

This renders php as expected…

if ($request_method !~* (GET|HEAD|POST) ) {
return 400;
}

… while this sends php files as text downloads …

rewrite_by_lua ’
method = ngx.re.match(ngx.var.request_method, “GET”, “i”)
if method == nil then
method = ngx.re.match(ngx.var.request_method, “POST”, “i”)
if method == nil then
method = ngx.re.match(ngx.var.request_method, “HEAD”, “i”)
if method == nil then
ngx.exit(ngx.HTTP_BAD_REQUEST)
end
end
end
';

See debug output: http://pastebin.com/A7LNeCf0

Cheers

On 13 October 2011 04:34, agentzh [email protected] wrote:

The debug log data you’ve provided contains a lot of references to the
location @pretty_urls which is missing from your original nginx.conf
snippet. Could you please provide a minimized but complete nginx.conf
that can reproduce this problem as well as the corresponding debug.log
snippet?

server {

location @pretty_urls {
# some rewrite rules to php
}
location @proxy {
include /etc/nginx/firewall.default;
proxy_pass http://127.0.0.1:8080;

}
location ~ ^.+.php$ {
content_by_lua ‘ngx.exec("@proxy");’;
}
location / {
try_files $uri $uri/ @pretty_urls;
}
}

If request ends in “.php” it is sent to @proxy by php location. If not
“.php”, Nginx tries to find and serve resource. If not found, sent to
@pretty_urls where it will either be rewritten to “.php” or a 404
error returned.

On 13 October 2011 05:06, agentzh [email protected] wrote:

Basically, calling ngx.exec to jump to a named location is buggy in
that ngx_http_named_location() in the nginx core does not clear module
ctx structures as ngx_http_internal_redirect() so ngx_lua should have
cleared its ctx itself.
I see. Don’t know much about this but does this mean the httpRewrite
module does clear this ctx? I.E. why doesn’t this affect the standard
rewrite module?

I’ll tag another rc release of ngx_lua for you to try when I finish
running the whole test suite :slight_smile:

Thank you very much for the bug report! Hopefully it’ll be no longer evil :wink:
Thanks are to you.

On 13 October 2011 07:30, Nginx U. [email protected] wrote:

   # some rewrite rules to php
   try_files $uri $uri/ @pretty_urls;

}
}

If request ends in “.php” it is sent to @proxy by php location. If not
“.php”, Nginx tries to find and serve resource. If not found, sent to
@pretty_urls where it will either be rewritten to “.php” or a 404
error returned.

PS: I had earlier thought the issue might be with the content_by_lua
‘ngx.exec("@proxy");’; line but when I tried to use the following …

  location ~ ^.+\.php$ {
          rewrite_by_lua 'ngx.exec("@proxy");';
  }

… which I thought should work and seems more appropriate, I got an
error equivalent to issuing “return 444”.

On 13 October 2011 06:57, agentzh [email protected] wrote:

Please try out the latest ngx_lua v0.3.1rc10 release:
I’ll have a bash later!

Cheers!

On Thu, Oct 13, 2011 at 12:32 PM, Nginx U. [email protected]
wrote:

On 13 October 2011 05:06, agentzh [email protected] wrote:

Basically, calling ngx.exec to jump to a named location is buggy in
that ngx_http_named_location() in the nginx core does not clear module
ctx structures as ngx_http_internal_redirect() so ngx_lua should have
cleared its ctx itself.
I see. Don’t know much about this but does this mean the httpRewrite
module does clear this ctx? I.E. why doesn’t this affect the standard
rewrite module?

For ngx_lua, for example, it does have a flag field named
“entered_content_phase” in its context object (ctx) which marks
whether the current request has entered the content phase. So when
doing an internal redirection to a named location, this field must be
cleared out or some bad things will happen in rewrite_by_lua because
ngx_lua now believes it’s in a content_by_lua not rewrite_by_lua.

Well, there’re more flags like this in ngx_lua ctx. For ngx_rewrite,
it’s probably safe just because of its simplicity.

Well, I still believe it is a bug in ngx_http_named_location function
in the nginx core. The thumb of rule is to avoid using named locations
like @proxy but use normal locations configured with the “internal”
directive. And we’ve been keeping doing this in our production apps.

Regards,
-agentzh

Rebuilt with 0.31rc11. Also noticed note about, and implemented,
module loading order.
Everything seems straight as an arrow and no evil going ons detected so
far.

Cheers!

On Thu, Oct 13, 2011 at 12:52 PM, Nginx U. [email protected]
wrote:

This did not work due to exactly the same issue with ngx.exec + named
locations. Please try the latest v0.3.1rc10 release mentioned earlier.
It should also work now :slight_smile:

Regards,
-agentzh

Hi,

Unfortunately, I am still seeing the “0 0” outputs in my logs on (only
some) requests. I had noticed that some pages just seemed to have the
“loading” wheel spin for ever and on looking, I found the instances.

It is not affecting standard php requests anymore from what I can see
but it seems some Ajax type requests where php is being used for
dynamic js (! need to look closer into how I set those up) and with
phpMyAdmin which is aliased somewhere.

Probably has to do with named locations as you mentioned earlier but I
prefer to use these so I have rolled back to my back to my 1.0.6
install so as to get back my ngx_rewrite based setup which works fine
with this.

Will sort out upgrading and further tests later.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs