Catch ALL requests by LUA-script

Hi,

i’m using the nginx-lua-module by agentzh on the current stable nginx
(0.8.54) (also the drizzle and echo module and php via php-fpm)

it’s working fine with a plain nginx running

What i would like to to:
Is it possible to create a location-entry in the nginx.conf that ALL
requests coming to the Server are handled by a LUA-script?

i.e. i have set the *.php and / locations to internal so only the lua
script should have access to the files (it’s responsible for the in- and
output)

i created another “location * { }” … which contains
“rewrite_by_lua_file /path/to/the/script” and for testing purposes
“default_type ‘text/html’”

But now, every request to the Server ends with a 404.

Background: I would like to have a LUA-script which handles every
request and filters certain Headers, Parameters, etc.

Posted at Nginx Forum:

On Tue, Mar 1, 2011 at 3:34 PM, andiL [email protected] wrote:

What i would like to to:
Is it possible to create a location-entry in the nginx.conf that ALL
requests coming to the Server are handled by a LUA-script?

I’ve just allowed use of rewrite_by_lua_file (and its friends) at the
server and http block levels in ngx_lua’s git master HEAD (
http://github.com/chaoslawful/lua-nginx-module ).

Here’s a small example from the ngx_lua test suite:

server {
    listen          1984;
    server_name     'localhost';

    location / {
        root html;
        index index.html index.htm;
    }

    access_by_lua 'ngx.header["X-Foo"] = "bar" ngx.send_headers()';
}

Then

curl -i localhost:1984/

gives

HTTP/1.1 200 OK
Server: nginx/0.8.54 (without pool)
Date: Tue, 01 Mar 2011 08:34:45 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
X-Foo: bar

<html><head><title>It works!</title></head><body>It 

works!

This is just for demonstration purposes, blindly sending headers
directly from within an access or rewrite handler is usually
considered bad practice :wink:

i.e. i have set the *.php and / locations to internal so only the lua
script should have access to the files (it’s responsible for the in- and
output)

Well, you do not need an ad-hoc location for this. Use directive
inheritance instead.

Background: I would like to have a LUA-script which handles every
request and filters certain Headers, Parameters, etc.

Hopefully my solution given above works for you :slight_smile:

Cheers,
-agentzh

Thanks for your answer.

I just figured out, that “location ~ { …}” will catch all requests,
too … but i don’t know if it’s realy that reliable, so the new feature
of the nginx-lua-plugin seems to be the perfect solution for my problem
(thanks again :slight_smile: )

Being able to have direct access by lua in the server/http blocks is
great for global actions on the requests.

I just have one more question:

What would be the best phase-handler to check/manipulate http-requests
and http-responses?

LUA can work with rewrite (URI transformation on location level), access
(access restrictions check), content (generating content for output) and

  • as i understand - these are processed by nginx in this given order.

Theoretically the best would be to get in directly in the begining with
rewrite_by_lua(_file) to be able to check/manipulate request-headers,
request-body, etc.?

But nginx will still output it’s “default content” after that…
Is there any chance LUA can have access to all of the HTTP-Response, too
(especially the headers - the content is not that important at the
moment)?

I know that it can be done via ngx.headers[“HEADER”] but i don’t think
this will be possible when still working in the rewrite/access phase?

Hope i don’t understand something very wrong there :slight_smile:

Thanks, Andi

Posted at Nginx Forum:

Thanks for getting this accross to me.

Still digging through the sources of nginx and the plugins :slight_smile:

Trying to implement something “good” for the request-handling for now.
Perhaps i can contribute somthing to the plugin besides being a heavy
tester in some weeks :smiley:

Thanks, Andi

Posted at Nginx Forum:

On Tue, Mar 1, 2011 at 5:53 PM, andiL [email protected] wrote:

Thanks for your answer.

I just figured out, that “location ~ { …}” will catch all requests,
too

Once a location gets matched, it’ll be difficult to persuade nginx to
match against other locations because a hit is already made.

I just have one more question:

What would be the best phase-handler to check/manipulate http-requests
and http-responses?

LUA can work with rewrite (URI transformation on location level), access
(access restrictions check), content (generating content for output) and

  • as i understand - these are processed by nginx in this given order.

Another difference between rewrite phase handlers and access phase
handlers is that access phase is skipped completely for subrequests
while rewrite phase handlers will be run even in a subrequest.

Theoretically the best would be to get in directly in the begining with
rewrite_by_lua(_file) to be able to check/manipulate request-headers,
request-body, etc.?

access_by_lua(_file) is fine too :slight_smile:

But nginx will still output it’s “default content” after that…

Sure.

Is there any chance LUA can have access to all of the HTTP-Response, too
(especially the headers - the content is not that important at the
moment)?

This is exactly what output filters are for :wink:

For now, ngx_lua does not provide output filter hooks like
output_header_filter_by_lua(_file) and
output_body_filter_by_lua(_file). But these are on our TODO list. And
as always: patches welcome, and volunteers welcome!

I know that it can be done via ngx.headers[“HEADER”] but i don’t think
this will be possible when still working in the rewrite/access phase?

No, the rewrite and access phases are just too early :slight_smile:

Hope i don’t understand something very wrong there :slight_smile:

Almost right :slight_smile:

Cheers,
-agentzh

Maybe slightly off-topic but then maybe it is useful for you (andiL) or
someone else.

Before i start, @agentzh: every ?nth request, one of nginx
(0.8.54.4rc2-jit2beta6) worker processes segfaults, is this behaviour
related to the pcre issue? have not patched it yet… ah and thank you
very much for giving nginx this all new powers!! ((:

Ok, im a long time nginx/redis/ruby user and this are my first steps
with lua and a lightweight (M)VC framework around three core “classes”,
which uses haml for templates/layout and markdown for text processing (5
minute ffi discount binding), drizzle and redis for data and redis also
for caching. Successfully running a little app with it. yay! . For now
i can give you some snippets just to show some basic aspects, ill
release a fully working example sometime soon, if the segfaults doesnt
belong to the pcre-issue ill try to debug that too.

Below ill use luasql, instead you can use
ngx.location.capture(“/some-internal-rds-json”) and lua-yajl, but this
approach ends as soon as you want INSERT followed by LAST_INSERT_ID() or
other complex queries/transactions (as long as libdrizzle has no support
for multiple queries, maybe this is not the best option to go with, ive
tried alternatives and workarounds, but for now this is what I want).

At the moment i do not need an ORM for my data, but have already done
some functions for both rds_json and luasql. redis has some commands
exposed in nginx.conf for easy access with lua (: The core classes:

Request; which contains several ngx.var wrappers (for nil instead of ‘’
for empty keys), present(str), simple cookie/session/login, before/after
filters, haml/markdown rendering, etc…

MethodRequest < Request; which calls functions for HTTP Methods aka GET,
POST alike…
and finally Resource < MethodRequest; which has an
Rails-ActionController-like interface.

Greetz

HTML

— ATTACHMENT —

ookeh to bad no code formatting…?

http://exc.eu/paste/deadbeaf.txt

Posted at Nginx Forum:

On Mon, May 2, 2011 at 12:11 PM, HowToMeetLadies [email protected]
wrote:

Maybe slightly off-topic but then maybe it is useful for you (andiL) or
someone else.

Your feedback is always welcome! Sorry for the delay on my side for I
was enjoying my vocation in the Fuzhou city in the southern region of
China in the last few days :slight_smile:

Before i start, @agentzh: every ?nth request, one of nginx
(0.8.54.4rc2-jit2beta6) worker processes segfaults, is this behaviour
related to the pcre issue? have not patched it yet… ah and thank you
very much for giving nginx this all new powers!! ((:

What’s your operating system? We used to observe similar periodic
segfaults with exactly the same version of ngx_openresty on RedHat
Enterprise Linux (RHEL) 5.4 in production but now all is fine after we
switched LuaJIT to a more recent git HEAD version which includes an
important work-around for an old libgcc bug explained here in the
lua-l mailing list:

http://lua-users.org/lists/lua-l/2011-03/msg00576.html

I’ll make a new release of ngx_openresty to include a new version of
LuaJIT as well as some other modules’ updates like ngx_lua’s and
ngx_srcache’s. I’ll inform you here when I’m done.

Also, it’ll be very appreciated if you can obtain a gdb backtrace for
your segfaults so that we can look at the details :wink:

Ok, im a long time nginx/redis/ruby user and this are my first steps
with lua and a lightweight (M)VC framework around three core “classes”,
which uses haml for templates/layout and markdown for text processing (5
minute ffi discount binding), drizzle and redis for data and redis also
for caching. Successfully running a little app with it. yay! . For now
i can give you some snippets just to show some basic aspects, ill
release a fully working example sometime soon, if the segfaults doesnt
belong to the pcre-issue ill try to debug that too.

Oh, that’s awesome :slight_smile:

Below ill use luasql, instead you can use
ngx.location.capture(“/some-internal-rds-json”) and lua-yajl, but this
approach ends as soon as you want INSERT followed by LAST_INSERT_ID() or
other complex queries/transactions (as long as libdrizzle has no support
for multiple queries, maybe this is not the best option to go with, ive
tried alternatives and workarounds, but for now this is what I want).

The luasql library will certainly block nginx worker processes. Please
dont’ use that.

I know libdrizzle is very limited in terms of functionality and it’s
not maintained very well. We’ll start a completely new nginx
upstream modules for mysql named ngx_mysql by implementing a
non-blocking full-fledged highly-optimized mysql client from scratch.
I’d love to see it happen this year :wink:

Another direction we’re taking is to introduce the “cosocket”
mechanism into ngx_lua such that one can use a synchronous Lua
socket API that is transparently asynchronous and non-blocking on
the nginx C level. Atop that, we can build pure-Lua non-blocking mysql
client driver and thus putting everything onto the Lua land and no
longer need to mess up the nginx.conf file and nginx subrequests. I’d
also want to see it happen by this Christmas :wink:

and finally Resource < MethodRequest; which has an
Rails-ActionController-like interface.

The high-level abstraction for ngx_lua is excellent! We’re taking a
slightly different approach along the line in our web applications:

http://agentzh.org/misc/nginx/lzsql-manual.html

We’re using a DSL (Domain Specific Language) compiler that emits Lua
code and business logic is mostly implemented in this DSL. It’s
targeting our business though, not meant to be useful for general web
application development (yet).

Thanks for sharing!

-agentzh

Your feedback is always welcome! Sorry for the
delay on my side for I
was enjoying my vocation in the Fuzhou city in the
southern region of
China in the last few days :slight_smile:

Heh, I know what delays are and that was no delay… Here it is best
weather for bicycling, so i do all the time! :wink: One thing on my todo is
to visit china, my cousin lived there past 5-6 years and i’ve seen many
pictures and videos… Last month he married his wife there, but i could
not get away :frowning:

What’s your operating system? We used to observe
similar periodic
segfaults with exactly the same version of
ngx_openresty on RedHat
Enterprise Linux (RHEL) 5.4 in production but now

My good old development box is a quad i686 with deb5 (old)stable gcc
4.3.2-1.1

Tried both samples and luajit detected it. Anyway thank you for sharing
this, so ill try that next.

I’ll make a new release of ngx_openresty to
include a new version of
LuaJIT as well as some other modules’ updates like
ngx_lua’s and
ngx_srcache’s. I’ll inform you here when I’m done.

Also, it’ll be very appreciated if you can obtain
a gdb backtrace for
your segfaults so that we can look at the details
:wink:

Sure, thats no problem. Tomorrow ill attach to it and take a sample for
you. Have also a full dump, as every build has its own /prefix.

Oh, that’s awesome :slight_smile:

Thanks, but yours is epic.

tried alternatives and workarounds, but for now
this is what I want).

The luasql library will certainly block nginx
worker processes. Please
dont’ use that.

Tbh, youre absolutely right. Realized it before i used this and I had
no problems, also do not have much inserts (about 25-40k text each,
stored in a fraction of a second). Tested it also further with luasql
all queries, no problems so far but only with circumventions and all
under pre-calculated conditions with enough instances, fast i/o,
caching etc. This is at no will a reliable solution and much likely a
show-stopper! So do not try at home eh?!!

In the meantime or maybe forever ill stick to redis only (: Exchanges to
or from intermediates are stressing me, but still better as seeing ruby
eating up resources and doing “nothing” on ngx_passenger/ree… sinatra,
merb, rails, foo, bla -.-" Redis helped there a lot, but even in my
experimental Rack written with D ive done better! :wink: But thats true
another story…

As a last resort for mysql i thought of getting an unique id from a
(lua) location which is then used for insertion and afterwards on
success, i.e. for redirects. Or have i missed something, what would you
do?

Another direction we’re taking is to introduce the
nginx subrequests. I’d
also want to see it happen by this Christmas :wink:

You mentioned “cosocket” somewhere (github?) already and IMO that would
be really amazing! I’m following “agentzh” and “chaoslawful” on github
and will take a deeper look when i get more time. Will be linking to
your projects when i release something useful (safe) for it.

compiler that emits Lua
code and business logic is mostly implemented in
this DSL. It’s
targeting our business though, not meant to be
useful for general web
application development (yet).

Looks very promising, i seen that page already on my researches and
would love to get my hands on it. I do a lot of research, trying as much
as possible, protocols like XMPP and PSYC and such good things, but
mostly everything apart from PHP, .Net and Java… :>

Thanks for sharing!

-agentzh

Heh. >.< All the thanks belong to you and who else is also behind that
projects! Had so much fun with it so far, you wont believe it how it
boosted everything at our groups site, ookeh not that much, but enough
to already left ruby (was riding since little rails). And personally it
helped me alot to grok lua, as i needed that for deeper reworks of my
prosody-im fork (another lua project thats awesome, i prefer it over
ejabberd/erl which indeed is both neat feature-wise, but wont let me
change (easily) from mnesia to redis or other nice tricks).

Thanks again!

HTML

Posted at Nginx Forum:

Looking forward to that :slight_smile:
Will do both as soon i’ll arrive at home later this day!

help a lot due to the

Thank you for this, my conf applied already to this, but maybe you
should tell about that in your docs too. (:

means of pure Lua
location /test {
echo_location /mysql “select * from foo;”;

{"errcode":0}
{"errcode":0}
{"errcode":0,"insert_id":1,"affected_rows":1}
[{"id":1,"val":3.1415926}]

Have you noticed the “insert_id” field in the 3rd
JSON response? :wink:

Unbelievable! :smiley: Ive seen “insert id” header in ngx_drizzle, but hadnt
looked further at inserts with rds_json (damn stupid me, i should write
an article about that).

LOL

I have a php-fpm install for exactly 3 apps which are our
webmail/cms/board, because php-users are not aware of their risks ive
disabled it apart from that. NCARS rulez, but im replacing it already →
xmpp (psyc) + web frontend.

I used techniques like MEF and such in .Net, but imho its desktop-only…
One can do amazing things with it, also without using client-side
runtime, but if you’re not bound to that platform, why would you do?

Its good to have varnish, ha and/or simple nginx in front of such
servers/frameworks, but now nginx+lua is enough, its so tiny and fast…

Happy hacking!

Greetz (;

HTML

Posted at Nginx Forum:

On Thu, May 5, 2011 at 8:05 AM, HowToMeetLadies [email protected]
wrote:

Heh, I know what delays are and that was no delay… Here it is best
weather for bicycling, so i do all the time! :wink: One thing on my todo is
to visit china, my cousin lived there past 5-6 years and i’ve seen many
pictures and videos… Last month he married his wife there, but i could
not get away :frowning:

Oh, I’m so happy to hear that :wink: Please bring my best wishes to your
cousin!

Sure, thats no problem. Tomorrow ill attach to it and take a sample for
you. Have also a full dump, as every build has its own /prefix.

Looking forward to that :slight_smile:

Tbh, youre absolutely right. Realized it before i used this and I had
no problems, also do not have much inserts (about 25-40k text each,
stored in a fraction of a second). Tested it also further with luasql
all queries, no problems so far but only with circumventions and all
under pre-calculated conditions with enough instances, fast i/o,
caching etc. This is at no will a reliable solution and much likely a
show-stopper! So do not try at home eh?!!

Well, it will only cause problems when you have too few nginx workers
and nontrivial number of concurrent requests and your database is
going mad and becoming slow :wink: Anyway, it’s good for personal use, not
for production apps :wink:

In the meantime or maybe forever ill stick to redis only (: Exchanges to
or from intermediates are stressing me, but still better as seeing ruby
eating up resources and doing “nothing” on ngx_passenger/ree… sinatra,
merb, rails, foo, bla -.-" Redis helped there a lot, but even in my
experimental Rack written with D ive done better! :wink: But thats true
another story…

When you’re using ngx_redis2 + lua-redis-parser, please ensure you’re
using a tcp connection pool (provided by ngx_http_upstream_keepalive)
and redis pipelining wherever possible. These features will
significantly improve performance. Also, using multiple instance of
redis servers on your multi-core machines also help a lot due to the
sequential processing nature of a single redis server instance.

Also, when you’re benchmarking performance, please ensure that your
error log level is high enough to prevent nginx workers spend too much
cycles on flushing the error.log file, which is certainly very
expensive :slight_smile:

As a last resort for mysql i thought of getting an unique id from a
(lua) location which is then used for insertion and afterwards on
success, i.e. for redirects. Or have i missed something, what would you
do?

You can certainly generate a globally unique id by means of pure Lua
or by accessing redis, though that’s a bit silly :wink:

If all you want is to get LAST_INSERT_ID, then ngx_drizzle already
returns that automatically for you when you’re doing a SQL insert
query. Consider the following sample nginx.conf snippet:

location /test {
    echo_location /mysql "drop table if exists foo";
    echo;
    echo_location /mysql "create table foo (id serial not null,

primary key (id), val real);";
echo;
echo_location /mysql “insert into foo (val) values
(3.1415926);”;
echo;
echo_location /mysql “select * from foo;”;
echo;
}
location /mysql {
drizzle_pass backend;
drizzle_module_header off;
drizzle_query $query_string;
rds_json on;
}

Then GET /test gives the following outputs:

{"errcode":0}
{"errcode":0}
{"errcode":0,"insert_id":1,"affected_rows":1}
[{"id":1,"val":3.1415926}]

Have you noticed the “insert_id” field in the 3rd JSON response? :wink:

Full transaction support will land into ngx_drizzle (maybe
ngx_postgres as well). It will be implemented by locking and tagging
database connections in the pool. But there’s no ETA yet for that.

You mentioned “cosocket” somewhere (github?) already and IMO that would
be really amazing! I’m following “agentzh” and “chaoslawful” on github
and will take a deeper look when i get more time. Will be linking to
your projects when i release something useful (safe) for it.

Oh, I’m feeling honored :slight_smile:

Looks very promising, i seen that page already on my researches and
would love to get my hands on it. I do a lot of research, trying as much
as possible, protocols like XMPP and PSYC and such good things, but
mostly everything apart from PHP, .Net and Java… :>

LOL

Heh. >.< All the thanks belong to you and who else is also behind that
projects! Had so much fun with it so far, you wont believe it how it
boosted everything at our groups site, ookeh not that much, but enough
to already left ruby (was riding since little rails). And personally it
helped me alot to grok lua, as i needed that for deeper reworks of my
prosody-im fork (another lua project thats awesome, i prefer it over
ejabberd/erl which indeed is both neat feature-wise, but wont let me
change (easily) from mnesia to redis or other nice tricks).

ngx_openresty still has many rough corners and has a long way to go.
We’ll surely try to make it better and better unfailingly :slight_smile:

Happy hacking!

-agentzh

On Fri, May 6, 2011 at 12:29 AM, HowToMeetLadies [email protected]
wrote:

Thank you for this, my conf applied already to this, but maybe you
should tell about that in your docs too. (:

Just added to ngx_redis2’s README :wink:

Unbelievable! :smiley: Ive seen “insert id” header in ngx_drizzle, but hadnt
looked further at inserts with rds_json (damn stupid me, i should write
an article about that).

Just added this to ngx_drizzle’s README too :slight_smile: My bad, it’s indeed not
that obvious for new users :wink:

Thanks!
-agentzh