Lua parser and redis2 questions

Server NGINX
1

Hi Folks -

Looking for some examples or hints in order to leverage lua_parser &&
redis2. To date, I’ve been using httpredis and content_by_lua code to
perform a number of simple 'get’s which has worked well. Now, I’d like
to pipeline a number of redis requests together in order to leverage
‘auth’ within redis and eventually pipeline numerous 'get’s at once.

This is what I have working so far, appreciate any assistance or
commentary…

location /redis2 {
set $key $http_host;
redis2_query auth foobar;
redis2_query get $key;
redis2_pass redis-backend;
}

This auth + get works great and yields the following as expected:

+OK
$12
blahblahblah

Goal 1:
I’d like to be able to strip out the +OK\r\n$12\r\n and be left with the
actual value of blahblahblah. Hints or example?

Goal 2:
Eventually I’d like to be able to pipeline an ‘auth’ plus numerous
‘gets’ together and leverage each value, if a $-1 isn’t returned for a
‘get’, ie:
location /redis2 {
set $key $http_host;
set $key2 $http_host:some_arbitrary_value;
redis2_query auth foobar;
redis2_query get $key;
redis2_query get $key2;
redis2_pass redis-backend;
}

Thanks in advance for any pointers…

Posted at Nginx Forum:

2

I’m new to redis/nginx combinations, however I think you should also add
following modules:
-GitHub - openresty/lua-redis-parser: Lua module for parsing raw redis responses
-https://github.com/chaoslawful/lua-nginx-module.

Then you can use

content_by_lua ’
local replies = ngx.location.capture(“/redis2”)
local parser = require(‘redis.parser’)
– assuming the replies variable holds n redis responses
– to be parsed:
local results = parser.parse_replies(replies, n)
for i, result in ipairs(results) do
local res = result[1]
local typ = result[2]
– res and typ have exactly the same meaning as in
– the parse_reply method documented above
end
’

Posted at Nginx Forum:

3

On 06/04/2011 08:19 AM, logar.damir wrote:

– assuming the replies variable holds n redis responses
Posted at Nginx Forum:
Re: lua parser and redis2 questions

nginx mailing list
[email protected]
nginx Info Page
Here on the blog of Mike Ferrier, you can find a excellent user-case for
the combination of Redis, Lua, Nginx and Rails
http://mikeferrier.ca/2011/05/14/my-beautiful-dark-twisted-reverse-proxy-lru-cache/

4

On 06/04/2011 11:12 AM, Marcos O. wrote:

local parser = require(‘redis.parser’)

–
Marcos Luis Ortiz Valmaseda
Software Engineer (Distributed Systems)
http://uncubanitolinuxero.blogspot.com
Sorry, the right link is this:
http://mikeferrier.ca/2011/05/14/my-beautiful-dark-twisted-reverse-proxy-LRU-cache/

5

On Thu, Jun 2, 2011 at 7:27 AM, gregr401 [email protected] wrote:

actual value of blahblahblah. Hints or example?

As documented in ngx_redis2’s README, this module always returns the
raw TCP responses from the remote redis server. You need to parse the
redis response yourself (for example, by combining ngx_lua +
lua-redis-parser).

   redis2_query get $key2;
   redis2_pass redis-backend;

}

The latest version of ngx_redis2 fully supports redis command
pipelining. Isn’t it working for you?

Regards,
-agentzh

6

Thanks for the responses! Everything on the redis end is fine and I am
expecting the raw responses by leveraging your redis parser, it’s within
the parser examples I’m not making any progress. I’d like to be able to
parse out and set each value to a variable.

Here’s my test locations and log snippet, hopefully I’m just missing
something obvious :slight_smile:

redis auth is off at this point, just trying to get a pipeline parse

working ok
location /redis2 {
set $key $http_host;

            redis2_query get $key;
            redis2_query get foo;
            redis2_pass redis-backend;

}

##hitting this directly in a browser returns:
$11
auth is off
$3
bar

location /mytest {
content_by_lua ’
local replies = ngx.location.capture(“/redis2”)
local parser = require(“redis.parser”)
– assuming the replies variable holds n redis
responses
– to be parsed:
– tried this in case n needed set: local
results = parser.parse_replies(replies, 2), same error
local results = parser.parse_replies(replies,
n)
for i, result in ipairs(results) do
local res = result[1]
local typ = result[2]
ngx.print(res)
ngx.print(typ)
– res and typ have exactly the same
meaning as in
– the parse_reply method documented
above
end
';
}

results in 500 and below error snippet

I see the following in the error log:

[error] 4921#0: *3 lua handler aborted: runtime error: [string
“content_by_lua”]:6: attempt to call field ‘parse_replies’ (a nil value)
while sending to client request: “GET /mytest HTTP/1.1”

On a separate note, I’ve noticed when using Maxim’s wonderful keepalive
upstream module between redis, an established connection that performs a
redis AUTH will actually maintain that ‘authed’ state for any subsequent
redis calls (even those with no specific AUTH call). I’d imagine these
will work till the connections eventually timeout, keep this in mind
when testing when using redis && auth.

Posted at Nginx Forum:

7

Thanks, logar. I mimicked your test config and the below is the
result:

location /testluaparser {
content_by_lua ’
local replies = ngx.location.capture(“/redis2”)
local parser = require(“redis.parser”)
ngx.say(“reply status is:”,replies.status)
ngx.say(“reply body is:”,replies.body)
ngx.say(“end of showresults”)
local res,typ = parser.parse_reply(replies.body)
ngx.say(“type=”, typ)
ngx.say(“res=”, res)
';
}

reply status is:200
reply body is:$11
auth is off
$3
bar

end of showresults
type=4
res=auth is off

I’m still stuck with properly parsing pipelined requests as I’m
attempting to perform an AUTH check in conjunction with a few gets.

Posted at Nginx Forum:

8

Halo!

Sorry for the delay! I’ve been on vocation again :stuck_out_tongue:

I’d like to write some quick notes below regarding your code sample.

On Sun, Jun 5, 2011 at 2:15 AM, gregr401 [email protected] wrote:

       local results = parser.parse_replies(replies,

n)

Here, you assumed that ngx.location.capture() returns response body
directly, which is not the case. The ngx.location.capture() returns a
Lua table which has three slots, header, body, and status. See
ngx_lua’s README for more details.

So here you should write

local res = ngx.location.capture("/redis2")
if res.status ~= ngx.HTTP_OK then
    ngx.exit(res.status)
end

local results = parser.parse_replies(res.body, 2) -- 2 pipelined

requests in /redis2

results in 500 and below error snippet

I see the following in the error log:

[error] 4921#0: *3 lua handler aborted: runtime error: [string
“content_by_lua”]:6: attempt to call field ‘parse_replies’ (a nil value)
while sending to client request: “GET /mytest HTTP/1.1”

This error implies that you’re using a too old version of the
lua-redis-parser library which does not support the parse_replies
function at all. The latest tagged version of lua-redis-parser was
v0.04, which does not include the pipelining feature implemented in
master HEAD. I’m guessing you were using that.

I’ve just tagged master HEAD as the v0.05 release that you can download
below:

https://github.com/agentzh/lua-redis-parser/downloads

On a separate note, I’ve noticed when using Maxim’s wonderful keepalive
upstream module between redis, an established connection that performs a
redis AUTH will actually maintain that ‘authed’ state for any subsequent
redis calls (even those with no specific AUTH call). I’d imagine these
will work till the connections eventually timeout, keep this in mind
when testing when using redis && auth.

Well, you cannot rely on that because you cannot really control how
and when the connection pool opens and closes connections.

For similar reasons, ngx_drizzle implements the charset option all by
itself on the C level by ensuring “set names ‘some_charset’” was sent
to mysql exactly once for a connection. It does not use
ngx_http_upstream_keepalive, rather, its own connection pool
implementation, so it is possible.

Regards,
-agentzh

9

On Mon, Jun 6, 2011 at 6:40 AM, gregr401 [email protected] wrote:

      ngx.say("end of showresults")
      local res,typ = parser.parse_reply(replies.body)

You should have used the parse_replies function instead. The
parse_reply function always tries to parse only one single redis
command reply.

end of showresults
type=4
res=auth is off

Your output indeed indicates that only the first redis reply is parsed
:wink:

Try lua-redis-parser v0.05 instead, as suggested in my another mail in
this thread.

Good luck!
-agentzh

10

Agentzh, how about when using select
(SELECT | Redis) to select specific database #.
Does pipelining work ok in this case?

My command at the moment is:
redis2_query select 2;
redis2_query evalsha $sha1keyvickrey 1 $val;
redis2_query select 0;
redis2_pass redisbackend;

Are this 3 redis2_query commands running on the same connection? If not,
could you maybe add another variable for database number. Simething
similar to HttpRedis that has $redis_db (The number of redis database)
which I think is broken (Nginx redis, select database not working?).

Posted at Nginx Forum:

11

Gregr, I’m using multibulk reply without pipelining. If it helps,
here’s a working sample.

I haven’t used pipelining yet, since support for this a quite new
(2011-04-03 for lua-redis-parser). Maybe try to update agentzh modules
first.

#First save some test values to redis:
location /testluasetnginx
{
redis2_query hset testkey “testhash1” “value1”;
redis2_query hset testkey “testhash2” “value2”;
redis2_pass redisbackend;
}

#internal location for nonblocking fetch
location /testluagetnginx
{
redis2_query hgetall testkey;
redis2_pass redisbackend;
}

#test code
location /testluaparser
{
content_by_lua ’
local replies = ngx.location.capture(“/testluagetnginx”)
local parser = require(“redis.parser”)
ngx.say(“reply status is:”,replies.status)
ngx.say(“reply body is:”,replies.body)
ngx.say(“end of showresults”)
local res,typ = parser.parse_reply(replies.body)
ngx.say(“type=”, typ)
ngx.say(“res=”, res[1])
ngx.say(“res=”, res[2])
ngx.say(“res=”, res[3])
ngx.say(“res=”, res[4])
';
}

#my results when using curl ‘http://localhost/testluaparser’
reply status is:200
reply body is:*4
$9
testhash1
$6
value1
$9
testhash2
$6
value2

end of showresults
type=5
res=testhash1
res=value1
res=testhash2
res=value2

Posted at Nginx Forum:

12

On Mon, Jun 6, 2011 at 7:16 PM, logar.damir [email protected]
wrote:

Agentzh, how about when using select
(SELECT | Redis) to select specific database #.
Does pipelining work ok in this case?

My command at the moment is:
redis2_query select 2;
redis2_query evalsha $sha1keyvickrey 1 $val;
redis2_query select 0;
redis2_pass redisbackend;

Your sample above should work. If not, please consider filing a bug
ticket on GitHub :wink:

Are this 3 redis2_query commands running on the same connection?

Yes. What I meant in my last email is that there is no such guarantee
across multiple nginx (sub-)requests. Your pipelined redis commands
specified in the same nginx location are always sent on one single
redis TCP connection. So you shouldn’t worry about that in this
context.

If not,
could you maybe add another variable for database number. Simething
similar to HttpRedis that has $redis_db (The number of redis database)
which I think is broken (Nginx redis, select database not working?).

No, I do not want ngx_redis2 to mud with specific redis command
details because the consistency and simplicity of its uniformed wired
protocol is one of the beauties and powers of redis. Leveraging the
redis pipelining feature of ngx_redis2 is sufficient.

Regards,
-agentzh

13

On Tue, Jun 7, 2011 at 12:50 AM, gregr401 [email protected] wrote:

Well, all this seems to be tracked back to an env issue on my side,
argh. I ended up setting a lua_package_cpath and now parsing replies is
working exactly as they should (interesting that some other features
were working, tho, hmmm).

Yeah, using the lua_package_cpath directive is considered one of the
best practices :wink:

Thanks again for the above help and validation agentzh and logar!

nginx + redis pipelining + lua FTW =)

Happy hacking!
-agentzh

14

Well, all this seems to be tracked back to an env issue on my side,
argh. I ended up setting a lua_package_cpath and now parsing replies is
working exactly as they should (interesting that some other features
were working, tho, hmmm).

Thanks again for the above help and validation agentzh and logar!

nginx + redis pipelining + lua FTW =)

Posted at Nginx Forum: