# Lua parser and redis2 questions

**URL:** <https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759>\
**Category:** NGINX\
**Created:** [June 2, 2011, 1:28am UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759 "2011-06-02T01:28:30Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [June 2, 2011, 1:28am UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/1 "2011-06-02T01:28:30Z")

</div>

Hi Folks -

Looking for some examples or hints in order to leverage lua\_parser &&  
redis2. To date, I’ve been using httpredis and content\_by\_lua code to  
perform a number of simple 'get’s which has worked well. Now, I’d like  
to pipeline a number of redis requests together in order to leverage  
‘auth’ within redis and eventually pipeline numerous 'get’s at once.

This is what I have working so far, appreciate any assistance or  
commentary…

location /redis2 {  
set $key $http\_host;  
redis2\_query auth foobar;  
redis2\_query get $key;  
redis2\_pass redis-backend;  
}

This auth + get works great and yields the following as expected:

+OK  
$12  
blahblahblah

Goal 1:  
I’d like to be able to strip out the +OK\r\n$12\r\n and be left with the  
actual value of blahblahblah. Hints or example?

Goal 2:  
Eventually I’d like to be able to pipeline an ‘auth’ plus numerous  
‘gets’ together and leverage each value, if a $-1 isn’t returned for a  
‘get’, ie:  
location /redis2 {  
set $key $http\_host;  
set $key2 $http\_host:some\_arbitrary\_value;  
redis2\_query auth foobar;  
redis2\_query get $key;  
redis2\_query get $key2;  
redis2\_pass redis-backend;  
}

Thanks in advance for any pointers…

Posted at Nginx Forum:

> **[lua parser and redis2 questions](https://forum.nginx.org/read.php?2,203154,203154#msg-203154)**
>
> Hi Folks - Looking for some examples or hints in order to leverage lua\_parser && redis2. To date, I've been using httpredis and content\_by\_lua code to perform a number of simple 'get's which has worked well. Now, I'd like to pipeline a number...

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 4, 2011, 2:50pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/2 "2011-06-04T14:50:24Z")

</div>

I’m new to redis/nginx combinations, however I think you should also add  
following modules:  
-[GitHub - openresty/lua-redis-parser: Lua module for parsing raw redis responses](https://github.com/agentzh/lua-redis-parser)  
-[https://github.com/chaoslawful/lua-nginx-module](https://github.com/chaoslawful/lua-nginx-module).

Then you can use

content\_by\_lua ’  
local replies = ngx.location.capture(“/redis2”)  
local parser = require(‘redis.parser’)  
– assuming the replies variable holds n redis responses  
– to be parsed:  
local results = parser.parse\_replies(replies, n)  
for i, result in ipairs(results) do  
local res = result[1]  
local typ = result[2]  
– res and typ have exactly the same meaning as in  
– the parse\_reply method documented above  
end  
’

Posted at Nginx Forum:

> **[Re: lua parser and redis2 questions](https://forum.nginx.org/read.php?2,203154,203862#msg-203862)**
>
> Hi Folks - Looking for some examples or hints in order to leverage lua\_parser && redis2. To date, I've been using httpredis and content\_by\_lua code to perform a number of simple 'get's which has worked well. Now, I'd like to pipeline a number...

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 4, 2011, 5:11pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/3 "2011-06-04T17:11:09Z")

</div>

On 06/04/2011 08:19 AM, logar.damir wrote:

> – assuming the replies variable holds n redis responses  
> Posted at Nginx Forum:  
> [http://forum.nginx.org/read.php?2,203154,203862#msg-203862](http://forum.nginx.org/read.php?2,203154,203862#msg-203862)
> 
> * * *
> 
> nginx mailing list  
> [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
> [mailman.nginx.org archive](http://nginx.org/mailman/listinfo/nginx)  
> Here on the blog of Mike Ferrier, you can find a excellent user-case for  
> the combination of Redis, Lua, Nginx and Rails  
> [http://mikeferrier.ca/2011/05/14/my-beautiful-dark-twisted-reverse-proxy-lru-cache/](http://mikeferrier.ca/2011/05/14/my-beautiful-dark-twisted-reverse-proxy-lru-cache/)

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 4, 2011, 5:16pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/4 "2011-06-04T17:16:05Z")

</div>

On 06/04/2011 11:12 AM, Marcos O. wrote:

> > local parser = require(‘redis.parser’)
> 
> –  
> Marcos Luis Ortiz Valmaseda  
> Software Engineer (Distributed Systems)  
> [http://uncubanitolinuxero.blogspot.com](http://uncubanitolinuxero.blogspot.com)  
> Sorry, the right link is this:  
> [http://mikeferrier.ca/2011/05/14/my-beautiful-dark-twisted-reverse-proxy-LRU-cache/](http://mikeferrier.ca/2011/05/14/my-beautiful-dark-twisted-reverse-proxy-LRU-cache/)

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 4, 2011, 5:21pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/5 "2011-06-04T17:21:39Z")

</div>

On Thu, Jun 2, 2011 at 7:27 AM, gregr401 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> actual value of blahblahblah. Hints or example?

As documented in ngx\_redis2’s README, this module always returns the  
raw TCP responses from the remote redis server. You need to parse the  
redis response yourself (for example, by combining ngx\_lua +  
lua-redis-parser).

> ```
> redis2_query get $key2;
> redis2_pass redis-backend;
> 
> ```
> 
> }

The latest version of ngx\_redis2 fully supports redis command  
pipelining. Isn’t it working for you?

Regards,  
-agentzh

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 4, 2011, 8:15pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/6 "2011-06-04T20:15:46Z")

</div>

Thanks for the responses! Everything on the redis end is fine and I am  
expecting the raw responses by leveraging your redis parser, it’s within  
the parser examples I’m not making any progress. I’d like to be able to  
parse out and set each value to a variable.

Here’s my test locations and log snippet, hopefully I’m just missing  
something obvious 🙂

## redis auth is off at this point, just trying to get a pipeline parse

working ok  
location /redis2 {  
set $key $http\_host;

```
            redis2_query get $key;
            redis2_query get foo;
            redis2_pass redis-backend;

```

}

##hitting this directly in a browser returns:  
$11  
auth is off  
$3  
bar

location /mytest {  
content\_by\_lua ’  
local replies = ngx.location.capture(“/redis2”)  
local parser = require(“redis.parser”)  
– assuming the replies variable holds n redis  
responses  
– to be parsed:  
– tried this in case n needed set: local  
results = parser.parse\_replies(replies, 2), same error  
local results = parser.parse\_replies(replies,  
n)  
for i, result in ipairs(results) do  
local res = result[1]  
local typ = result[2]  
ngx.print(res)  
ngx.print(typ)  
– res and typ have exactly the same  
meaning as in  
– the parse\_reply method documented  
above  
end  
';  
}

## results in 500 and below error snippet

I see the following in the error log:

[error] 4921#0: \*3 lua handler aborted: runtime error: [string  
“content\_by\_lua”]:6: attempt to call field ‘parse\_replies’ (a nil value)  
while sending to client request: “GET /mytest HTTP/1.1”

On a separate note, I’ve noticed when using Maxim’s wonderful keepalive  
upstream module between redis, an established connection that performs a  
redis AUTH will actually maintain that ‘authed’ state for any subsequent  
redis calls (even those with no specific AUTH call). I’d imagine these  
will work till the connections eventually timeout, keep this in mind  
when testing when using redis && auth.

Posted at Nginx Forum:  
[http://forum.nginx.org/read.php?2,203154,203926#msg-203926](http://forum.nginx.org/read.php?2,203154,203926#msg-203926)

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 6, 2011, 12:41am UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/7 "2011-06-06T00:41:12Z")

</div>

Thanks, logar. I mimicked your test config and the below is the  
result:

location /testluaparser {  
content\_by\_lua ’  
local replies = ngx.location.capture(“/redis2”)  
local parser = require(“redis.parser”)  
ngx.say(“reply status is:”,replies.status)  
ngx.say(“reply body is:”,replies.body)  
ngx.say(“end of showresults”)  
local res,typ = parser.parse\_reply(replies.body)  
ngx.say(“type=”, typ)  
ngx.say(“res=”, res)  
';  
}

reply status is:200  
reply body is:$11  
auth is off  
$3  
bar

end of showresults  
type=4  
res=auth is off

I’m still stuck with properly parsing pipelined requests as I’m  
attempting to perform an AUTH check in conjunction with a few gets.

Posted at Nginx Forum:

> **[Re: lua parser and redis2 questions](https://forum.nginx.org/read.php?2,203154,204202#msg-204202)**
>
> Hi Folks - Looking for some examples or hints in order to leverage lua\_parser && redis2. To date, I've been using httpredis and content\_by\_lua code to perform a number of simple 'get's which has worked well. Now, I'd like to pipeline a number...

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 6, 2011, 10:05am UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/8 "2011-06-06T10:05:23Z")

</div>

Halo!

Sorry for the delay! I’ve been on vocation again 😛

I’d like to write some quick notes below regarding your code sample.

On Sun, Jun 5, 2011 at 2:15 AM, gregr401 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> ```
> local results = parser.parse_replies(replies,
> 
> ```
> 
> n)

Here, you assumed that ngx.location.capture() returns response body  
directly, which is not the case. The ngx.location.capture() returns a  
Lua table which has three slots, header, body, and status. See  
ngx\_lua’s README for more details.

So here you should write

```
local res = ngx.location.capture("/redis2")
if res.status ~= ngx.HTTP_OK then
    ngx.exit(res.status)
end

local results = parser.parse_replies(res.body, 2) -- 2 pipelined

```

requests in /redis2

> ## results in 500 and below error snippet
> 
> I see the following in the error log:
> 
> [error] 4921#0: \*3 lua handler aborted: runtime error: [string  
> “content\_by\_lua”]:6: attempt to call field ‘parse\_replies’ (a nil value)  
> while sending to client request: “GET /mytest HTTP/1.1”

This error implies that you’re using a too old version of the  
lua-redis-parser library which does not support the parse\_replies  
function at all. The latest tagged version of lua-redis-parser was  
v0.04, which does not include the pipelining feature implemented in  
master HEAD. I’m guessing you were using that.

I’ve just tagged master HEAD as the v0.05 release that you can download  
below:

```
https://github.com/agentzh/lua-redis-parser/downloads

```

> On a separate note, I’ve noticed when using Maxim’s wonderful keepalive  
> upstream module between redis, an established connection that performs a  
> redis AUTH will actually maintain that ‘authed’ state for any subsequent  
> redis calls (even those with no specific AUTH call). I’d imagine these  
> will work till the connections eventually timeout, keep this in mind  
> when testing when using redis && auth.

Well, you cannot rely on that because you cannot really control how  
and when the connection pool opens and closes connections.

For similar reasons, ngx\_drizzle implements the charset option all by  
itself on the C level by ensuring “set names ‘some\_charset’” was sent  
to mysql exactly once for a connection. It does not use  
ngx\_http\_upstream\_keepalive, rather, its own connection pool  
implementation, so it is possible.

Regards,  
-agentzh

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 6, 2011, 10:10am UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/9 "2011-06-06T10:10:57Z")

</div>

On Mon, Jun 6, 2011 at 6:40 AM, gregr401 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> ```
> ngx.say("end of showresults")
> local res,typ = parser.parse_reply(replies.body)
> 
> ```

You should have used the parse\_replies function instead. The  
parse\_reply function always tries to parse only one single redis  
command reply.

> end of showresults  
> type=4  
> res=auth is off

Your output indeed indicates that only the first redis reply is parsed  
😉

Try lua-redis-parser v0.05 instead, as suggested in my another mail in  
this thread.

Good luck!  
-agentzh

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 6, 2011, 1:16pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/10 "2011-06-06T13:16:29Z")

</div>

Agentzh, how about when using select  
([SELECT | Redis](http://www.redis.io/commands/select)) to select specific database #.  
Does pipelining work ok in this case?

My command at the moment is:  
redis2\_query select 2;  
redis2\_query evalsha $sha1keyvickrey 1 $val;  
redis2\_query select 0;  
redis2\_pass redisbackend;

Are this 3 redis2\_query commands running on the same connection? If not,  
could you maybe add another variable for database number. Simething  
similar to HttpRedis that has $redis\_db (The number of redis database)  
which I think is broken ([Nginx redis, select database not working?](http://forum.nginx.org/read.php?2,203861)).

Posted at Nginx Forum:

> **[Re: lua parser and redis2 questions](https://forum.nginx.org/read.php?2,203154,204264#msg-204264)**
>
> Hi Folks - Looking for some examples or hints in order to leverage lua\_parser && redis2. To date, I've been using httpredis and content\_by\_lua code to perform a number of simple 'get's which has worked well. Now, I'd like to pipeline a number...

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 4, 2011, 10:26pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/11 "2011-06-04T22:26:38Z")

</div>

Gregr, I’m using multibulk reply without pipelining. If it helps,  
here’s a working sample.

I haven’t used pipelining yet, since support for this a quite new  
(2011-04-03 for lua-redis-parser). Maybe try to update agentzh modules  
first.

#First save some test values to redis:  
location /testluasetnginx  
{  
redis2\_query hset testkey “testhash1” “value1”;  
redis2\_query hset testkey “testhash2” “value2”;  
redis2\_pass redisbackend;  
}

#internal location for nonblocking fetch  
location /testluagetnginx  
{  
redis2\_query hgetall testkey;  
redis2\_pass redisbackend;  
}

#test code  
location /testluaparser  
{  
content\_by\_lua ’  
local replies = ngx.location.capture(“/testluagetnginx”)  
local parser = require(“redis.parser”)  
ngx.say(“reply status is:”,replies.status)  
ngx.say(“reply body is:”,replies.body)  
ngx.say(“end of showresults”)  
local res,typ = parser.parse\_reply(replies.body)  
ngx.say(“type=”, typ)  
ngx.say(“res=”, res[1])  
ngx.say(“res=”, res[2])  
ngx.say(“res=”, res[3])  
ngx.say(“res=”, res[4])  
';  
}

#my results when using curl ‘[http://localhost/testluaparser](http://localhost/testluaparser)’  
reply status is:200  
reply body is:\*4  
$9  
testhash1  
$6  
value1  
$9  
testhash2  
$6  
value2

end of showresults  
type=5  
res=testhash1  
res=value1  
res=testhash2  
res=value2

Posted at Nginx Forum:

> **[Re: lua parser and redis2 questions](https://forum.nginx.org/read.php?2,203154,203946#msg-203946)**
>
> Hi Folks - Looking for some examples or hints in order to leverage lua\_parser && redis2. To date, I've been using httpredis and content\_by\_lua code to perform a number of simple 'get's which has worked well. Now, I'd like to pipeline a number...

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 6, 2011, 4:56pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/12 "2011-06-06T16:56:03Z")

</div>

On Mon, Jun 6, 2011 at 7:16 PM, logar.damir [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid)  
wrote:

> Agentzh, how about when using select  
> ([SELECT | Docs](http://www.redis.io/commands/select)) to select specific database #.  
> Does pipelining work ok in this case?
> 
> My command at the moment is:  
> redis2\_query select 2;  
> redis2\_query evalsha $sha1keyvickrey 1 $val;  
> redis2\_query select 0;  
> redis2\_pass redisbackend;

Your sample above should work. If not, please consider filing a bug  
ticket on GitHub 😉

> Are this 3 redis2\_query commands running on the same connection?

Yes. What I meant in my last email is that there is no such guarantee  
across multiple nginx (sub-)requests. Your pipelined redis commands  
specified in the same nginx location are always sent on one single  
redis TCP connection. So you shouldn’t worry about that in this  
context.

> If not,  
> could you maybe add another variable for database number. Simething  
> similar to HttpRedis that has $redis\_db (The number of redis database)  
> which I think is broken ([http://forum.nginx.org/read.php?2,203861](http://forum.nginx.org/read.php?2,203861)).

No, I do not want ngx\_redis2 to mud with specific redis command  
details because the consistency and simplicity of its uniformed wired  
protocol is one of the beauties and powers of redis. Leveraging the  
redis pipelining feature of ngx\_redis2 is sufficient.

Regards,  
-agentzh

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 7, 2011, 3:15am UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/13 "2011-06-07T03:15:20Z")

</div>

On Tue, Jun 7, 2011 at 12:50 AM, gregr401 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Well, all this seems to be tracked back to an env issue on my side,  
> argh. I ended up setting a lua\_package\_cpath and now parsing replies is  
> working exactly as they should (interesting that some other features  
> were working, tho, hmmm).

Yeah, using the lua\_package\_cpath directive is considered one of the  
best practices 😉

> Thanks again for the above help and validation agentzh and logar!
> 
> nginx + redis pipelining + lua FTW =)

Happy hacking!  
-agentzh

---

<div class="post-metadata">

**Author:** ![gregr401](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/g/f07891/32.png) [@gregr401](https://www.ruby-forum.com/u/gregr401)\
**Post date:** [June 6, 2011, 6:51pm UTC](https://www.ruby-forum.com/t/lua-parser-and-redis2-questions/207759/14 "2011-06-06T18:51:01Z")

</div>

Well, all this seems to be tracked back to an env issue on my side,  
argh. I ended up setting a lua\_package\_cpath and now parsing replies is  
working exactly as they should (interesting that some other features  
were working, tho, hmmm).

Thanks again for the above help and validation agentzh and logar!

nginx + redis pipelining + lua FTW =)

Posted at Nginx Forum:

> **[Re: lua parser and redis2 questions](https://forum.nginx.org/read.php?2,203154,204376#msg-204376)**
>
> Hi Folks - Looking for some examples or hints in order to leverage lua\_parser && redis2. To date, I've been using httpredis and content\_by\_lua code to perform a number of simple 'get's which has worked well. Now, I'd like to pipeline a number...