Setting memcache keys in eval block

Server NGINX
#1

Hello,

How can i set a memcache key using an arbitrary variable? It seems so
straightforward but i cannot get it up and running. I try to fetch
memcache
results using eval, which i will then pass to memcache once again.

This works:
set $memcached_key $cookie_token;
memcached_pass 127.0.0.1:11211;

This doesn’t work:
set $cookie_token ‘bla’;
set $memcached_key $cookie_token;
memcached_pass 127.0.0.1:11211;

Both Memcached modules will complain that the memcached_key or memc_key
variable was not set, this all happens inside an eval block, outside the
eval
block does work as expected . I’m using the latest 0.7 stable Nginx and
tried
both Memcached modules.

Regards,

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

#2

Hello,

Thanks for your reply. The manual indeed tells us to set it inside the
block,
which i am doing. However, the first result cannot be passed to the
second
eval block;

The following does not work; it sends the error message $memc_key
variable is
not set:

  eval $key
  {
    # Check if we got the token, a 160bit SHA1 hash
    if ($cookie_token ~* "([A-Za-z0-9]{40})")
    {
      # Attempt to retrieve the token from memcache
      set $memc_cmd 'get';
      set $memc_key $cookie_token;

      # Retrieve the token's value and store it in the local key

variable
memc_pass 127.0.0.1:11211;
}
}
eval $email
{
set $memc_cmd ‘get’;
set $memc_key $key;
memc_pass 127.0.0.1:11211;
}

  echo 'test';
  echo $email;
  echo_flush;

I really need two seperate gets. Nesting the second eval in the first
is also
not allowed because then we have a duplicate memc_pass which it
complains
about.

For the moment i solved this issue by adding a second location which
the
first eval proxies to, this does work but shows up as a second HTTP
request in
the logs; this feels a bit slow and i really would like to handle this
subject
within one location directive.

To summarize the problem; using a variable set by eval in a subsequent
eval
block does not work.

Some advice is much appreciated :slight_smile:

Cheers,

The eval module’s handler runs before the ngx_rewrite module. So make
sure you always “set” variables inside the eval block. The “set”
directives outside the eval block does not work because they run after
the “eval” block and hence the memcached handler complete.

IIRC, this was mentioned in the ngx_eval module’s documentation.

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

#3

On Mon, Feb 8, 2010 at 8:36 PM, Markus J. [email protected] wrote:

Both Memcached modules will complain that the memcached_key or memc_key
variable was not set, this all happens inside an eval block, outside the eval
block does work as expected.

The eval module’s handler runs before the ngx_rewrite module. So make
sure you always “set” variables inside the eval block. The “set”
directives outside the eval block does not work because they run after
the “eval” block and hence the memcached handler complete.

IIRC, this was mentioned in the ngx_eval module’s documentation.

Cheers,
-agentzh

#4

On Wed, Feb 10, 2010 at 6:18 PM, Markus J. [email protected]
wrote:

Thanks for your reply. The manual indeed tells us to set it inside the block,
which i am doing. However, the first result cannot be passed to the second
eval block;

No, it won’t work. According to the current implementation, only one
eval block can take effect in a single location.

I must admit parallel eval blocks can be very useful and I believe
it should not be that hard to implement. But I have other missions to
do first atm :wink:

Actually I want something much more general, that can be mixed with
other rewrite directives, like this:

 set $foo 'hi';
 set_capture_location $res '/foo';
 if ($res ~ 'xxx') { ... }
 set_capture_subrequest $res POST '/bar' 'body here';
 if ($res ~ 'xxx') { ... }
 ....

I’ll ask Marcus C. if he has any plan to add support for such
things to his set_var submodule in his grand NDK project. Then an
ngx_capture module should be straightforward :wink:

Cheers,
-agentzh

#5

Markus J. wrote:

I did perform some simple benchmarks but they aren’t really alright since i

I must admit parallel eval blocks can be very useful and I believe
if ($res ~ ‘xxx’) { … }
…

I’ll ask Marcus C. if he has any plan to add support for such
things to his set_var submodule in his grand NDK project. Then an
ngx_capture module should be straightforward :wink:

Yes, this currently isn’t possible (AFAICT), because of how the
http_rewrite_module’s phase handler works, but it should be possible
with easy modification of that, which will allow for recalling rewrites
after subrequests. I’m busy with other stuff right now, but I’ll try to
build the generic interface (if not the subrequest part) soon and
include it in the first launched version of the NDK.

Marcus.

#6

Thank you for your quick answer, although it is not quite a satisfaction
to
read it’s not possible at the moment.

How can i stay up to date for such a feature if it were to be
implemented in
the - hopefully nearby - future?

I have another question, is my current solution really much slower than
doing
it all in one location directive? I am now proxying the request the
myself
where i can set the second eval block.

I did perform some simple benchmarks but they aren’t really alright
since i
cannot really compare the situations but 2 http requests + 2 memcache
requests
are quite slow in the end.

Thanks.

set $foo 'hi';
set_capture_location $res '/foo';
if ($res ~ 'xxx') { ... }
set_capture_subrequest $res POST '/bar' 'body here';
if ($res ~ 'xxx') { ... }
....

I’ll ask Marcus C. if he has any plan to add support for such
things to his set_var submodule in his grand NDK project. Then an
ngx_capture module should be straightforward :wink:

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

#7

On Thu, Feb 11, 2010 at 5:35 PM, agentzh [email protected] wrote:

        set $res2 $2;
        ...
   }

}

Oops, my bad! new subrequests issued by
echo_location/echo_location_async in an eval block will not work at
all because ngx_eval’s output filter cannot capture its
sub-sub-request’s output.

Sorry about that. My fork of ngx_eval is currently limited to content
handlers and output filters that do not issue subrequests themselves.

Cheers,
-agentzh

#8

On Wed, Feb 10, 2010 at 6:45 PM, Markus J. [email protected]
wrote:

How can i stay up to date for such a feature if it were to be implemented in
the - hopefully nearby - future?

Yes, hopefully in the near future :slight_smile: I’m busy with the ngx_array_var
module development as well as the ngx_srcache module atm. Maybe I’ll
have some spare time to hack that in after these modules are out.

But there’s a workaround that you can try out now. Please read on.

I do have a personal fork of the ngx_eval module here:

http://github.com/agentzh/nginx-eval-module

Currently it has support for arbitrary content handlers as well as
output filters. Here’s some quick examples that you may be
interested in:

location /echo {
    eval_subrequest_in_memory off;
    eval $a {
        echo_before_body BEFORE;
        echo THIS;
    }
    echo '[$a]';
}

Then GET /echo yields

[BEFORE
THIS]

asssuming you configured the ngx_echo module after the ngx_eval
module such that the echo_before_body filter runs before ngx_eval’s.

This also means that you can take advantage of the echo_location_async
or echo_location directives provided by ngx_echo to do your multiple
eval’s in a single location. Like this:

location /echo {
    eval_subrequest_in_memory off;
    eval $union {
        echo_location_async /memc1;
        echo 'XXXX';
        echo_location_async /memc2;
    }
    if ($union ~ '(.*)XXXX\n(.*)') {
         set $res1 $1;
         set $res2 $2;
         ...
    }
}
location /memc1 {
      memcached_pass ...;
}
location /memc2 {
      memcached_pass ...;
}

This should be more efficient than the internal proxying approach.
Feel free to do some benchmark on your side to confirm this :wink:

Cheers,
-agentzh

P.S. I’ve sent a pull request to Valery K. in the hope to get
my patch for ngx_eval merged into the mainstream. But no reply yet.
Sigh.

#9

On Thu, Feb 11, 2010 at 12:05 PM, Marcus C. [email protected]
wrote:

Yes, this currently isn’t possible (AFAICT), because of how the
http_rewrite_module’s phase handler works, but it should be possible with
easy modification of that, which will allow for recalling rewrites after
subrequests.

Yay! Marcus you rock :wink:

I’m busy with other stuff right now, but I’ll try to build the
generic interface (if not the subrequest part) soon and include it in the
first launched version of the NDK.

Heh, I can do the subrequest part, which is easy for me :wink:

Thanks!
-agentzh

#10

On Thu, Feb 11, 2010 at 5:40 PM, agentzh [email protected] wrote:

Oops, my bad! new subrequests issued by
echo_location/echo_location_async in an eval block will not work at
all because ngx_eval’s output filter cannot capture its
sub-sub-request’s output.

Sorry about that. My fork of ngx_eval is currently limited to content
handlers and output filters that do not issue subrequests themselves.

I suddenly realized last night that I can fix this by walking through
the r->parent chain to find the “sentinel subrequest” (or “top-level
subrequest”) that is issued directly by the ngx_eval module and the
corresponding ctx object (if any). This also applies to the
ngx_srcache module that I’ve been working on, which also needs to
capture response in an output filter.

Cheers,
-agentzh

#11

Hi,

It sounds quite mysterious to me - i’m not a Nginx developer afterall -
but i
suppose this is good news for the use case?

Cheers,

I suddenly realized last night that I can fix this by walking through
the r->parent chain to find the “sentinel subrequest” (or “top-level
subrequest”) that is issued directly by the ngx_eval module and the
corresponding ctx object (if any). This also applies to the
ngx_srcache module that I’ve been working on, which also needs to
capture response in an output filter.

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

#12

Markus J. wrote:

Thank you for your quick answer, although it is not quite a satisfaction to
read it’s not possible at the moment.

How can i stay up to date for such a feature if it were to be implemented in
the - hopefully nearby - future?

I just have pushed a change to nginx eval module which allows multiple
eval blocks in one location:

http://github.com/vkholodkov/nginx-eval-module/tree/bba2d53fc1d8f118fb79424250db8c9e832b66c1

Hope this helps!

set $foo 'hi';
set_capture_location $res '/foo';
if ($res ~ 'xxx') { ... }
set_capture_subrequest $res POST '/bar' 'body here';
if ($res ~ 'xxx') { ... }
....

I’ll ask Marcus C. if he has any plan to add support for such
things to his set_var submodule in his grand NDK project. Then an
ngx_capture module should be straightforward :wink:

–
Best regards,
Valery K.

#13

On Fri, Feb 12, 2010 at 5:42 PM, Markus J. [email protected]
wrote:

It sounds quite mysterious to me - i’m not a Nginx developer afterall - but i
suppose this is good news for the use case?

Good news :slight_smile: But there’s one missing bit that I’ve forgotten to
mention: we’ll have to take into account the postponed chain for
subrequests.

Cheers,
-agentzh

#14

Valery!

It works like a charm! I can now have multiple *_pass directives inside
a
single location block. Also, multiple eval blocks are supported and work
as
expected!

Thanks thanks thanks!

Cheers,

http://github.com/vkholodkov/nginx-eval-module/blob/bba2d53fc1d8f118fb794242
50db8c9e832b66c1/ngx_http_eval_module.c

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

#15

Valery!

This sounds lovely! I’ll try this back at the office tomorrow. I assume
this also allows for multiple memc_pass and proxy_pass directives in a
single location block?

Anyway, i’m quite happy and thank you very much for your efforts!

Cheers,

Valery K. said:

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs