Forum: NGINX Setting memcache keys in eval block

Hello,


How can i set a memcache key using an arbitrary variable? It seems so
straightforward but i cannot get it up and running. I try to fetch 
memcache
results using eval, which i will then pass to memcache once again.


This works:
    set $memcached_key $cookie_token;
    memcached_pass     127.0.0.1:11211;

This doesn't work:
    set $cookie_token 'bla';
    set $memcached_key $cookie_token;
    memcached_pass     127.0.0.1:11211;


Both Memcached modules will complain that the memcached_key or memc_key
variable was not set, this all happens inside an eval block, outside the 
eval
block does work as expected . I'm using the latest 0.7 stable Nginx and 
tried
both Memcached modules.



Regards,

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

On Mon, Feb 8, 2010 at 8:36 PM, Markus Jelsma <markus@buyways.nl> wrote:
> Both Memcached modules will complain that the memcached_key or memc_key
> variable was not set, this all happens inside an eval block, outside the eval
> block does work as expected.

The eval module's handler runs before the ngx_rewrite module. So make
sure you always "set" variables inside the eval block. The "set"
directives outside the eval block does not work because they run after
the "eval" block and hence the memcached handler complete.

IIRC, this was mentioned in the ngx_eval module's documentation.

Cheers,
-agentzh

Hello,


Thanks for your reply. The manual indeed tells us to set it inside the 
block,
which i am doing. However, the first result cannot be passed to the 
second
eval block;

The following does not work; it sends the error message $memc_key 
variable is
not set:

      eval $key
      {
        # Check if we got the token, a 160bit SHA1 hash
        if ($cookie_token ~* "([A-Za-z0-9]{40})")
        {
          # Attempt to retrieve the token from memcache
          set $memc_cmd 'get';
          set $memc_key $cookie_token;

          # Retrieve the token's value and store it in the local key 
variable
          memc_pass 127.0.0.1:11211;
        }
      }
      eval $email
      {
        set $memc_cmd 'get';
        set $memc_key $key;
        memc_pass 127.0.0.1:11211;
      }

      echo 'test';
      echo $email;
      echo_flush;

I really need two seperate gets. Nesting the  second eval in the first 
is also
not allowed because then we have a duplicate memc_pass which it 
complains
about.

For the moment i `solved` this issue by adding a second location which 
the
first eval proxies to, this does work but shows up as a second HTTP 
request in
the logs; this feels a bit slow and i really would like to handle this 
subject
within one location directive.

To summarize the problem; using a variable set by eval in a subsequent 
eval
block does not work.

Some advice is much appreciated :)



Cheers,





The eval module's handler runs before the ngx_rewrite module. So make
sure you always "set" variables inside the eval block. The "set"
directives outside the eval block does not work because they run after
the "eval" block and hence the memcached handler complete.

IIRC, this was mentioned in the ngx_eval module's documentation.


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

On Wed, Feb 10, 2010 at 6:18 PM, Markus Jelsma <markus@buyways.nl> 
wrote:
> Thanks for your reply. The manual indeed tells us to set it inside the block,
> which i am doing. However, the first result cannot be passed to the second
> eval block;
>

No, it won't work. According to the current implementation, only one
eval block can take effect in a single location.

I must admit parallel eval blocks can be *very* useful and I believe
it should not be that hard to implement. But I have other missions to
do first atm ;)

Actually I want something much more general, that can be mixed with
other rewrite directives, like this:

     set $foo 'hi';
     set_capture_location $res '/foo';
     if ($res ~ 'xxx') { ... }
     set_capture_subrequest $res POST '/bar' 'body here';
     if ($res ~ 'xxx') { ... }
     ....

I'll ask Marcus Clyne if he has any plan to add support for such
things to his set_var submodule in his grand NDK project. Then an
ngx_capture module should be straightforward ;)

Cheers,
-agentzh

Thank you for your quick answer, although it is not quite a satisfaction 
to
read it's not possible at the moment.

How can i stay up to date for such a feature if it were to be 
implemented in
the - hopefully nearby - future?

I have another question, is my current solution really much slower than 
doing
it all in one location directive? I am now proxying the request the 
myself
where i can set the second eval block.

I did perform some simple benchmarks but they aren't really alright 
since i
cannot really compare the situations but 2 http requests + 2 memcache 
requests
are quite slow in the end.

Thanks.


>     set $foo 'hi';
>     set_capture_location $res '/foo';
>     if ($res ~ 'xxx') { ... }
>     set_capture_subrequest $res POST '/bar' 'body here';
>     if ($res ~ 'xxx') { ... }
>     ....
>
>I'll ask Marcus Clyne if he has any plan to add support for such
>things to his set_var submodule in his grand NDK project. Then an
>ngx_capture module should be straightforward ;)

Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

Markus Jelsma wrote:
> I did perform some simple benchmarks but they aren't really alright since i 
>> I must admit parallel eval blocks can be very useful and I believe
>>     if ($res ~ 'xxx') { ... }
>>     ....
>>
>> I'll ask Marcus Clyne if he has any plan to add support for such
>> things to his set_var submodule in his grand NDK project. Then an
>> ngx_capture module should be straightforward ;)
>>     
Yes, this currently isn't possible (AFAICT), because of how the
http_rewrite_module's phase handler works, but it should be possible
with easy modification of that, which will allow for recalling rewrites
after subrequests.  I'm busy with other stuff right now, but I'll try to
build the generic interface (if not the subrequest part) soon and
include it in the first launched version of the NDK.

Marcus.

On Wed, Feb 10, 2010 at 6:45 PM, Markus Jelsma <markus@buyways.nl> 
wrote:
>
> How can i stay up to date for such a feature if it were to be implemented in
> the - hopefully nearby - future?
>

Yes, hopefully in the near future :) I'm busy with the ngx_array_var
module development as well as the ngx_srcache module atm. Maybe I'll
have some spare time to hack that in after these modules are out.

But there's a workaround that you can try out *now*. Please read on.

I do have a personal fork of the ngx_eval module here:

    http://github.com/agentzh/nginx-eval-module

Currently it has support for arbitrary content handlers as well as
output filters. Here's some quick examples that you *may* be
interested in:

    location /echo {
        eval_subrequest_in_memory off;
        eval $a {
            echo_before_body BEFORE;
            echo THIS;
        }
        echo '[$a]';
    }

Then GET /echo yields

    [BEFORE
    THIS]

asssuming you configured the ngx_echo module *after* the ngx_eval
module such that the echo_before_body filter runs before ngx_eval's.

This also means that you can take advantage of the echo_location_async
or echo_location directives provided by ngx_echo to do your multiple
eval's in a single location. Like this:


    location /echo {
        eval_subrequest_in_memory off;
        eval $union {
            echo_location_async /memc1;
            echo 'XXXX';
            echo_location_async /memc2;
        }
        if ($union ~ '(.*)XXXX\n(.*)') {
             set $res1 $1;
             set $res2 $2;
             ...
        }
    }
    location /memc1 {
          memcached_pass ...;
    }
    location /memc2 {
          memcached_pass ...;
    }

This should be more efficient than the internal proxying approach.
Feel free to do some benchmark on your side to confirm this ;)

Cheers,
-agentzh

P.S. I've sent a pull request to Valery Kholodkov in the hope to get
my patch for ngx_eval merged into the mainstream. But no reply yet.
Sigh.

On Thu, Feb 11, 2010 at 5:35 PM, agentzh <agentzh@gmail.com> wrote:
>             set $res2 $2;
>             ...
>        }
>    }

Oops, my bad! new subrequests issued by
echo_location/echo_location_async in an eval block will not work at
all because ngx_eval's output filter cannot capture its
sub-sub-request's output.

Sorry about that. My fork of ngx_eval is currently limited to content
handlers and output filters that do not issue subrequests themselves.

Cheers,
-agentzh

On Thu, Feb 11, 2010 at 5:40 PM, agentzh <agentzh@gmail.com> wrote:
> Oops, my bad! new subrequests issued by
> echo_location/echo_location_async in an eval block will not work at
> all because ngx_eval's output filter cannot capture its
> sub-sub-request's output.
>
> Sorry about that. My fork of ngx_eval is currently limited to content
> handlers and output filters that do not issue subrequests themselves.
>

I suddenly realized last night that I can fix this by walking through
the r->parent chain to find the "sentinel subrequest" (or "top-level
subrequest") that is issued directly by the ngx_eval module and the
corresponding ctx object (if any). This also applies to the
ngx_srcache module that I've been working on, which also needs to
capture response in an output filter.

Cheers,
-agentzh

On Thu, Feb 11, 2010 at 12:05 PM, Marcus Clyne <ngx.eugaia@gmail.com> 
wrote:
>
> Yes, this currently isn't possible (AFAICT), because of how the
> http_rewrite_module's phase handler works, but it should be possible with
> easy modification of that, which will allow for recalling rewrites after
> subrequests.

Yay! Marcus you rock ;)

>  I'm busy with other stuff right now, but I'll try to build the
> generic interface (if not the subrequest part) soon and include it in the
> first launched version of the NDK.
>

Heh, I can do the subrequest part, which is easy for me ;)

Thanks!
-agentzh

Hi,


It sounds quite mysterious to me - i'm not a Nginx developer afterall - 
but i
suppose this is good news for the use case?


Cheers,


>I suddenly realized last night that I can fix this by walking through
>the r->parent chain to find the "sentinel subrequest" (or "top-level
>subrequest") that is issued directly by the ngx_eval module and the
>corresponding ctx object (if any). This also applies to the
>ngx_srcache module that I've been working on, which also needs to
>capture response in an output filter.


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

On Fri, Feb 12, 2010 at 5:42 PM, Markus Jelsma <markus@buyways.nl> 
wrote:
> It sounds quite mysterious to me - i'm not a Nginx developer afterall - but i
> suppose this is good news for the use case?
>

Good news :) But there's one missing bit that I've forgotten to
mention: we'll have to take into account the postponed chain for
subrequests.

Cheers,
-agentzh

Markus Jelsma wrote:
> Thank you for your quick answer, although it is not quite a satisfaction to 
> read it's not possible at the moment.
> 
> How can i stay up to date for such a feature if it were to be implemented in 
> the - hopefully nearby - future?

I just have pushed a change to nginx eval module which allows multiple
eval blocks in one location:

http://github.com/vkholodkov/nginx-eval-module/tree/bba2d53fc1d8f118fb79424250db8c9e832b66c1

Hope this helps!

> 
>>
>>     set $foo 'hi';
>>     set_capture_location $res '/foo';
>>     if ($res ~ 'xxx') { ... }
>>     set_capture_subrequest $res POST '/bar' 'body here';
>>     if ($res ~ 'xxx') { ... }
>>     ....
>>
>> I'll ask Marcus Clyne if he has any plan to add support for such
>> things to his set_var submodule in his grand NDK project. Then an
>> ngx_capture module should be straightforward ;)

--
Best regards,
Valery Kholodkov

Valery!


This sounds lovely! I'll try this back at the office tomorrow. I assume
this also allows for multiple memc_pass and proxy_pass directives in a
single location block?

Anyway, i'm quite happy and thank you very much for your efforts!


Cheers,


Valery Kholodkov said:

Valery!


It works like a charm! I can now have multiple *_pass directives inside 
a
single location block. Also, multiple eval blocks are supported and work 
as
expected!

Thanks thanks thanks!


Cheers,


>http://github.com/vkholodkov/nginx-eval-module/blob/bba2d53fc1d8f118fb794242
>50db8c9e832b66c1/ngx_http_eval_module.c


Markus Jelsma - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350