Issue with 3rd-party memc and eval modules

Server NGINX
1

Hi,

There is an issue with the 3rd-party memc module for Nginx 1 where it
doesn’t close the connection when it resides in an eval 2 block.

Consider the following with memc in eval:

   eval $memc_return
   {
     set $memc_cmd 'set';
     set $memc_key 'key';
     set $memc_value 'value';

     memc_pass memcache_cluster;
   }

  echo 'hello!';
  echo_flush;

When this location is called via curl, you’ll get to see the following:

  • About to connect() to localhost port 81 (#0)
  • Trying 127.0.0.1… connected
  • Connected to localhost (127.0.0.1) port 81 (#0)

GET
/session/synchronize/da39a3ee5e6b4b0d3255bfef95601890afd80709/da39a3ee5e6b4b0d3255bfef95601890afd80709/abc
HTTP/1.1
User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g
zlib/1.2.3.3 libidn/1.15
Host: localhost:81
Accept: /

^C
markus@zealand:/opt/nginx#

You can see i have to close the connection manually. I have also got a
verbose
memcache server log which exactly closes the connection at the point i
press
CTRL+C.

Now consider the following with memcached in eval, i have changed the
above
configuration to use memcached instead of memc:

  • About to connect() to localhost port 81 (#0)
  • Trying 127.0.0.1… connected
  • Connected to localhost (127.0.0.1) port 81 (#0)

GET
/session/synchronize/da39a3ee5e6b4b0d3255bfef95601890afd80709/da39a3ee5e6b4b0d3255bfef95601890afd80709/abc
HTTP/1.1
User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g
zlib/1.2.3.3 libidn/1.15
Host: localhost:81
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 16 Feb 2010 10:46:22 GMT
< Content-Type: text/plain
< Transfer-Encoding: chunked
< Connection: close
<
hello!

  • Closing connection #0

You can clearly see what’s going on: this is what it should look like.
Now,
also consider using the memc module with the eval lines commented out. I
also
had to comment out the echo lines because otherwise it won’t pass to
memcache:

  • About to connect() to localhost port 81 (#0)
  • Trying 127.0.0.1… connected
  • Connected to localhost (127.0.0.1) port 81 (#0)

GET
/session/synchronize/da39a3ee5e6b4b0d3255bfef95601890afd80709/da39a3ee5e6b4b0d3255bfef95601890afd80709/abc
HTTP/1.1
User-Agent: curl/7.19.5 (i486-pc-linux-gnu) libcurl/7.19.5 OpenSSL/0.9.8g
zlib/1.2.3.3 libidn/1.15
Host: localhost:81
Accept: /

< HTTP/1.1 201 Created
< Server: nginx
< Date: Tue, 16 Feb 2010 10:50:31 GMT
< Content-Type: text/plain
< Content-Length: 8
< Connection: close
<
STORED

  • Closing connection #0

There is also a difference in HTTP response code and body if only using
set,
here is what i found:

memc returns code=HTTP 201 body=STORED
memcached returns code=HTTP 200 body= ← this is empty

Can the contributers of the modules please tell us something about this
behaviour? I’m using the latest stable 0.7 nginx and modules.

Many many thanks in advance

Cheers,

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

2

Hi Markus,
could you check if the same happens with agentzh’s modified eval module?

Best regards,
Piotr S. < [email protected] >

3

Hello Piotr,

Thanks for you reply but i am already using the latest module which
allows for
multiple eval blocks in a single location, works fine by the way.

Cheers,

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

4

Hello Markus,

Thanks for you reply but i am already using the latest module which allows
for
multiple eval blocks in a single location, works fine by the way.

Yes, I’m aware of that. But the version you’re using (official one)
doesn’t
include changes made by agentzh, hence my request for you to check if it
works with memc. Please note that this modification was made before
multiple
evel blocks were added, so it will work with only one block.

Anyway, if memc will work fine with modifications made by agentzh, then
you
should ask Valery to include them in official release.

Best regards,
Piotr S. < [email protected] >

5

Hi,

There is a small mistake in the first post; the standard memcache module
does
not support commands, hence it cannot be used to set variables. The
issue with
the 3rd-party memc module persists though.

Cheers,

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

6

Hello Piotr,

Sorry for the late reply, the ML is not very happy with me i guess for i
didn’t get this in my mailbox. I’ve downloaded and compiled agentzh’s
eval
module.

What comes out:

Well, agentzh’s latest eval source - directly taken from the most recent
commit - not only works with multiple eval blocks, it also neatly allows
the
memc module to close it’s connection to the memcache server. So it seems
it is
not memc issue but has to do with the eval module.

First i used Valery’s source from his github, it seems the source at the
time
did allow for multiple eval blocks but could not let the connection
close.

The question now is, will their come an official source for the eval
module?
Or may we assume Valery’s page 1 is the one point to download from?

Thanks Piotr for the tip!

Regards,

Yes, I’m aware of that. But the version you’re using (official one) doesn’t
include changes made by agentzh, hence my request for you to check if it
works with memc. Please note that this modification was made before multiple
evel blocks were added, so it will work with only one block.

Anyway, if memc will work fine with modifications made by agentzh, then you
should ask Valery to include them in official release.

Best regards,
Piotr S. < [email protected] >

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

7

Hi Markus,

Well, agentzh’s latest eval source - directly taken from the most recent
commit - not only works with multiple eval blocks,

This must be some nice side-effect :slight_smile: As far as I know agentzh intended
to
make eval module work with ngx_drizzle / ngx_postgres / ngx_oracle, he
didn’t mention anything about multiple blocks to me.

it also neatly allows the
memc module to close it’s connection to the memcache server. So it seems
it is
not memc issue but has to do with the eval module.

Great!

The question now is, will their come an official source for the eval
module?
Or may we assume Valery’s page [1] is the one point to download from?

You mean version with agentzh’s modification?
I don’t think so, unless Valery will pull the changes into his
repository :wink:

Best regards,
Piotr S. < [email protected] >

8

Sounds good!

But Valery, i previously used your new eval module that allows for
multiple
blocks. That version had an issue with not letting memc_pass close the
memcached connection when setting values (getting values worked
properly).

Now, for the moment, i use agentzh’s eval which for some reason also
allows
for multiple eval blocks and does let memc_pass close the connection.

I’d prefer to use your release if a fix could be made that closes the
connection with the memcache server.

Regards,

avoided, if an appropriate state machine is implemented.

As soon as these issues are resolved, I’ll accept this code.

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

9

----- Markus J. [email protected] wrote:

I’d prefer to use your release if a fix could be made that closes the
connection with the memcache server.

I’ll try to fix this problem.

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

nginx mailing list
[email protected]
nginx Info Page

–

Regards,
Valery K.

10

On Wed, Feb 17, 2010 at 7:07 PM, Valery K.
[email protected] wrote:

In particular I don’t like the presence of “eval_subrequest_in_memory” directive at all, because it specifies which way of internal communication the module needs to use, which doesn’t make any sense for end user.

Thanks for replying to my pull request (although indirectly). Yeah, I
don’t like the name of that directive as well. I chose that just
because I can’t find a better name at that moment :stuck_out_tongue:

Second, the contributed code allocates a buffer and uses it in order to assemble the output of subrequest in it. This allocation can be totally avoided, if an appropriate state machine is implemented.

I just copied (almost) the way that ngx_http_upstream does buffered
outputing :wink: To be honest, I didn’t and don’t see how that allocation
can be totally avoided for general use cases. Silly me! :stuck_out_tongue:

As soon as these issues are resolved, I’ll accept this code.

It’ll be great if you provide some more elaborate suggestions or even
patches to my patches if you do have the tuits :slight_smile:

Just as Piotr mentioned ealier, the initial motivation for my ngx_eval
branch is to provide support for ngx_drizzle + ngx_rds_json. Adding
subrequest-in-memory support to every content handler/upstream handler
with output filter support is boring and no fun. So I chose to fix
ngx_eval rather than my content handler/upstream modules :wink:

Thanks!
-agentzh

11

The question now is, will their come an official source for the eval
module?
Or may we assume Valery’s page [1] is the one point to download from?

You mean version with agentzh’s modification?
I don’t think so, unless Valery will pull the changes into his repository :wink:

I appreciate your interest in my software, but the problem with this
contributed code at the moment is that it is not as efficient as it
could possibly be and it is not universal enough.

In particular I don’t like the presence of “eval_subrequest_in_memory”
directive at all, because it specifies which way of internal
communication the module needs to use, which doesn’t make any sense for
end user.

Second, the contributed code allocates a buffer and uses it in order to
assemble the output of subrequest in it. This allocation can be totally
avoided, if an appropriate state machine is implemented.

As soon as these issues are resolved, I’ll accept this code.

–
Regards,
Valery K.

12

Great!! Let us know, we’ll test it as soon as we can and report back on
it.

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

13

Hi Markus,

Anyway, i wonder if creating a module on my own would be easier in the
end,
dealing with many conditional operators and eval blocks inside the
configuration is either quite a challenge or not feasible.

It would actually help if you would share with us what are you trying to
achieve :wink:

Best regards,
Piotr S. < [email protected] >

14

Hi,

I’ve tested a bit further and the both agentzh and Valery’s module do
not
close the connection inside an interal location called by
echo_subrequest. In
that internal location i try to use proxy_pass to fetch an uuid from a
backend.

Earlier i had that block of code in a larger location but due to various
module’s order of execution i tried to split the parts into internal
location.

I immediately got the same problem as before. I’ve then tried to use the
other
module but it doesn’t work either.

Anyway, i wonder if creating a module on my own would be easier in the
end,
dealing with many conditional operators and eval blocks inside the
configuration is either quite a challenge or not feasible.

Cheers,

I’ll try to fix this problem.

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

15

Hi,

In the end, i need two pieces of server side logic:

  1. HTTP request filter that looks up a key in memcache (specified in a
    cookie). Then, an additional lookup is required and finally either the
    URL can
    be rewritten or the original cookie can be overwritten.

In pseudo:
request.cookie.token = memcache_get(memcache_get(request.cookie.token));

Of course, some validation checks need to be executed such as verifying
if the
client’s IP address is actually linked to the token.

This part already works and uses two eval blocks.

  1. A HTTP request with multiple parameters needs to be processed and a
    cookie
    needs to be set (eval) which value is a random hash generated by some
    backend
    (eval), that value then will be stored in memcache as well (eval) as a
    HTTP
    specified token. This step is a token synchronization process that
    initializes
    a session so it can be called using the logic above.

This second part is filled with checks, validations etc and is hard to
implement because of order of exection (eval before rewrite). But i
really
need rewrite’s conditional statement a lot of times. Splitting it all up
in
seperate named locations is quite a challenge.

Doing all this in the config would be nice but on the other hand making
a
module for it would be even better. It would be some sort of combination
of
the memcache module and user_id module.

Cheers,

It would actually help if you would share with us what are you trying to
achieve :wink:

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

16

Yes, i can provide a snippet. In the end i have decided to abandon the
project
of doing it in configuration. It is extremely hard to try and write
imperative
code in a declarative language; although we do have the if operator 1,
working with it is more troublesome than ever before :wink:

We will now try and build a module for our purpose, at least, i hope so.

Anyway, here is a snippet to reproduce a non -closing connection using
eval in
a named location called by echo_exec - it gets complicated indeed.

location /test
{
  echo_exec @initialize;
}

location @initialize
{
  eval_override_content_type 'text/plain';

  eval $id
  {
    rewrite ^(.*)$ /id;
    proxy_pass http://data-1:8080;
  }
}

This will fail and also fail with a memc_pass call. However, it will
work if
you replace the echo_exec call with the entire contents of the named
location.

Even this little piece of code gave me a hard time; imagine how some 50
lines
using various eval blocks, proxy_pass and memc_pass directives and many
evil
if operators :wink:

Anyway, thanks for all the support on the way!

Regards,

On Wednesday 24 February 2010 05:26:04 agentzh wrote:

tested a bit further and the both agentzh and Valery’s module do not

close the connection inside an interal location called by echo_subrequ

Markus J. - Technisch Architect - Buyways BV
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350

17

On Wed, Feb 24, 2010 at 9:52 PM, Markus J. [email protected]
wrote:

Yes, i can provide a snippet. In the end i have decided to abandon the project
of doing it in configuration. It is extremely hard to try and write imperative
code in a declarative language; although we do have the if operator 1,
working with it is more troublesome than ever before :wink:

Sigh. I’m sure there’s some bug in one of those modules that you were
using. I’m afraid you’ll also have a hard time in writing a module
yourself because the nginx’s core does not have a nice API :wink:

We will now try and build a module for our purpose, at least, i hope so.

Anyway, here is a snippet to reproduce a non -closing connection using eval in
a named location called by echo_exec - it gets complicated indeed.

I’ve reproduced this connection hang using the following (slightly
modified) version of config:

location /test
{
echo_exec @initialize;
}

location @initialize
{
eval_override_content_type ‘text/plain’;

 eval $id
 {
   #rewrite ^(.*)$ /id;
   proxy_pass http://127.0.0.1:$server_port/id;
 }
 echo $id;

}
location /id {
echo hi;
}

Our Test::Nginx::Socket module reports a timeout that the nginx server
does not close the client connection in time.

But I’ve found a work-around by avoid using named location:

location /test
{
echo_exec /initialize;
}

location /initialize
{
internal;
eval_override_content_type ‘text/plain’;

 eval $id
 {
   #rewrite ^(.*)$ /id;
   proxy_pass http://127.0.0.1:$server_port/id;
 }
 echo $id;

}
location /id {
echo hi;
}

Then GET /test gives the output as expected:

 hi

It seems to be a named location issue. According to the implementation
in the nginx core, “named location” is quite buggy. And we’ve run into
some other issues earlier as well. So my suggestion is to use the
“internal” directive combined with normal locations for such kind of
things.

Even this little piece of code gave me a hard time; imagine how some 50 lines
using various eval blocks, proxy_pass and memc_pass directives and many evil
if operators :wink:

I do understand the frustration. Without considerable knowledge to the
nginx internals, it’s quite difficult to diagnose issues when things
go wrong.

Anyway, thanks for all the support on the way!

You’re always welcome to report detailed issues to the list and please
don’t give up config file scripting :wink:

Well, the “if” directive in the http rewrite module is not that evil
as long as we know how it works. I believe the evilness of “if” has
been exaggerated. It just not works the way you’re used to in most of
other languages like C and Perl.

We’ve been enjoying “if” in our nginx.conf for something big:

http://agentzh.org/misc/nginx.conf

I’ve given some hints regarding “if”'s behavior here:

Module vars set inside if's are not set

It’s not that hard to predict how it works :wink:

On the other hand, I’d say named locations are indeed evil because it
does not clear the request object like normal locations and may lead
to consequences that are very hard to predict.

Cheers,
-agentzh

18

On Thu, Feb 18, 2010 at 6:55 PM, Markus J. [email protected]
wrote:

I’ve tested a bit further and the both agentzh and Valery’s module do not
close the connection inside an interal location called by echo_subrequest.

Can you provide a minimized config snippet that can reproduce this
problem? It’s getting a bit complicated it seems :wink:

BTW, I’m a big fan of nginx.conf scripting myself :wink:

Cheers,
-agentzh