Memcached/JSONP

[dylanz]

Gotcha. Thanks Marcus, that makes perfect sense.
In regard to echo_location_async, I’m assuming it would allow something
like so?

location /first {
echo “before”;
echo_location_aysnc /second$request_uri;
echo “after”
}

location /second(.)?(.)^ {
set $memcached_key $1; # needing this to be keyed on the
request_path, not the entire uri
memcached_pass 127.0.0.1;
}

That doesn’t seem to work, and am not able to extrapolate the request
path out of there.

The other concern is that if I was to patch the before/after echo
methods, if they would be
returned in the response even if there was no valid body being passed
back? For example:

location /first {
echo_before_body -n “before”;
echo_location_aysnc /null; # this returns nothing
echo_after_body -n “after”
}

In the above scenario, I think I’d have a response of “before after”,
even if the async call returned an empty response.
Think I’m slowly getting there, and just need to make sure I’m picking
the correct route
Thanks!

Posted at Nginx Forum:

Re: Memcached/JSONP

[dylanz]

On Wed, Dec 23, 2009 at 8:36 AM, dylanz [email protected] wrote:

Well, I think you should not assume the query string part, i.e.,
“?xxx”, can be specified in the location name Try the following:

location /first {
echo “before”;
echo_location_async /second $request_uri;
echo “after”;
}

location = /second {
set $memcached_key $query_string;
memcached_pass 127.0.0.1;
}

In the above scenario, I think I’d have a response of “before after”, even if the async call returned an empty response.

I think you should use custom error pages and the try_files directive
so that it fall back to an empty page or an null json object:

location @null_page {
echo_duplicate 1 “null”; # generate null
}

Think I’m slowly getting there, and just need to make sure I’m picking the correct route

The ngx_echo module does very little itself, it just exposes the
internal API of the nginx core. So it often requires to utilize other
goodies in the ngx_http_core and/or ngx_http_rewrite modules

Cheers,
-agentzh

[dylanz]

On Tue, Dec 22, 2009 at 8:58 AM, dylanz [email protected] wrote:

/* append the newline character /
/ TODO add support for -n option to suppress

• the trailing newline */
  newline_buf = ngx_calloc_buf(r->pool);

Hah! I asked too soon. Should have read the source I’ll investigate adding a patch for this.

Nice Looking forward to that

Thanks!
-agentzh

[dylanz]

On Wed, Dec 23, 2009 at 5:01 AM, dylanz [email protected] wrote:

Hey agentzh!

I noticed that “echo” isn’t outputting on non-async events, for example:
echo “before”;
set $memcached_key $uri;
memcached_pass 127.0.0.1:11211;
echo “after”;

Just as Marcus has said, it makes little sense to specify two content
handlers in the same location (here, ngx_echo’s handler and
ngx_memcached’s). Or you can use the filter directives of the ngx_echo
module, just as what I demonstrated earlier:

location /foo {
echo_before_body “before”;
set $memcached_key $uri;
memcached_pass 127.0.0.1:11211;
echo_after_body “after”;
}

That results in the contents of the memcached_pass, but doesn’t include the echo output.

Because ngx_memcached wins in the tie ngx_echo sets the content
handler for that location first, and then ngx_memcached overrides it
with its own

I tried throwing in some echo_flush commands to see if that would help, but it didn’t.
However, it does work if I use the after/before echo commands, for example:

echo_before_body “before”;
set $memcached_key $uri;
memcached_pass 127.0.0.1:11211;
echo_after_body “after”;

Sure, because here you use ngx_echo’s output filter and it always works

I added the -n flag option to echo.c, and it works when I’m not doing doing that proxy pass,

What do you mean by “when I’m not doing doing that proxy pass”?

for example, this:

echo -n “hello”
echo -n “there”
echo “world”

… produces “hello there world”, all on one line.

Is that known behaviour?

Sure. And I’d like the -n option gets parsed before variable
interpolation happens. For example

set $foo ‘-n hello’;
echo $foo;

will output “-n hello\n”, not “hello”

If so, I’ll work around it. Otherwise, if it’s not, let me know and
I can see what I can do about fixing it

It’s very nice of you to contribute to the ngx_echo module

Thanks!
-agentzh

[dylanz]

Hi,

agentzh wrote:

memcached_pass 127.0.0.1;
}

Two problems here :

1. The location directive doesn’t test query strings, it tests just the
   URL (before the ?), so this will always fail unless you’ve URL-encoded
   the ‘?’ character - you should use

if ($arg_[name] ~ [REGEX PATTERN]) {
# do something
}

if you want to test an argument

2. The ‘^’ should be a ‘$’ if you wanted it to mean the end of the
   string - I think here it would be testing a literal ‘^’ character

As an extra note, if you’re planning on doing such subqueries, you’re
probably better off not using regexes on the subquery location - it’s a
waste of resources.

Do something like

location /first {
…
echo_location_async /sub/$request_uri;
…
}

location ^~ /sub {
internal; # see below

set   $memcached_key ...

}

The ^~ is the most efficient way of testing a static location (rather
than just location /sub or using regexes).
Specifying the location as ‘internal’ will prevent direct access to the
url from a request (it’s ok for internal subrequests or redirects,
though). You may or may not want to do this.

If you can avoid doing rewrites to get the $memcached_key, it would be
better - no wasting resources.

Marcus.

[dylanz]

agentzh Wrote:

 memcached_pass 127.0.0.1;
echo_location_async /second $request_uri;
echo “after”;
}

location = /second {
set $memcached_key $query_string;
memcached_pass 127.0.0.1;
}

Great point! What you proposed is almost there, but, I don’t want the
$query_string, but simply the path.
I’ve tried a few different combinations of location directives, but
can’t seem to get it right. For example:

If I made a request to:
/first/11111.ext?callback=222222

I’d want to look up the following path in Memcached:
/first/11111.ext

So, when I do that echo_async_location, I don’t see how I’m able to
extrapolate the path out.
If there was an internal variable that held the $uri, minus the
$query_string, that would do it!
However, I don’t think that exists at the moment.

 echo_after_body -n “after”
json object:

location @null_page {
echo_duplicate 1 “null”; # generate null
}

That’s perfect!

Cheers,
-agentzh

---

nginx mailing list
[email protected]
nginx Info Page

Posted at Nginx Forum:

Re: Memcached/JSONP

[dylanz]

Hi,

dylanz wrote:

string part, i.e.,
set $memcached_key $query_string;

I’d want to look up the following path in Memcached:
/first/11111.ext

So, when I do that echo_async_location, I don’t see how I’m able to extrapolate the path out.
If there was an internal variable that held the $uri, minus the $query_string, that would do it!
However, I don’t think that exists at the moment.

set $memcached_key $uri;

This will give the current (possibly rewritten) URI, without arguments.

Marcus.

[dylanz]

On Wed, Dec 23, 2009 at 1:12 PM, dylanz [email protected] wrote:

Great point! What you proposed is almost there, but, I don’t want the $query_string, but simply the path.

Understood. How about this?

location /first {
echo “before”;
echo_location_async /second $uri;
echo “after”;
}

location = /second {
set $memcached_key $query_string;
memcached_pass 127.0.0.1;
}

If you’re actually worried about special characters in the parsed URI
in $uri when feeding it as a memcached key, try Marcus C.'s
ngx_set_var module to pre-hash the key

http://simpl.it/ngx/set_var/ngx_http_set_var_module-0.1.0.tar.gz

I’ve tried a few different combinations of location directives, but can’t seem to get it right. For example:

If I made a request to:
/first/11111.ext?callback=222222

I’d want to look up the following path in Memcached:
/first/11111.ext

So, when I do that echo_async_location, I don’t see how I’m able to extrapolate the path out.

Use $uri rather than $request_uri. I know the difference between these
two is very subtle, especially when not looking into the source. Blame
Igor or some others

If there was an internal variable that held the $uri, minus the $query_string, that would do it!
However, I don’t think that exists at the moment.

$uri is the parsed form of $request_uri, with query string excluded.
And $request_uri is always the main request’s URI while $uri is the
current (sub)request’s

Cheers,
-agentzh

[dylanz]

On Wed, Dec 23, 2009 at 1:47 PM, agentzh [email protected] wrote:

memcached_pass 127.0.0.1;

}

Actually there’s always more than one way to do it in nginx. If you
really want to use regexes and match captures, try the following
version of config with “if” statement:

location /first {
    echo "before";
    echo_location_async /second $request_uri;
    echo "after";
}

location = /second {
     if ($query_string ~ '([^?]+)') {
         set $memcached_key $1;
         echo $memcached_key;
     }
}

Tested successfully on my side

Happy hacking nginx config files!

Cheers,
-agentzh

[dylanz]

Hi,

If you’re actually worried about special characters in the parsed URI
in $uri when feeding it as a memcached key, try Marcus C.'s
ngx_set_var module to pre-hash the key

http://simpl.it/ngx/set_var/ngx_http_set_var_module-0.1.0.tar.gz

Thanks for the mention, but I wouldn’t recommend hashing memcached keys
unless you absolutely have to. Apart from the (albeit slim) chance of
hash collisons (i.e. two URLs matching the same hash), it’s a waste of
resources.

You should remember that the URLs will be passed URL-unencoded, though
(I believe - best to check that).

$uri is the parsed form of $request_uri, with query string excluded.
And $request_uri is always the main request’s URI while $uri is the
current (sub)request’s

Looks like I duplicated this info in my last email. Oops.

Marcus.