Forum: NGINX limit_req_zone: How to apply only to some requests containing some string in the URL?

2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-23 05:43
(Received via mailing list)
Hi,

I'm using the limit_req_zone module. I would like it to act only on some
requests that have a certain string in one variable in the query string
of
the URL.For example, lets say that I'm providing a IP geolocation
service,
and that in the URL there is a query string like this:

http://api.acme.com/ipgeolocation/locate?key=NANDS...

I would like the rule to detect when the "key" parameter ends with "2X",
and
in such case to apply the limitation.
What I really need is to give NGINX a secret message. The "key"
parameter
would end in "01X",  "02X", "03X" (etc). This would indicate Nginx the
limitation of queries per minute, and Nginx would apply a different rate
for
each request, depending on the "message".

Is there a way to do that?

Thanks in advance!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244015#msg-244015
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-10-23 10:09
(Received via mailing list)
On Tue, Oct 22, 2013 at 11:42:35PM -0400, Brian08275660 wrote:

Hi there,

> I'm using the limit_req_zone module. I would like it to act only on some
> requests that have a certain string in one variable in the query string of
> the URL.

> http://api.acme.com/ipgeolocation/locate?key=NANDS...
>
> I would like the rule to detect when the "key" parameter ends with "2X", and
> in such case to apply the limitation.

http://nginx.org/r/limit_req_zone

See $variable.

Use, for example, "map" to make your variable have a value when you want
the limit to apply, and be empty when you don't.

> What I really need is to give NGINX a secret message. The "key" parameter
> would end in "01X",  "02X", "03X" (etc). This would indicate Nginx the
> limitation of queries per minute, and Nginx would apply a different rate for
> each request, depending on the "message".

For that, I think you'd need a different limit_req zone for each rate.

After you've got the first part working, it shouldn't be too hard to
set up a test system to see if you can get this part to work too.

  f
--
Francis Daly        francis@daoine.org
2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-23 14:57
(Received via mailing list)
Could you please give me an example? A few lines of code would be great!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244035#msg-244035
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-10-23 18:42
(Received via mailing list)
On Wed, Oct 23, 2013 at 08:56:33AM -0400, Brian08275660 wrote:

Hi there,

> Could you please give me an example? A few lines of code would be great!

Completely untested, but something like:

  map $arg_key $key_ends_in_02X {
    default "";
    ~02X$ "A";
  }

(where "A" might be, for example, $binary_remote_addr, if you want to
limit the requests per client IP, rather than enforce the limit across
all clients where "key" ends "02X")

and then

  limit_req_zone $key_ends_in_02X zone=two:10m rate=2r/s;

and

  limit_req zone=two;

should probably do what you asked for.

If anything is unclear or broken, and the documentation doesn't make
it clear how it should be, then feel free to respond pointing out which
specific parts need fixing.

Cheers,

  f
--
Francis Daly        francis@daoine.org
2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-23 19:39
(Received via mailing list)
Thanks a lot Francis! Now I just have to learn how to use the custom
variables and the "map" directive. I haven't ever used them before,
never
needed them.
By the way, it will be easier than what I thought. I decided to ask my
users
to add an extra parameter, something like "&capacity=3X" instead of
"hiding"
some characters in the key that would indicate what to do. So at least I
wont have to make a lot of effort with the regular expressions, the
parameter will be very clear. My logic will chech its value and apply
the
respective limit.

Its interesting how weird is the configuration of Nginx. No XML tags,
but
weird directives that are not really straightforward.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244049#msg-244049
2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-24 15:18
(Received via mailing list)
Hi Francis,

Thanks a lot! You saved me probably a couple of days of research. It is
working now!

I did this:

The user will send my a "capacity" parameter, with a value of 2X or 3X
or 4X
or......(etc)

  map $arg_capacity $2X_key{~*2X $http_x_forwarded_for;default "";}
  map $arg_capacity $3X_key{~*3X $http_x_forwarded_for;default "";}
  map $arg_capacity $4X_key{~*4X $http_x_forwarded_for;default "";}
  map $arg_capacity $5X_key{~*5X $http_x_forwarded_for;default "";}
  map $arg_capacity $6X_key{~*6X $http_x_forwarded_for;default "";}
  map $arg_capacity $7X_key{~*7X $http_x_forwarded_for;default "";}
  map $arg_capacity $8X_key{~*8X $http_x_forwarded_for;default "";}
  map $arg_capacity $9X_key{~*9X $http_x_forwarded_for;default "";}
  map $arg_capacity $10X_key{~*10X $http_x_forwarded_for;default "";}

  limit_req_zone $2X_key zone=2X:1m rate=600r/m;
  limit_req_zone $3X_key zone=3X:1m rate=900r/m;
  limit_req_zone $4X_key zone=4X:1m rate=1200r/m;
  limit_req_zone $5X_key zone=5X:1m rate=1500r/m;
  limit_req_zone $6X_key zone=6X:1m rate=1800r/m;
  limit_req_zone $7X_key zone=7X:1m rate=2100r/m;
  limit_req_zone $8X_key zone=8X:1m rate=2400r/m;
  limit_req_zone $9X_key zone=9X:1m rate=2700r/m;
  limit_req_zone $10X_key zone=10X:1m rate=3000r/m;

  limit_req zone=2X burst=600;
  limit_req zone=3X burst=900;
  limit_req zone=4X burst=1200;
  limit_req zone=5X burst=1500;
  limit_req zone=6X burst=1800;
  limit_req zone=7X burst=2100;
  limit_req zone=8X burst=2400;
  limit_req zone=9X burst=2700;
  limit_req zone=10X burst=3000;

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244064#msg-244064
2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-25 19:32
(Received via mailing list)
Hi Francis,

Now I need to create a limit that acts if the parameter ("capacity") is
empty (not provided). How do I do that? I can't find how to, at least
not
using "map".
(If provided, the other rules will evaluate it and one of them will act
according to the value).

Thanks in advanced!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244123#msg-244123
C32118085c49a5e4974825cb2ea2652d?d=identicon&s=25 Jader H. Silva (Guest)
on 2013-10-25 19:59
(Received via mailing list)
Hi,

you could use a map that matches all cases to empty string and default
value as non-empty

map $arg_capacity $my_default_key{~*([2-9]|10)X ""; default
$http_x_forwarded_for;}

If it matches 2X to 10X, $my_default_key will be empty.

Cheers
Jader H. Silva


2013/10/25 Brian08275660 <nginx-forum@nginx.us>
D0a98f6304ff122ac9b487ff8275d8bb?d=identicon&s=25 Payam Chychi (Guest)
on 2013-10-25 23:02
(Received via mailing list)
Maybe ive misunderstood but cant you very simply do this by injecting a
cookie on the origina req page and then have nginx match, count it and
apply rates? Or maybe im comicating it... If even possible

--
Payam Chychi
Network Engineer / Security Specialist
2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-25 23:14
(Received via mailing list)
Hi Jader,

Thanks a lot, that looks like a nice solution!

I barely know how to build regex expressions, and I'm too lazy to learn
just
right now. Just a final question: Actually they will send me any value
from
2X to 25X, but that could even increase to more than 25. I would like a
simpler and more open REGEX, something like:

If it is something that includes an "X"

I will use it as this:

map $arg_capacity $my_default_key{   If it is something that includes an
"X"
           ""; default $http_x_forwarded_for;}

Simple as that. This would catch any case, from 2X to 1000000000X.
Actually
it would catch even illegal values. But that is not a problem for me,
cause
in my java code that follows Nginx I'm validating the value they send me
anyway, so I don't mind if they send me a value like "-1X" because my
app
would return an error status code and force them to send me a valid
value
anyway. The important thing is that Nginx should apply the limit if the
received value is not empty, and asking if it contains an "X" covers
that
boolean question.

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244129#msg-244129
2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-25 23:18
(Received via mailing list)
Hi Payam,

I dont have that option. My users are not using real browsers, but
objects
that model an HTTP client. Probably these object can't inject cookies.
And I
don't want to ask them to so so, it would make things more complex to
them,
whereas including an extra parameter in the query string is a piece of
cake.

Thanks anyway for the idea!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244130#msg-244130
2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-26 01:08
(Received via mailing list)
I read some information about REGEX and think I found the way to express
"X
or x, preceded with something before":

~*(.*)X

I think that the first two characters mean "match anycase", then the
"(.*)"
would mean "any quantity of characters" and the "X" would mean that
specific
letter.

Am I right?

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244136#msg-244136
36a8284995fa0fb82e6aa2bede32adac?d=identicon&s=25 Francis Daly (Guest)
on 2013-10-26 14:53
(Received via mailing list)
On Fri, Oct 25, 2013 at 07:07:20PM -0400, Brian08275660 wrote:

Hi there,

> ~*(.*)X
>
> I think that the first two characters mean "match anycase", then the "(.*)"
> would mean "any quantity of characters" and the "X" would mean that specific
> letter.
>
> Am I right?

Yes.

But unless you're going to do something with the bit before the X,
then in the context of a map{}, it is equivalent to ~*X, since all you
care about is whether it matches.

It will just as easily match x123 or 123x123.

  f
--
Francis Daly        francis@daoine.org
2974d09ac2541e892966b762aad84943?d=identicon&s=25 Brian08275660 (Guest)
on 2013-10-26 15:38
(Received via mailing list)
Oh, ok. Then it is similar to REGEX in Java.
Well, then I think I have a nice and elegant solution, Thanks!

Posted at Nginx Forum:
http://forum.nginx.org/read.php?2,244015,244149#msg-244149
Please log in before posting. Registration is free and takes only a minute.
Existing account

NEW: Do you have a Google/GoogleMail, Yahoo or Facebook account? No registration required!
Log in with Google account | Log in with Yahoo account | Log in with Facebook account
No account? Register here.