# Blocking by user agent if ip doesn't match

**URL:** <https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690>\
**Category:** NGINX\
**Created:** [May 31, 2011, 8:50pm UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690 "2011-05-31T20:50:08Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![dubstep](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/d/cdc98d/32.png) [@dubstep](https://www.ruby-forum.com/u/dubstep)\
**Post date:** [May 31, 2011, 8:50pm UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/1 "2011-05-31T20:50:08Z")

</div>

Hello everyone. Sorry for double posting this question in How to section  
of the forum but I’ve noticed later there is a lot of un-replied threads  
there and that mailing list is more active so I am assuming I have more  
chance of getting some help here.

I am hoping this is possible and I’d really appreciate some help on  
configuring it.  
We are having some bots on our site that are using google spider user  
agent but they are fake and their ip range has nothing to do with  
Google.  
So I am looking for some solution that would match visitors with user  
agent is Google who’s ip doesn’t start with for example 66.x or 70.x and  
block them from accessing.

I’ve found this example on one site  
if ($http\_user\_agent ~ (Purebot|Lipperhey|MaMa  
CaSpEr|libwww-perl|Mail.Ru|gold crawler) ) {  
return 403;  
}

But I don’t know how to add ip regex match to that and what end result  
would be. Or negative match in this case I guess.  
It is something like this for htaccess !^66.\*$ to match those ips that  
don’t start with 66. for example. But again I am not sure what the right  
syntax would be in nginx config or if it is even possible to match both  
statements.

Posted at Nginx Forum:

> **[Blocking by user agent if ip doesn't match](https://forum.nginx.org/read.php?2,202715,202715#msg-202715)**
>
> Hello everyone. Sorry for double posting this question in How to section of the forum but I've noticed later there is a lot of un-replied threads there and that mailing list is more active so I am assuming I have more chance of getting some help...

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [May 31, 2011, 9:05pm UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/2 "2011-05-31T21:05:36Z")

</div>

On 31 Mai 2011 19h49 WEST, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> start with for example 66.x or 70.x and block them from accessing.
> 
> I’ve found this example on one site  
> if ($http\_user\_agent ~ (Purebot|Lipperhey|MaMa  
> CaSpEr|libwww-perl|Mail.Ru|gold crawler) ) {  
> return 403;  
> }

Try this:

if ($http\_user\_agent ~\* “Google Bot”) {  
allow 66.x;  
allow 70.x;  
deny all;  
}

— appa

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [May 31, 2011, 9:19pm UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/3 "2011-05-31T21:19:50Z")

</div>

On 5/31/11, karabaja [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> syntax would be in nginx config or if it is even possible to match both  
> statements.

It is possible and there are different ways to match two conditions. I  
like this one:

geo $google {  
default 0;  
66.0.0.0/8 1;  
}  
map $http\_user\_agent $googlebot {  
default 0;  
~google $google;  
}

server {  
location / {  
if ($googlebot) {  
…  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [May 31, 2011, 9:35pm UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/4 "2011-05-31T21:35:17Z")

</div>

On Tue, May 31, 2011 at 10:19:13PM +0300, Alexandr G. wrote:

> }  
> }  
> }

It works since 0.9.6. I’m going to change case sensitivity like this

~Google # case sensitive  
~\*google # case insensitive

Expression compatible with old and new syntax (for gracefull upgrade):

~(?i)google # case insensitive

As to configuraiton it’s better to use this logic:

geo $not\_google {  
default 1;  
66.0.0.0/8 0;  
}

map $http\_user\_agent $bots {  
default 0;  
~(?i)google $not\_google;  
“~(?i)(Purebot|Lipperhey|MaMaCaSpEr|libwww-perl|Mail.Ru|gold  
crawler)” 1;  
}

server {  
location / {  
if ($bots) {  
return 403;  
}  
}  
}

–  
Igor S.

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [May 31, 2011, 10:35pm UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/5 "2011-05-31T22:35:16Z")

</div>

On 5/31/11, Igor S. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> It works since 0.9.6. I’m going to change case sensitivity like this
> 
> ~Google # case sensitive  
> ~\*google # case insensitive

It would be nice. 'Cause current implementation is a bit confusing,  
i.e. inconsistent with other regexes.

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [May 31, 2011, 11:05pm UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/6 "2011-05-31T23:05:38Z")

</div>

On 31 Mai 2011 20h34 WEST, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> It works since 0.9.6. I’m going to change case sensitivity like this
> 
> ~Google # case sensitive  
> ~\*google # case insensitive
> 
> Expression compatible with old and new syntax (for gracefull  
> upgrade):

Question: Is using the map directive more efficient than using a  
“naive” if with access control directives?

Thanks,  
— appa

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [May 31, 2011, 11:29pm UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/7 "2011-05-31T23:29:08Z")

</div>

On 6/1/11, Antnio P. P. Almeida [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Question: Is using the map directive more efficient than using a  
> “naive” if with access control directives?

I think it’s just considered to be a bad practice to use anything  
rather than return or rewrite in ifs.  
In terms of efficiency: geo uses rbtrees and access module uses  
arrays. But I don’t think it’s about that. It’s more of an example to  
illustrate how to use multiple conditions and new features in nginx.

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 12:26am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/8 "2011-06-01T00:26:34Z")

</div>

On 31 Mai 2011 23h01 WEST, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Thanks everyone for being so helpful. I’ve ended up applying Igor’s  
> suggestion. But I’ve dropped this line as I wasn’t sure what to do  
> with it:  
> “~(?i)(Purebot|Lipperhey|MaMaCaSpEr|libwww-perl|Mail.Ru|gold  
> crawler)” 1;
> 
> I am guessing it can be used if I want to match more then just  
> google’s user agent. But in any case what I did worked very nice. I  
> tested it using Firefox user agent and I got forbidden page, then  
> tried adding my ip to geo bit and I was allowed.

Yes following Alexandr and Igor’s advice you can create similar  
variables using more IP blocks inside the geo directive. E.g.:

geo $bad\_bot {  
default 1;  
66.0.0.0/8 0;  
xx.yy.zz.ww/16 1; # for Yahoo!  
(…)  
}

You’ll need also to add the regexes for User Agent string of the  
remaining bots that you want to whitelist in the map directive.

map $http\_user\_agent $bots {  
default 0;  
~(?i)(google|yahoo) $bad\_bot;  
}

— appa

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 12:01am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/9 "2011-06-01T00:01:53Z")

</div>

Thanks everyone for being so helpful. I’ve ended up applying Igor’s  
suggestion.  
But I’ve dropped this line as I wasn’t sure what to do with it:  
“~(?i)(Purebot|Lipperhey|MaMaCaSpEr|libwww-perl|Mail.Ru|gold crawler)”  
1;

I am guessing it can be used if I want to match more then just google’s  
user agent. But in any case what I did worked very nice.  
I tested it using Firefox user agent and I got forbidden page, then  
tried adding my ip to geo bit and I was allowed.

And there is lot less “google spiders” on our site now, and all of them  
have ips I added to allowed list.

Thanks again for fast and great advices.

Since I am a noob when it comes to Nginx and now that I know I can get  
some help here I am sure I’ll have more questions 🙂

Posted at Nginx Forum:

> **[Re: Blocking by user agent if ip doesn't match](https://forum.nginx.org/read.php?2,202715,202770#msg-202770)**
>
> Hello everyone. Sorry for double posting this question in How to section of the forum but I've noticed later there is a lot of un-replied threads there and that mailing list is more active so I am assuming I have more chance of getting some help...

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 12:45am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/10 "2011-06-01T00:45:05Z")

</div>

Thanks Antonio, so far we’ve only had fake google bots so it seems fine  
as it is for now. But I’ll apply those if there is any issue with Yahoo  
or some other spider being faked.  
I am assuming that other user agents are not affected by the current  
rule.

Posted at Nginx Forum:

> **[Re: Blocking by user agent if ip doesn't match](https://forum.nginx.org/read.php?2,202715,202776#msg-202776)**
>
> Hello everyone. Sorry for double posting this question in How to section of the forum but I've noticed later there is a lot of un-replied threads there and that mailing list is more active so I am assuming I have more chance of getting some help...

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 12:32am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/11 "2011-06-01T00:32:45Z")

</div>

On 31 Mai 2011 23h25 WEST, removed\_email\_address@domain.invalid wrote:

Oops. Now correct.

geo $bad\_bot {  
default 1;  
66.0.0.0/8 0;  
xx.yy.zz.ww/16 0; # for Yahoo!  
(…)  
}

— appa

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 12:58am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/12 "2011-06-01T00:58:38Z")

</div>

On 31 Mai 2011 23h44 WEST, [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Thanks Antonio, so far we’ve only had fake google bots so it seems  
> fine as it is for now. But I’ll apply those if there is any issue  
> with Yahoo or some other spider being faked. I am assuming that  
> other user agents are not affected by the current rule.

No the default value of $bots is 0. Only in the case of a UA string  
containing google (case insensitive) is the IP verified. Otherwise,  
since you dropped the line that matched the bad bots UA, all other  
bots are allowed.

This is a good approach for handling bots spoofing the UA string and  
also a cleaner way to block unwanted bots.

— appa

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 9:28am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/13 "2011-06-01T09:28:36Z")

</div>

On Tue, May 31, 2011 at 06:01:19PM -0400, karabaja wrote:

> Thanks everyone for being so helpful. I’ve ended up applying Igor’s  
> suggestion.  
> But I’ve dropped this line as I wasn’t sure what to do with it:  
> “~(?i)(Purebot|Lipperhey|MaMaCaSpEr|libwww-perl|Mail.Ru|gold crawler)”  
> 1;
> 
> I am guessing it can be used if I want to match more then just google’s  
> user agent. But in any case what I did worked very nice.  
> I tested it using Firefox user agent and I got forbidden page, then  
> tried adding my ip to geo bit and I was allowed.

I misread your message and thought that you block alreay these bots,  
so I have added them to block despite their IPs.

–  
Igor S.

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 9:26am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/14 "2011-06-01T09:26:06Z")

</div>

On Tue, May 31, 2011 at 10:04:11PM +0100, Antnio P. P. Almeida wrote:

> > 
> 
> Question: Is using the map directive more efficient than using a  
> “naive” if with access control directives?

Due to “if” block implementation, this may not work in some  
configurations.

–  
Igor S.

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 10:47am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/15 "2011-06-01T10:47:45Z")

</div>

On 6/1/2011 2:34 AM, Igor S. wrote:

> As to configuraiton it’s better to use this logic:  
> geo $not\_google {  
> default 1;  
> 66.0.0.0/8 0;  
> }

Does geo directive will also work for ipv6 ?

ie:

geo $not\_google {  
default 1;  
66.0.0.0/8 0;  
2404:6800::/32 0;  
}

I hardly find wiki explained about ipv6 (  
[GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/HttpGeoModule) )

thanks

Powered By [http://www.3g-net.net](http://www.3g-net.net)

---

<div class="post-metadata">

**Author:** ![karabaja](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/7ea924/32.png) [@karabaja](https://www.ruby-forum.com/u/karabaja)\
**Post date:** [June 1, 2011, 10:48am UTC](https://www.ruby-forum.com/t/blocking-by-user-agent-if-ip-doesnt-match/207690/16 "2011-06-01T10:48:22Z")

</div>

On Wed, Jun 01, 2011 at 03:42:12PM +0700, Hari Hendaryanto wrote:

> ie:
> 
> geo $not\_google {  
> default 1;  
> 66.0.0.0/8 0;  
> 2404:6800::/32 0;  
> }
> 
> I hardly find wiki explained about ipv6 (  
> [GitHub - nginxinc/nginx-wiki: ARCHIVED -- Source for the now archived NGINX Wiki section of https://www.nginx.com · GitHub](http://wiki.nginx.org/HttpGeoModule) )

No, currently geo does not support ipv6.

–  
Igor S.