# Two type of rate limiting for based on IP address

**URL:** <https://www.ruby-forum.com/t/two-type-of-rate-limiting-for-based-on-ip-address/217764>\
**Category:** NGINX\
**Created:** [March 12, 2012, 5:18am UTC](https://www.ruby-forum.com/t/two-type-of-rate-limiting-for-based-on-ip-address/217764 "2012-03-12T05:18:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\
**Post date:** [March 12, 2012, 5:18am UTC](https://www.ruby-forum.com/t/two-type-of-rate-limiting-for-based-on-ip-address/217764/1 "2012-03-12T05:18:03Z")

</div>

Hi all,

How can I maintain two rate limit strategies?One for spiders and one  
for  
regular users?

I can get the IP address list of spiders from

[http://www.iplists.com/](http://www.iplists.com/) . Can I separate it by geo? Have people  
attempted  
this?

My website is being pounded by some screen scrapers and I want to block  
them, but not at the risk of blocking search engine spiders.

-Quintin

---

<div class="post-metadata">

**Author:** ![Quintin\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/q/13edae/32.png) [@Quintin\_P](https://www.ruby-forum.com/u/Quintin_P)\
**Post date:** [March 12, 2012, 3:34pm UTC](https://www.ruby-forum.com/t/two-type-of-rate-limiting-for-based-on-ip-address/217764/2 "2012-03-12T15:34:42Z")

</div>

> My website is being pounded by some screen scrapers and I want to block  
> them, but not at the risk of blocking search engine spiders.

Do you understand that by going that way, regular users will be subject  
to  
the same request limiting of the bad spiders? You can try to do further  
selection on UA, but bad spiders have the habit of providing bogus UA  
strings.

At the http level:

geo $good\_spider {  
default 0;  
#list all good spider IPs  
}

limit\_req\_zone $binary\_remote\_addr zone=bad\_spiders:10m rate=1r/s;

On the vhost (server level):

location / {  
limit\_req zone=bad\_spiders burst=5;

```
error_page 418 @good-spiders;

if ($good_spider) {
    return 418;
}
#...

```

}

location @good-spiders {  
# no limits here  
#…  
}

–appa

---

<div class="post-metadata">

**Author:** ![Quintin\_P](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/q/13edae/32.png) [@Quintin\_P](https://www.ruby-forum.com/u/Quintin_P)\
**Post date:** [March 12, 2012, 3:47pm UTC](https://www.ruby-forum.com/t/two-type-of-rate-limiting-for-based-on-ip-address/217764/3 "2012-03-12T15:47:10Z")

</div>

\> \> location / { \> limit\_req zone=bad\_spiders burst=5; \> \> error\_page 418 @good-spiders;

Oops. This should be:

error\_page 418 =200 @good-spiders;

–appa