# Limit\_req for spiders only

**URL:** <https://www.ruby-forum.com/t/limit-req-for-spiders-only/236955>\
**Category:** NGINX\
**Created:** [October 14, 2013, 1:59pm UTC](https://www.ruby-forum.com/t/limit-req-for-spiders-only/236955 "2013-10-14T13:59:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![alcina](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/a/a587f6/32.png) [@alcina](https://www.ruby-forum.com/u/alcina)\
**Post date:** [October 14, 2013, 1:59pm UTC](https://www.ruby-forum.com/t/limit-req-for-spiders-only/236955/1 "2013-10-14T13:59:56Z")

</div>

Hi,

I would like to put a brake on spiders which are hammering a site with  
dynamic content generation. They should still get to see the content,  
but only not generate excessive load. I therefore constructed a map to  
identify spiders, which works well, and then tried to

limit\_req\_zone $binary\_remote\_addr zone=slow:10m …;

if ($is\_spider) {  
limit\_req zone=slow;  
}

Unfortunately, limit\_req is not allowed inside “if”, and I don’t see an  
obvious way to achieve this effect otherwise.

If you have any tips, that would be much appreciated!

Kind regards,  
–Toni++

---

<div class="post-metadata">

**Author:** ![Toni\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f04885/32.png) [@Toni\_M](https://www.ruby-forum.com/u/Toni_M)\
**Post date:** [October 14, 2013, 3:25pm UTC](https://www.ruby-forum.com/t/limit-req-for-spiders-only/236955/2 "2013-10-14T15:25:51Z")

</div>

Hello.

Doesnt robots.txt “Crawl-Delay” directive satisfy your needs?  
Normal spiders should obey robots.txt, if they dont - they can be  
banned.

Posted at Nginx Forum:

> **[Re: limit\_req for spiders only](https://forum.nginx.org/read.php?2,243670,243674#msg-243674)**
>
> Hi, I would like to put a brake on spiders which are hammering a site with dynamic content generation. They should still get to see the content, but only not generate excessive load. I therefore constructed a map to identify spiders, which works...

---

<div class="post-metadata">

**Author:** ![Toni\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f04885/32.png) [@Toni\_M](https://www.ruby-forum.com/u/Toni_M)\
**Post date:** [October 14, 2013, 4:03pm UTC](https://www.ruby-forum.com/t/limit-req-for-spiders-only/236955/3 "2013-10-14T16:03:14Z")

</div>

Hello,

On Mon, Oct 14, 2013 at 09:25:24AM -0400, Sylvia wrote:

> Doesnt robots.txt “Crawl-Delay” directive satisfy your needs?

I have it already there, but I don’t know how long it takes for such a  
directive, or any changes to robots.txt for that matter, to take effect.  
Observing the logs, I’d say that this delay between changing robots.txt  
and a change in robot behaviour would take several days, as I cannot see  
any effects so far.

> Normal spiders should obey robots.txt, if they dont - they can be banned.

Banning Google is not a good idea, no matter how abusive they might be,  
and they incidentically operate one of those robots which keep hammering  
the site. I’d much prefer a technical solution to enforce such limits,  
over convention.

I’d also like to limit the request frequency over an entire pool, so  
that I can say “clients from this pool can make requests only with this  
fequency, combined, not per client IP”, because it doesn’t buy me  
anything if I can limit the individual search robot to a decent  
frequency, but then get hammered by 1000 search robots in parallel, each  
one observing the request limit. Right?

Kind regards,  
–Toni++

---

<div class="post-metadata">

**Author:** ![Toni\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f04885/32.png) [@Toni\_M](https://www.ruby-forum.com/u/Toni_M)\
**Post date:** [October 14, 2013, 4:23pm UTC](https://www.ruby-forum.com/t/limit-req-for-spiders-only/236955/4 "2013-10-14T16:23:42Z")

</div>

On Mon, Oct 14, 2013 at 01:59:23PM +0200, Toni M. wrote:

Hi there,

This is untested, but follows the docs at  
[http://nginx.org/r/limit\_req\_zone:](http://nginx.org/r/limit_req_zone:)

> I therefore constructed a map to  
> identify spiders, which works well, and then tried to
> 
> limit\_req\_zone $binary\_remote\_addr zone=slow:10m …;
> 
> if ($is\_spider) {  
> limit\_req zone=slow;  
> }

> If you have any tips, that would be much appreciated!

In your map, let $is\_spider be empty if is not a spider (“default”,  
presumably), and be something else if it is a spider (possibly  
$binary\_remote\_addr if every client should be counted individually,  
or something else if you want to group some spiders together.)

Then define

limit\_req\_zone $is\_spider zone=slow:10m …;

instead of what you currently have.

## f

Francis D. removed\_email\_address@domain.invalid

---

<div class="post-metadata">

**Author:** ![Toni\_M](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/t/f04885/32.png) [@Toni\_M](https://www.ruby-forum.com/u/Toni_M)\
**Post date:** [October 14, 2013, 4:52pm UTC](https://www.ruby-forum.com/t/limit-req-for-spiders-only/236955/5 "2013-10-14T16:52:56Z")

</div>

H Francis,

On Mon, Oct 14, 2013 at 03:23:03PM +0100, Francis D. wrote:

> In your map, let $is\_spider be empty if is not a spider (“default”,  
> presumably), and be something else if it is a spider (possibly  
> $binary\_remote\_addr if every client should be counted individually,  
> or something else if you want to group some spiders together.)

thanks a bunch! This works like a charm!

Kind regards,  
–Toni++