# Slow down, but not stop, serving pages to bot that doesn't respect robots.txt delay

**URL:** <https://www.ruby-forum.com/t/slow-down-but-not-stop-serving-pages-to-bot-that-doesnt-respect-robots-txt-delay/210591>\
**Category:** NGINX\
**Created:** [August 11, 2011, 5:38pm UTC](https://www.ruby-forum.com/t/slow-down-but-not-stop-serving-pages-to-bot-that-doesnt-respect-robots-txt-delay/210591 "2011-08-11T17:38:00Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![luislavena](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/l/0ea827/32.png) [@luislavena](https://www.ruby-forum.com/u/luislavena)\
**Post date:** [August 11, 2011, 5:38pm UTC](https://www.ruby-forum.com/t/slow-down-but-not-stop-serving-pages-to-bot-that-doesnt-respect-robots-txt-delay/210591/1 "2011-08-11T17:38:00Z")

</div>

Hi all.

I want to differentiate the nginx awesome rate limiting for requests, in  
order to enforce bots to respect my directives (but not to block them):

so i created these limit\_req\_zone:

limit\_req\_zone $binary\_remote\_addr zone=antiddosspider:1m rate=1r/m;  
limit\_req\_zone $binary\_remote\_addr zone=antiddosphp:1m rate=1r/s;  
limit\_req\_zone $binary\_remote\_addr zone=antiddosstatic:1m rate=10r/s;

Now, i ask, is it possible to configure something like this?

if ( $http\_user\_agent ~\* (?:bot|spider) ) {  
limit\_req zone=antiddosspider burst=1;  
}

```
location / {
            limit_req zone=antiddosphp burst=100;
            proxy_pass http://localhost:8181;
            include /etc/nginx/proxy.conf;
        }

```

I don’t know many spider that are crawling my site, but i don’t wanna  
lose the possibility to be indexed if they are not malware! …But 1  
page for minute please 😉

Best regards,  
Stefano

Posted at Nginx Forum:

> **[Slow down, but not stop, serving pages to bot that doesn't respect robots.txt...](https://forum.nginx.org/read.php?2,213704,213704#msg-213704)**
>
> Hi all. I want to differentiate the nginx awesome rate limiting for requests, in order to enforce bots to respect my directives (but not to block them): so i created these limit\_req\_zone: limit\_req\_zone $binary\_remote\_addr zone=antiddosspider:1m...

---

<div class="post-metadata">

**Author:** ![rastrano](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e36b37/32.png) [@rastrano](https://www.ruby-forum.com/u/rastrano)\
**Post date:** [August 11, 2011, 5:56pm UTC](https://www.ruby-forum.com/t/slow-down-but-not-stop-serving-pages-to-bot-that-doesnt-respect-robots-txt-delay/210591/2 "2011-08-11T17:56:33Z")

</div>

with this conf i get:  
Testing nginx configuration: [emerg]: “limit\_req” directive is not  
allowed here in /etc/nginx/sites-enabled/default:52

Posted at Nginx Forum:

> **[Re: Slow down, but not stop, serving pages to bot that doesn't respect...](https://forum.nginx.org/read.php?2,213704,213705#msg-213705)**
>
> Hi all. I want to differentiate the nginx awesome rate limiting for requests, in order to enforce bots to respect my directives (but not to block them): so i created these limit\_req\_zone: limit\_req\_zone $binary\_remote\_addr zone=antiddosspider:1m...

---

<div class="post-metadata">

**Author:** ![rastrano](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e36b37/32.png) [@rastrano](https://www.ruby-forum.com/u/rastrano)\
**Post date:** [August 11, 2011, 11:08pm UTC](https://www.ruby-forum.com/t/slow-down-but-not-stop-serving-pages-to-bot-that-doesnt-respect-robots-txt-delay/210591/3 "2011-08-11T23:08:31Z")

</div>

I dont believe you can use limi\_req within an if statement. You might  
have  
to rewire the if to a different location and then apply the rule:

Example:

if ( $http\_user\_agent ~\* (?:bot|spider) ) {  
error\_page 403 = @bots;  
return 403;  
}

location @bots {  
limit\_req zone=antiddosspider burst=1;  
}

GL.

Rami

---

<div class="post-metadata">

**Author:** ![rastrano](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e36b37/32.png) [@rastrano](https://www.ruby-forum.com/u/rastrano)\
**Post date:** [August 12, 2011, 11:34am UTC](https://www.ruby-forum.com/t/slow-down-but-not-stop-serving-pages-to-bot-that-doesnt-respect-robots-txt-delay/210591/4 "2011-08-12T11:34:46Z")

</div>

Thank you, but i want to serve the real content to the bot not an error  
page.

Is this possible?

Posted at Nginx Forum:

> **[Re: Slow down, but not stop, serving pages to bot that doesn't respect...](https://forum.nginx.org/read.php?2,213704,213727#msg-213727)**
>
> Hi all. I want to differentiate the nginx awesome rate limiting for requests, in order to enforce bots to respect my directives (but not to block them): so i created these limit\_req\_zone: limit\_req\_zone $binary\_remote\_addr zone=antiddosspider:1m...

---

<div class="post-metadata">

**Author:** ![rastrano](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/r/e36b37/32.png) [@rastrano](https://www.ruby-forum.com/u/rastrano)\
**Post date:** [August 12, 2011, 1:33pm UTC](https://www.ruby-forum.com/t/slow-down-but-not-stop-serving-pages-to-bot-that-doesnt-respect-robots-txt-delay/210591/5 "2011-08-12T13:33:44Z")

</div>

What’s this @ thing mean?

@location is a named location. Named locations preserve $uri as it was  
before entering such location. They were introduced in 0.6.6 and can be  
reached only via  
error\_page[http://wiki.nginx.org/NginxHttpCoreModule#error\_page](http://wiki.nginx.org/NginxHttpCoreModule#error_page)  
, post\_action  
[http://wiki.nginx.org/NginxHttpCoreModule#post\_action](http://wiki.nginx.org/NginxHttpCoreModule#post_action)(since  
0.6.26) and try\_files  
[http://wiki.nginx.org/NginxHttpCoreModule#try\_files](http://wiki.nginx.org/NginxHttpCoreModule#try_files) (since  
0.7.27, backported to 0.6.36)."

Essentially, you can still server any content you want using that  
syntax, it  
is just a way to redirect nginx without rewriting the $uri.