# Block robots for some urls

**URL:** <https://www.ruby-forum.com/t/block-robots-for-some-urls/191596>\
**Category:** NGINX\
**Created:** [July 15, 2010, 12:52am UTC](https://www.ruby-forum.com/t/block-robots-for-some-urls/191596 "2010-07-15T00:52:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![bvidinli](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/45deac/32.png) [@bvidinli](https://www.ruby-forum.com/u/bvidinli)\
**Post date:** [July 15, 2010, 12:52am UTC](https://www.ruby-forum.com/t/block-robots-for-some-urls/191596/1 "2010-07-15T00:52:41Z")

</div>

Hi,

I want to block some robots such as google, to access some urls.

here is my config:

```
  # robotlarin gezdigi luzumsuz adresleri blokla... zirt pirt

```

giriyorlar sanki bisey varmis gibi  
set $test “”;

```
  if ($request_filename ~* calendar) {
    set $test "blo";
  }

  if ($http_user_agent ~* google\.com ) {
    set $test "${test}ck1";
  }

  if ($test = "block1" ){
    return 444;
  }

```

this is as described in nginx config, for “and” ing some conditions.  
but, it do not work now,

what I am doing wrong ?

---

<div class="post-metadata">

**Author:** ![bvidinli](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/45deac/32.png) [@bvidinli](https://www.ruby-forum.com/u/bvidinli)\
**Post date:** [July 18, 2010, 11:35pm UTC](https://www.ruby-forum.com/t/block-robots-for-some-urls/191596/2 "2010-07-18T23:35:20Z")

</div>

Hello,

Why not using a robots.txt file ? [http://www.robotstxt.org/](http://www.robotstxt.org/)

Posted at Nginx Forum:

> **[Re: block robots for some urls](https://forum.nginx.org/read.php?2,108971,110262#msg-110262)**
>
> Hi, I want to block some robots such as google, to access some urls. here is my config: # robotlarin gezdigi luzumsuz adresleri blokla... zirt pirt giriyorlar sanki bisey varmis gibi set $test ""; if ($reque

---

<div class="post-metadata">

**Author:** ![bvidinli](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/45deac/32.png) [@bvidinli](https://www.ruby-forum.com/u/bvidinli)\
**Post date:** [July 18, 2010, 11:42pm UTC](https://www.ruby-forum.com/t/block-robots-for-some-urls/191596/3 "2010-07-18T23:42:21Z")

</div>

On Sun, Jul 18, 2010 at 22:34, toto2008 [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) wrote:

> Hello,
> 
> Why not using a robots.txt file ? [http://www.robotstxt.org/](http://www.robotstxt.org/)

It won’t block bots that don’t honor it (not would anything block bots  
that present themselves as mozilla).

–  
() ascii-rubanda kampajno - kontraŭ html-a retpoŝto  
/\Â ascii ribbon campaign - against html e-mail

---

<div class="post-metadata">

**Author:** ![bvidinli](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/45deac/32.png) [@bvidinli](https://www.ruby-forum.com/u/bvidinli)\
**Post date:** [July 19, 2010, 1:42am UTC](https://www.ruby-forum.com/t/block-robots-for-some-urls/191596/4 "2010-07-19T01:42:21Z")

</div>

1. In [robotstxt.org](http://robotstxt.org) site, it says:  
“robots can ignore your /robots.txt. Especially malware robots that  
scan the web for security vulnerabilities, and email address  
harvesters used by spammers will pay no attention.”

2. I am expeirencing nginx and learning its rewrite features.  
So, I try to write rewrite statements.

still i have no luck.

2010/7/19 Nuno Magalhães [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid):

---

<div class="post-metadata">

**Author:** ![bvidinli](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/b/45deac/32.png) [@bvidinli](https://www.ruby-forum.com/u/bvidinli)\
**Post date:** [July 19, 2010, 1:51pm UTC](https://www.ruby-forum.com/t/block-robots-for-some-urls/191596/5 "2010-07-19T13:51:07Z")

</div>

On Thu, Jul 15, 2010 at 01:51:36AM +0300, bvidinli wrote:

> ```
> }    
> 
> ```
> 
> this is as described in nginx config, for “and” ing some conditions.  
> but, it do not work now,

location ~ calendar {  
if ($http\_user\_agent ~\* google.com ) {  
return 444;  
}

```
   ...

```

}

…

–  
Igor S.

> **[Igor Sysoev](http://sysoev.ru/en/)**