# Google, MSN, Yahoo spiders crawling off my 'database universe'?

**URL:** <https://www.ruby-forum.com/t/google-msn-yahoo-spiders-crawling-off-my-database-universe/140911>\
**Category:** Rails\
**Created:** [June 19, 2008, 7:22pm UTC](https://www.ruby-forum.com/t/google-msn-yahoo-spiders-crawling-off-my-database-universe/140911 "2008-06-19T19:22:12Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [June 19, 2008, 7:22pm UTC](https://www.ruby-forum.com/t/google-msn-yahoo-spiders-crawling-off-my-database-universe/140911/1 "2008-06-19T19:22:12Z")

</div>

I recently figured out how to create a fairly complex Google Sitemap  
file and am happy to share this code with anyone who asks. As I have a  
highly nested database a common URL for me will look something like;

[mywebsite.com is available for purchase - Sedo.com](http://www.MyWebsite.com/parents/23/children/45).

The spiders come to my website and ‘crawl’ along and increase these  
sequences which eventually puts them into;

[www.MyWebsite.com/parents/23/children/46](http://www.MyWebsite.com/parents/23/children/46)

and as this URL has gone off the edge of my database universe, my  
exception\_notification features send me an email.  
Is there a way to put logic somewhere so that if a spider (or person)  
is messing around and requests a URL that isn’t there, that a routine  
kicks in telling them “You’ve gone off the edge of my database  
universe” on a view, and then takes them back to where they were?

Thank you for any thoughts you may offer.  
Kathleen

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [June 20, 2008, 6:54am UTC](https://www.ruby-forum.com/t/google-msn-yahoo-spiders-crawling-off-my-database-universe/140911/2 "2008-06-20T06:54:37Z")

</div>

Well u can setup proper robotx.txt and disallow certain things to be  
accessed from ur site…This is the only way to put restriction on  
crawling.

Hope this helps

Thanks

Dhaval P.  
Software Engineer

> **[RailsHouse : Ruby on Rails Development & Consulting, Waterloo, Canada](https://railshouse.com)**
>
> RailsHouse is Waterloo, Ontario based Ruby on Rails Consultancy firm which provides RoR Development, Migration, Staffing, Maintenace and Support Services

sales(AT)railshouse(DOT)com

---

<div class="post-metadata">

**Author:** ![casper\_the\_ghost](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/c/58956e/32.png) [@casper\_the\_ghost](https://www.ruby-forum.com/u/casper_the_ghost)\
**Post date:** [June 20, 2008, 8:30am UTC](https://www.ruby-forum.com/t/google-msn-yahoo-spiders-crawling-off-my-database-universe/140911/3 "2008-06-20T08:30:11Z")

</div>

removed\_email\_address@domain.invalid wrote:

> I recently figured out how to create a fairly complex Google Sitemap  
> file and am happy to share this code with anyone who asks. As I have a  
> highly nested database a common URL for me will look something like;
> 
> [mywebsite.com is available for purchase - Sedo.com](http://www.MyWebsite.com/parents/23/children/45).
> 
> The spiders come to my website and ‘crawl’ along and increase these  
> sequences which eventually puts them into;
> 
> [www.MyWebsite.com/parents/23/children/46](http://www.MyWebsite.com/parents/23/children/46)
> 
> and as this URL has gone off the edge of my database universe, my  
> exception\_notification features send me an email.  
> Is there a way to put logic somewhere so that if a spider (or person)  
> is messing around and requests a URL that isn’t there, that a routine  
> kicks in telling them “You’ve gone off the edge of my database  
> universe” on a view, and then takes them back to where they were?
> 
> Thank you for any thoughts you may offer.  
> Kathleen

Hi there,

I don’t think that spiders work that way. I thought that they follow  
existing links - rather than looking at the url & guessing what another  
one could be.

If there are no to a page, a spider should not get to it.

Unless the link used to exist, but does not now - ie: you are  
dynamically genereating your urls using table id’s & have removed some  
records from the table.

In this case, the links will still be in the search engines index &  
polled every so often. This is not a bad thing, as the link will  
eventually fall out of the search engines index. You can request removal  
through the webmaster admin tools if this bugs you.

Yahoo’s spider deliberately generates a dummy url to generate a “404 -  
page not found” error to understand how your site handles this. Do don’t  
be too worried that Yahoo has weird links for you site in its index.

Otherwise you can put something like this in your logic.

user\_agent = request.user\_agent.downcase  
if ![‘msnbot’, ‘yahoo! slurp’,‘googlebot’].detect { |b|  
user\_agent.include? b }  
#This request is not from one of the msn, yahoo, or google spiders

# and so process accordingly

end

rgds,

- matt  
[http://www.internetschminternet.com](http://www.internetschminternet.com)