# Do controllers get crawed by search engines?

**URL:** <https://www.ruby-forum.com/t/do-controllers-get-crawed-by-search-engines/71460>\
**Category:** Rails\
**Created:** [October 9, 2006, 3:54am UTC](https://www.ruby-forum.com/t/do-controllers-get-crawed-by-search-engines/71460 "2006-10-09T03:54:42Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![koloa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/4bbf92/32.png) [@koloa](https://www.ruby-forum.com/u/koloa)\
**Post date:** [October 9, 2006, 3:54am UTC](https://www.ruby-forum.com/t/do-controllers-get-crawed-by-search-engines/71460/1 "2006-10-09T03:54:42Z")

</div>

if not, i guess one must cache their page and turn common generated  
views into static files?

---

<div class="post-metadata">

**Author:** ![koloa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/4bbf92/32.png) [@koloa](https://www.ruby-forum.com/u/koloa)\
**Post date:** [October 9, 2006, 9:53am UTC](https://www.ruby-forum.com/t/do-controllers-get-crawed-by-search-engines/71460/2 "2006-10-09T09:53:55Z")

</div>

Sure, why shouldn’t they? The search engine doesn’t see any difference  
between a static page and a Rails page.

---

<div class="post-metadata">

**Author:** ![koloa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/4bbf92/32.png) [@koloa](https://www.ruby-forum.com/u/koloa)\
**Post date:** [October 9, 2006, 1:12pm UTC](https://www.ruby-forum.com/t/do-controllers-get-crawed-by-search-engines/71460/3 "2006-10-09T13:12:20Z")

</div>

really? my guess was that since views dont get generated till an action  
is performed in the controller or model, what is there to crawl? lets  
say my controller returns a persons profile from a table…

for example [mydomain.com/doctor/name](http://mydomain.com/doctor/name)

where name is == henry\_gale

a crawler cant get that right because no static page exist for henry  
gale?

Andreas S. wrote:

> Sure, why shouldn’t they? The search engine doesn’t see any difference  
> between a static page and a Rails page.

---

<div class="post-metadata">

**Author:** ![koloa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/4bbf92/32.png) [@koloa](https://www.ruby-forum.com/u/koloa)\
**Post date:** [October 9, 2006, 1:28pm UTC](https://www.ruby-forum.com/t/do-controllers-get-crawed-by-search-engines/71460/4 "2006-10-09T13:28:56Z")

</div>

bbqTree wrote:

> really? my guess was that since views dont get generated till an action  
> is performed in the controller or model, what is there to crawl? lets  
> say my controller returns a persons profile from a table…
> 
> for example [mydomain.com/doctor/name](http://mydomain.com/doctor/name)
> 
> where name is == henry\_gale
> 
> a crawler cant get that right because no static page exist for henry  
> gale?
> 
> Andreas S. wrote:
> 
> > Sure, why shouldn’t they? The search engine doesn’t see any difference  
> > between a static page and a Rails page.

crawler is not different from web browser. It does not work inside of  
your webserver so it does not see the difference between dynamic page  
and static page.

What is the difference between  
[domain.com/person/gale](http://domain.com/person/gale) (which is static)  
[domain.com/person/gale](http://domain.com/person/gale) (which is database generated)

Browser cannot see the difference and crawler also cannot.

Gokhan  
[www.sylow.net](http://www.sylow.net)

---

<div class="post-metadata">

**Author:** ![koloa](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/k/4bbf92/32.png) [@koloa](https://www.ruby-forum.com/u/koloa)\
**Post date:** [February 15, 2007, 4:16pm UTC](https://www.ruby-forum.com/t/do-controllers-get-crawed-by-search-engines/71460/5 "2007-02-15T16:16:20Z")

</div>

I understand what you’re saying… what they wind up crawling is all  
the files (not just the controllers), so they are  
scanning .rhtml, .rb, and all the other files on the server. Turns out  
they’re mostly meaningless, since there’s no ‘dynamic’ content  
generator behind the crawler.

A further question would be: how do you get the search engines to see/  
recognize the _dynamic_ content, rather than just the code that will  
wind up generating the dynamic content? Say I build a webapp that  
stores & displays case studies/book reviews/white papers… is there  
any way (short of using the webapp to actually create static pages)  
that you can get all your dynamic content searched? I’m guessing the  
answer is “no”, but then I’m not an expert at this.

Thanks…jon