# Non-Cache Handling for Bots **URL:** \ **Category:** Mongrel\ **Created:** [June 11, 2009, 12:54am UTC](https://www.ruby-forum.com/t/non-cache-handling-for-bots/169990 "2009-06-11T00:54:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mperham](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@mperham](https://www.ruby-forum.com/u/mperham)\ **Post date:** [June 11, 2009, 12:54am UTC](https://www.ruby-forum.com/t/non-cache-handling-for-bots/169990/1 "2009-06-11T00:54:41Z")
Hello, when a bot visits our page, we want to create a response that is different than the response for a human. In particular we want to limit the hierarchies of menus so that the bot doesnt think there are too may tags/keywords for the site (and thus they dont get indexed) as well we do not display ads for bots. We cache most of our pages using standard rails caching. The problem is that when a bot visits the site, it will get the standard cached page with the incorrect menus. We want to work around this so that bots do not get cached pages. (Yes, it is good if the bots get a bot-specific cached page, but lets keep it simple so far). We did try numerous the following: 1. using apache rewrite to add /robot to the URL. then mongrel will never find the cached page on disk. The problem here is that all the links in the page have /robot in front of them. So, having apache add /robot to the URI results in the PATH\_INFO as-seen by rails to have /robot in it. Another problem was that we effectivly duplicated the set of rules in routes.rb for the cases with /robot as a prefix. We have 2 ideas: 1. when apache detects a bot, send the request to a non-caching webserver. Does anyonme know one of these? 2. I edited the mongrel source code in mongrel-1.1.4/lib/mongrel/rails.rb and added this kind of thing to the process method: do\_not\_cache = KNOWN\_ROBOT\_AGENTS.detect{ |b| user\_agent.downcase.include?(b) } if user\_agent And then used that variable in the tests for @files.can\_serve(…) This works but we still want mongrel to serve static files as cached (so the rules above can take care of this too, it just gets more complicated to check for /stylesheets, /.images etc.). * * * Question: is there a way to plug-in our own logic into the mogrel process of handling a request? And/or can we set up a specific mongrel to never cache (are there options for this)? Any ideas are appreciated, Mike ---
**Author:** ![mperham](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@mperham](https://www.ruby-forum.com/u/mperham)\ **Post date:** [June 11, 2009, 3:24am UTC](https://www.ruby-forum.com/t/non-cache-handling-for-bots/169990/2 "2009-06-11T03:24:28Z")
What kind of caches are you talking about? Are these full page caches? The kind that get stored into /public? My question is, can you instead of adding /robot to the URL when apache finds the robot, can you instead change Apache’s DocumentRoot? It seems to be that this would prevent apache from finding the cached page. Also, if you actually point to another copy of your /public, you could get the normal static pages… I think perhaps since you are talking about changing mongrel’s caching behaviour that you aren’t talking about the page caches that get stored into /public. (Well, I’m rusty on terminology here) – Michael R. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) Director – Consumer Desktop Development, Simtone Corporation, Ottawa, Canada Personal: [Michael Richardson's Directory](http://www.sandelman.ca/mcr/) SIMtone Corporation fundamentally transforms computing into simple, secure, and very low-cost network-provisioned services pervasively accessible by everyone. Learn more at [www.simtone.net](http://www.simtone.net) and [www.SIMtoneVDU.com](http://www.SIMtoneVDU.com) ---
**Author:** ![mperham](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@mperham](https://www.ruby-forum.com/u/mperham)\ **Post date:** [June 11, 2009, 3:49am UTC](https://www.ruby-forum.com/t/non-cache-handling-for-bots/169990/3 "2009-06-11T03:49:04Z")
I am talking about Rails standard page-caching mechanism. Rails by default puts full pages into public/… and if mongrel sees them there, it serves them (without running rails dispatch et al). This is fine for normal but not good for the bot user agents. Here is a new solution: 1. set rails to cache in public/cache 2. Use Apache rewrite to serve these files directly (if found) 3. If not found, pass to mongrel which will not find the cached files either since MONGREL ONLY LOOKS IN public for cached files. Mongrel does not honor the config.action\_controller.page\_cache\_directory rails setting 4. Rails processes the file and puts it into public/cache/… …on the next request, apache serves from cache. I am working on the reqwrite rules etc. for this. Mike ---
**Author:** ![mperham](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@mperham](https://www.ruby-forum.com/u/mperham)\ **Post date:** [June 11, 2009, 4:22am UTC](https://www.ruby-forum.com/t/non-cache-handling-for-bots/169990/4 "2009-06-11T04:22:24Z")
> > > > > “Mike” == Mike P. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) writes: > > > > > Mike\> I am talking about Rails standard page-caching mechanism. > > > > > Rails by default > > > > > Mike\> puts full pages into public/… and if mongrel sees them > > > > > there, it serves > > > > > Mike\> them (without running rails dispatch et al). This is fine for > > > > > normal but not > > > > > Mike\> good for the bot user agents. right, so that’s what I thought you were talking about. Only, it’s not mongrel that serves up the pages, but Apache, usually. In your apache config, you have something like: ``` # Rewrite all non-static requests to cluster RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f RewriteRule ^/(.*)$ balancer://spartan_cluster%{REQUEST_URI} ``` [P,QSA,L] which basically serves up any files found in /public, otherwise, punts to the mongrel. I thought that rails put the files directly there for apache to use/see. (there are caveats if your mongrel and apache do not share the same file system, such as because they are on different machines) If you are telling me that actually mongrel does this, it’s news to me. ``` Mike> Here is a new solution: Mike> 1) set rails to cache in public/cache Mike> 2) Use Apache rewrite to serve these files directly (if found) Mike> 3) If not found, pass to mongrel which will not find the ``` cached files either Mike\> since MONGREL ONLY LOOKS IN public for cached files. Mongrel does not honor Mike\> the config.action\_controller.page\_cache\_directory rails setting Mike\> 4) Rails processes the file and puts it into public/cache/… ``` Mike> ...on the next request, apache serves from cache. Mike> I am working on the reqwrite rules etc. for this. ``` So, basically have apache pick a different cache location when it sees a robot. – Michael R. [removed\_email\_address@domain.invalid](mailto:removed_email_address@domain.invalid) Director – Consumer Desktop Development, Simtone Corporation, Ottawa, Canada Personal: [Michael Richardson's Directory](http://www.sandelman.ca/mcr/) SIMtone Corporation fundamentally transforms computing into simple, secure, and very low-cost network-provisioned services pervasively accessible by everyone. Learn more at [www.simtone.net](http://www.simtone.net) and [www.SIMtoneVDU.com](http://www.SIMtoneVDU.com) ---
**Author:** ![mperham](https://www.ruby-forum.com/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@mperham](https://www.ruby-forum.com/u/mperham)\ **Post date:** [June 11, 2009, 4:45am UTC](https://www.ruby-forum.com/t/non-cache-handling-for-bots/169990/5 "2009-06-11T04:45:12Z")
“Serving up different results based on user agent may cause your site to be perceived as deceptive and removed from the Google index.” > **[Spam Policies for Google Web Search | Google Search Central  | ...](https://developers.google.com/search/docs/essentials/spam-policies?hl=en&visit_id=638238841317985139-2693624551&rd=1#cloaking)** > > The spam policies detail the behaviors and tactics that can lead to a page or an entire site being ranked lower or completely omitted from Google Search.