Redirect for Search Engines (googlebot et.al.)

I want to use a (slower with limited amount of threads and
connections) backend server just for search engines which will not
have an impact on the users of the site (as they are using a
different upstream).

Will something like this work ?

location / {
if {$http_user_agent ~ Googlebot} {
proxy_pass http://search_engines;
}
}

Furthermore, is there a way to make this global for the server across
all locations (as I have 25 of them, so I would have to add the code
on each). Last but not least can I provide a variable with a list of
user_agents (as there are a ton of search engines)?

Thanks in advance and sorry if this has been asked before and I am
unable to find the answer using google.

Hi,

On Tue, 2007-10-02 at 06:05 +0200, Malte Sussdorff wrote:

on each). Last but not least can I provide a variable with a list of
user_agents (as there are a ton of search engines)?

We are using the following snippet (but not to solve the same problem as
you) in a file we are including in each server {} block.

    if ( $http_user_agent ~* \b(:?googlebot|webcrawler|<add here the 

other user_agents>)\b ) {
rewrite ^(.*)$ /friendly-spider/$1;
}

    location ^~ /friendly-spider/ {
            rewrite ^/friendly-spider/(.*)$ $1 last;
      ... do what you want here...
    }

I guess you can use a proxy_pass in the location section.
There are certainly more efficient ways to do that, but that one is
working.

This forum is not affiliated to the Ruby language, Ruby on Rails framework, nor any Ruby applications discussed here.

| Privacy Policy | Terms of Service | Remote Ruby Jobs