Redirect for Search Engines (googlebot et.al.)

Server NGINX
1

I want to use a (slower with limited amount of threads and
connections) backend server just for search engines which will not
have an impact on the users of the site (as they are using a
different upstream).

Will something like this work ?

location / {
if {$http_user_agent ~ Googlebot} {
proxy_pass http://search_engines;
}
}

Furthermore, is there a way to make this global for the server across
all locations (as I have 25 of them, so I would have to add the code
on each). Last but not least can I provide a variable with a list of
user_agents (as there are a ton of search engines)?

Thanks in advance and sorry if this has been asked before and I am
unable to find the answer using google.

2

Hi,

On Tue, 2007-10-02 at 06:05 +0200, Malte Sussdorff wrote:

on each). Last but not least can I provide a variable with a list of
user_agents (as there are a ton of search engines)?

We are using the following snippet (but not to solve the same problem as
you) in a file we are including in each server {} block.

    if ( $http_user_agent ~* \b(:?googlebot|webcrawler|<add here the

other user_agents>)\b ) {
rewrite ^(.*)$ /friendly-spider/$1;
}

    location ^~ /friendly-spider/ {
            rewrite ^/friendly-spider/(.*)$ $1 last;
      ... do what you want here...
    }

I guess you can use a proxy_pass in the location section.
There are certainly more efficient ways to do that, but that one is
working.