Short story

The problem was given. At the end of the linked posts there is an advised “reverse cloaking” solution link, but all I achieved after implementing that was the Unreachable network error at Google Webmaster Tools.

Days were gone, but finally I found this thread at WebmasterWorld.com. IncrediBILL suggested a solution which based on the reverse-forward DNS robot validation with the following php script:

// Get the user agent.
$ua = $_SERVER['HTTP_USER_AGENT'];
// Check the user agent to see if it's identifying itself as a search engine bot.
if(strstr($ua, 'msnbot') || stristr($ua, 'Googlebot') || stristr($ua, 'Yahoo! Slurp')){
// The user agent is purporting to be MSN's bot or Google's bot or Yahoo! Slurp.
// If the user agent string is spoofed, we won't find googlebot.com in the host name.
// Get the IP address requesting the page.
$ip = $_SERVER['REMOTE_ADDR'];
// Reverse DNS lookup the IP address to get a hostname.
$hostname = gethostbyaddr($ip);
// Check for '.googlebot.com' and '/search.live.com' in hostname.
if(!preg_match("/\.googlebot\.com$/", $hostname) &&!preg_match("/search\.live\.com$/", $hostname) &&!preg_match("/crawl\.yahoo\.net$/", $hostname)) {
// The host name does not belong to either live.com or googlebot.com.
// Remember the UA already said it is either MSNBot or Googlebot.
$block = TRUE;
header("HTTP/1.0 403 Forbidden");
exit;
} else {
// Now we have a hit that half-passes the check. One last go:
// Forward DNS lookup the hostname to get an IP address.
$real_ip = gethostbyname($hostname);
if($ip!= $real_ip){
$block = TRUE;
header("HTTP/1.0 403 Forbidden");
exit;
} else {
// Real bot.
$block = FALSE;
}
}
}
?>

The original script didn’t validate the Yahoo Slurp bot, but I additonally completed the script with it.

So all you need to do is to download the installation package and implement them according to attached guides. If you are a WordPress I have a really good new for you. Due to mosquito a WordPress plugin is also available with guide as well.

Yes, and don’t forget about the testing. Open your Firefox browser and insert “about:config” without quotes into the adress bar and press enter. Press mouse right click, select new string and add “general.useragent.override” as name and “Googlebot/2.1 (+http://www.googlebot.com/bot.html)” as value. Refresh your site after implementing the defending script/plugin and you will see exactly what the robot will see when come through a proxy site.

Downloads

Installation pack

WordPress plugin