Last week a Hungarian guy asked me if I could develop something effective solution against bad robots so some days ago I started to execute a fast survey on topic and found many solution, but most of them based on certain host placed into .htaccess, and none of them was automatic, so the challenge was given.

Some days ago I started a little survey on topic and found a lot of htaccess rules, where certain hosts were rejected via .htaccess, but they were not automatic, so the challange was given. The most useful site I found was this resource which let me know the basic attitude of bad robots to the robots.txt files. They ignore the specified restrictions.

1. Open your existing robots.txt file or upload one and place the following lines into it

User-agent: *
Disallow: /core

The name of the restricted folder is not important, but would be great if the humanoid atteckers would find it enough attractive as well since this folder will be the live-bait.

2. Create the folder on your hosting space which is specified in the robots.txt file, in my example this is called core and upload an index.php file with the following content:

<?php
$ip = $_SERVER[”REMOTE_ADDR”];
$logfile = ‘bannolnilog.txt’;
//collect the IP adresses or something else into the logfile
$fp = fopen($logfile, ‘a’);
fputs($fp, “$ip
“);
fputs($fp, ” “);
fclose($fp);
echo “your IP was logged for security reasons and your visit is now over”;
?>

3. As you may see in the code I defined a $logfile where the IP adresses will be collected and stored hence we need to upload to the same (core) folder a blank txt file called bannolnilog.txt (chmod 644).

4. We need to upload one more php file which will check if the visitor is bannished whenever a page is requested, I named this file validator.php and its content is the following.

<?php
$ip = $_SERVER[”REMOTE_ADDR”];
$logfile = ‘bannolnilog.txt’;
$target = file(dirname(__FILE__). “/core/bannolnilog.txt”);
foreach($target as $item){
$item = trim($item);
if(stristr($ip, $item)){
header(”HTTP/1.0 403 Forbidden”);
exit;
}
}
?>

5. As final step you need to insert this line into the very front of your script header or index file, the point is that this is how the script must started whenever a page is requested

<?php require “/you/need/to/insert/the/path/here/validator.php”;?>

Note: You may truncate the logfile deleting the collected IPs, and please take into consideration that WordPress is make quotation marks display a bit odd, so you may want to double check the syntax of the code.
I warrant nothing, but works very well at one of my sites.

Have a nice further day!

Short story

The problem was given. At the end of the linked posts there is an advised “reverse cloaking” solution link, but all I achieved after implementing that was the Unreachable network error at Google Webmaster Tools.

Days were gone, but finally I found this thread at WebmasterWorld.com. IncrediBILL suggested a solution which based on the reverse-forward DNS robot validation with the following php script:

// Get the user agent.
$ua = $_SERVER['HTTP_USER_AGENT'];
// Check the user agent to see if it's identifying itself as a search engine bot.
if(strstr($ua, 'msnbot') || stristr($ua, 'Googlebot') || stristr($ua, 'Yahoo! Slurp')){
// The user agent is purporting to be MSN's bot or Google's bot or Yahoo! Slurp.
// If the user agent string is spoofed, we won't find googlebot.com in the host name.
// Get the IP address requesting the page.
$ip = $_SERVER['REMOTE_ADDR'];
// Reverse DNS lookup the IP address to get a hostname.
$hostname = gethostbyaddr($ip);
// Check for '.googlebot.com' and '/search.live.com' in hostname.
if(!preg_match("/\.googlebot\.com$/", $hostname) &&!preg_match("/search\.live\.com$/", $hostname) &&!preg_match("/crawl\.yahoo\.net$/", $hostname)) {
// The host name does not belong to either live.com or googlebot.com.
// Remember the UA already said it is either MSNBot or Googlebot.
$block = TRUE;
header("HTTP/1.0 403 Forbidden");
exit;
} else {
// Now we have a hit that half-passes the check. One last go:
// Forward DNS lookup the hostname to get an IP address.
$real_ip = gethostbyname($hostname);
if($ip!= $real_ip){
$block = TRUE;
header("HTTP/1.0 403 Forbidden");
exit;
} else {
// Real bot.
$block = FALSE;
}
}
}
?>

The original script didn’t validate the Yahoo Slurp bot, but I additonally completed the script with it.

So all you need to do is to download the installation package and implement them according to attached guides. If you are a WordPress I have a really good new for you. Due to mosquito a WordPress plugin is also available with guide as well.

Yes, and don’t forget about the testing. Open your Firefox browser and insert “about:config” without quotes into the adress bar and press enter. Press mouse right click, select new string and add “general.useragent.override” as name and “Googlebot/2.1 (+http://www.googlebot.com/bot.html)” as value. Refresh your site after implementing the defending script/plugin and you will see exactly what the robot will see when come through a proxy site.

Downloads

Installation pack

WordPress plugin

Not very long ago some really smart guy helped me to understand that not the link building is to most importan part when optimizing website, but the keyword research. The situation is very similar to a guy who holds a gun in his hands and keeps on shooting without any specified targets. I guess you don’t want to be this guy.

Let’s see what kind of tools are freely available in order to help us to select our targets!

I will start the list with the widely used Google tools.

• The AdWords Keyword Tool was developed therefore to easy the advertising clients to select the suitable expressions for their needs.
• Google has another tool which may help you estimate searching volume and this is called Google trends.

My main problem with the previously mentioned tools is the not numeric value. The situation is a bit different concerning Google Trends since some days ago a very nice guy shared a Youtube video which helps to exploit the information of the Google Trend charts.

Before I share my personal favourite I would like to list two tools both provide numeric searching volumes.

• The first one is the Free Keyword Suggestion Tool from Wordtracker, where the numbers reflects daily search volumes.
• The other one is the Free Search Term Suggestion Tool by KeywordDiscovery.com. The numeric data next to the related keyphrase shows the number of times that each keyword or phrase has been searched for over the last 12 months.
  
  And now please let me introduce my personal favourite the SEO Book Keyword Suggestion Tool. I suppose that this tool provides the most accurate estimations concerning the three major search engines. Beside the monthly volumes you will see some other links, that you may find very useful. For instance at the right outside coloumn you will see the Quintura link. In my opinion that is a very powerful suggestion tool as well, since it shows the targetted keyphrases of the major competitors.

  Summarizing I would say you can’t live without keyword research at competetive markets, but you shouldn’t trust the information provided by the different tools blindly. Test them, test all of them, and check your log files periodically.

  If you also have a favourite tool or any thought let me know!

  Thank you,
  Bagi Zoltán

Web Directories need no introduction, however, its recent offshoot, bid for position or bid for placement web directories are so new that, they need to be introduced.

Bid for Position directories evolved from web directory customers desire to be listed in first position on web directories and sometime on the homepage of a directory.
This desire was meet when some clever webmaster created hybrid web directory, which allows customers to place link as high as they like on the listing pages or even the homepage of a bid for position directory. The position of a listing is determined by how much money you are willing to pay for your listing, the more you pay the higher your site is listed.

Because Bid directories are still growing and evolving, script for creating bid directories are just getting on the market. This article examines once of the few bid for position directory script currently on the market; Link Bid Script or LBS for short.

In 2006 when bid for position website first emerged on the directory scene, the team behind Haabaa Web Directory developed link Bid Script, originally to power A Pound a link directory, this meet with some success, which was then extended to creating Link Bid Script as a stand a long bidding directory script.

Link Bid Script was created using PHP scripting language, with popular open source database, MySQL as the back end. This makes the LBS a light but fast script to install and use.

One of the features that that distinguished Link Bid Script from other PHP based bidding directory script is that it combined the classic link bid script alphabetic listing with web directory style category listing, giving directory owner and directory users the best of both worlds.
The alphabetic listing make search by alphabets easy and the web directory style categories makes links listed on the bid for position directory more search engine friend.

To get more information about link bid script at its website Link Bid Script, support for LBS can be accessed at Link Bid Script support.

Blog post by Temi Odurinde
Respected owner of Haabaa premium webdirectory.

With this post I would like to draw attention on some brilliant WordPress plugins which could make your ranking easier concerning search engines.

Providing an xml sitemap will increase the visibility of your site in Google’s index. You don’t need to use online or desktop sitemap builder applications anymore, the google sitemap generator plugin will do the job instead of you.

A static sitemap page improves the PageRank flow among the inner pages and helps your visiors to navigate in your content.

The Category Tagging plugin has an excellent feature. It make the most relevant older posts display at the end of the new publications. Besides improving the PageRank flow it makes your visitors stay longer at your site by providing suitable content for their interest.

SEO can be seperated into two basic parts. The firsts is the onpage optimization and the second one is the link popularity increase. With the next two plugins you will be able to manage both much more succesfully:

• The seo title tag plugin let you customize your title tag all around your blog.
• Jim Westergren’s link to me textbox plugin displays a html code of a link points back to your posts in a textarea so your visitor can easily grab it and insert it into their pages if they like your thoughts. For live demo you need to scroll down a little and you will see what is it all about.

If you also have a favourite let me know!

Thank you,
Bagi Zoltán

I think we all agree that we live the era of the off page factors and the keyword density is already the part of the past. The relevance and the importance mostly determined by backlinks. If we want to achieve nice rankings we need to go for relevant backlinks. Many of us can’t afford the paid directory submissions and constanly look for free links. This post is dedicated to them.

Some month ago one of my friends Jumpenjuhosaphat published a blog post about a SEO study which threw light on the factors which determine the positive effect of the backlinks. Beside the quality (PageRank), the relevance and the keyword density in the title tag of the referrer page are the most important aspects.

How can we get relevant backlinks for free? Why not use google itself? I think about the excellent search operators such as inurl, intitle and inanchor in the following form:

• intitle:keyword “add link free” keyword
• intitle:keyword “add site free” keyword
• intitle:keyword “add URL free” keyword
• intitle:keyword “add website free” keyword
• intitle:keyword “submit link free” keyword
• intitle:keyword “submit site free” keyword

After checking the returned results you may want to substitute the intitle operator with inurl and inanchor as well.

Remember that finding relevant free links is not an easy thing, but i hope you agree that it worths a try.

Thank you,
Bagi Zoltán

My answer is don’t do that!

In my opinion submitting a webpage is the last appeal of a bitter webmaster. This isn’t the natural way to get recognised.

The spiders of the major search engines continuously crawl the content of the Internet across links. These connections among websites pass forward to different things, which are the most important seo factors, the relevance and the importance.

If I would link to your site using the “car rental” phrase as anchor text i would increase the relevance of your site for the mentioned expression and your site would share in the PageRank of my linking webpage.

Don’t waste your time with search engine submission. If you have serious plans concerning internet marketing, go for some backlinks and your site will start the competition with some relevance and importance as well.

If you have any opinion let me know!

Thank you,
Bagi Zoltán