You can limit who can crawling your site like,
-Googlebot
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
-MSNBot (bing,microsoft)
msnbot/2.0b (+http://search.msn.com/msnbot.htm)
msnbot/1.1 (+http://search.msn.com/msnbot.htm)
-Yahoosearch (now known as Slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)
You can block other bot using HTACCESS on my article, Or using robots.txt like this
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
Disallow: /admindirectory/
Disallow: /cache/
Disallow: /otherdirectory/
User-agent: Slurp
Allow: /
Disallow: /admindirectory/
Disallow: /cache/
Disallow: /otherdirectory/
User-agent: msnbot
Allow: /
Disallow: /admindirectory/
Disallow: /cache/
Disallow: /otherdirectory/
fyna 
0 comments:
Post a Comment
We are not responsible for comments expressed within this site. It is the account holder's personal views and all risks of comments posted his own account owner's responsibility. Comments wisely as it showed your maturity.
Click to see the code!
To insert emoticon you must added at least one space before the code.