Restricting Search Engine Robots
From indexing your website.
This can be done through robots.txt file. Remember you have to upload the robots.txt file only to the root directory only. The code should looks like below
User-agent: *
Disallow: /
To disallow only Google bot
User-agent: Googlebot
Disallow: /
To disallow only Yahoo bot
User-agent: slurp
Disallow: /
User-agent for MSN is "msnbot". I am not sure weather there is a seperate bot for Bing or not. If some one know it please update me in the comments.
If your site is not indexing in any or all of the search engines.. you have to first look into your robots.txt file
From indexing a directory
User-agent: *
Disallow: /directory-name/
Let all search engines index all of your site with out any restrictions using the following robots.txt syntax
User-agent: *
Disallow:
The list of user agents for major search engines
Google - Googlebot
AltaVista - AVSearch
Yahoo - Slurp
MSN - msnbot
Restrict search engines from following all the links on a web page
Write the following code in that page meta tags
< meta content="nofollow" name="robots" />
Restrict search engines from indexing a web page
< meta content="noindex" name="robots" />
Restrict search engines from both indexing a page and following links on that page
< meta content="noindex, nofollow" name="robots" />
Note: NOINDEX and NOFOLLOW was no more respected by Bing search engine (MSNbot) and Yahoo search engine (Slurp) - for more information see my post http://web-search-techniques.blogspot.com/2009/07/yahoo-and-bing-ignoring-noindex-and.html
Restrict search engines from using Dmoz listing title
< meta content="NOODP" name="robots" />
Restrict search engines from using Yahoo listing title
< meta content="NOYDIR" name="robots" />
Restrict search engines from using both Dmoz listing title and Yahoo listing title
< meta content="NOYDIR, NOODP" name="robots" />
I used the above image only for catching eye.. Don't try to break your head trying to find the relation between this content and that image.. :P
Posts
