Thursday, August 27, 2009

Restricting Search Engine Robots



Restricting Search Engine Robots 

From indexing your website.


This can be done through robots.txt file. Remember you have to upload the robots.txt file only to the root directory only. The code should looks like below

User-agent: *
Disallow: /


To disallow only Google bot


User-agent: Googlebot
Disallow: /


To disallow only Yahoo bot

User-agent: slurp
Disallow: /


User-agent for MSN is "msnbot". I am not sure weather there is a seperate bot for Bing or not. If some one know it please update me in the comments.

If your site is not indexing in any or all of the search engines.. you have to first look into your robots.txt file

From indexing a directory

User-agent: *
Disallow: /directory-name/


Let all search engines index all of your site with out any restrictions using the following robots.txt syntax



User-agent: *
Disallow:


The list of user agents for major search engines

Google - Googlebot
AltaVista - AVSearch
Yahoo - Slurp
MSN - msnbot


Restrict search engines from following all the links on a web page

 Write the following code in that page meta tags

< meta content="nofollow" name="robots" />

Restrict search engines from indexing a web page

< meta content="noindex" name="robots" />


Restrict search engines from both indexing a page and following links on that page


< meta content="noindex, nofollow" name="robots" />


Note: NOINDEX and NOFOLLOW was no more respected by Bing search engine (MSNbot) and Yahoo search engine (Slurp) - for more information see my post http://web-search-techniques.blogspot.com/2009/07/yahoo-and-bing-ignoring-noindex-and.html


Restrict search engines from using Dmoz listing title


< meta content="NOODP" name="robots" />


Restrict search engines from using Yahoo listing title 

< meta content="NOYDIR" name="robots" />


Restrict search engines from using both Dmoz listing title and Yahoo listing title

  < meta content="NOYDIR, NOODP" name="robots" />


I used the above image only for catching eye.. Don't try to break your head trying to find the relation between this content and that image.. :P

5 comments:

Star Tailor said...

is there any way to restrict SEO from showing our website for particular keyword ? I have removed the particular keyword from the meta tags,but still google list me in the search for that particular keyword.

Search Engineer said...

Could you please let me know that keyword and webpage.. so that i can give you the best suggestion.

Unknown said...

Hi my website is www.tailor.com and we make custom suits, tuxedos etc. We avoid alteration for people who are our customers .But still we get calls for alteration. So I removed "alteration" words from all meta tags and keywords, but still we rank first in New york local search for "alteration" and "alteration nyc".
And I need to know is there any coding or program i can write to edit the robots.txt file . or where can I check code for robots.txt file . I am talking about how to generate it, as I know that. I wan to design my own robots.txt file with new syntax. Is it possible ? like we disallow certain bot or particular url, i want more options for it.

Search Engineer said...

Well it is Google Local search, so i suggest you to delete the current listing and add your business again :)
Be careful if you have rankings for other keywords, do it with caution. All the best :)

Fid"s said...

nice post
great info bro

Post a Comment