Restricting Search Engine Robots
From indexing your website.
This can be done through robots.txt file. Remember you have to upload the robots.txt file only to the root directory only. The code should looks like below
User-agent: *
Disallow: /
To disallow only Google bot
User-agent: Googlebot
Disallow: /
To disallow only Yahoo bot
User-agent: slurp
Disallow: /
User-agent for MSN is "msnbot". I am not sure weather there is a seperate bot for Bing or not. If some one know it please update me in the comments.
If your site is not indexing in any or all of the search engines.. you have to first look into your robots.txt file
From indexing a directory
User-agent: *
Disallow: /directory-name/
Let all search engines index all of your site with out any restrictions using the following robots.txt syntax
User-agent: *
Disallow:
The list of user agents for major search engines
Google - Googlebot
AltaVista - AVSearch
Yahoo - Slurp
MSN - msnbot
Restrict search engines from following all the links on a web page
Write the following code in that page meta tags
< meta content="nofollow" name="robots" />
Restrict search engines from indexing a web page
< meta content="noindex" name="robots" />
Restrict search engines from both indexing a page and following links on that page
< meta content="noindex, nofollow" name="robots" />
Note: NOINDEX and NOFOLLOW was no more respected by Bing search engine (MSNbot) and Yahoo search engine (Slurp) - for more information see my post http://web-search-techniques.blogspot.com/2009/07/yahoo-and-bing-ignoring-noindex-and.html
Restrict search engines from using Dmoz listing title
< meta content="NOODP" name="robots" />
Restrict search engines from using Yahoo listing title
< meta content="NOYDIR" name="robots" />
Restrict search engines from using both Dmoz listing title and Yahoo listing title
< meta content="NOYDIR, NOODP" name="robots" />
I used the above image only for catching eye.. Don't try to break your head trying to find the relation between this content and that image.. :P
Posts
5 comments:
is there any way to restrict SEO from showing our website for particular keyword ? I have removed the particular keyword from the meta tags,but still google list me in the search for that particular keyword.
Could you please let me know that keyword and webpage.. so that i can give you the best suggestion.
Hi my website is www.tailor.com and we make custom suits, tuxedos etc. We avoid alteration for people who are our customers .But still we get calls for alteration. So I removed "alteration" words from all meta tags and keywords, but still we rank first in New york local search for "alteration" and "alteration nyc".
And I need to know is there any coding or program i can write to edit the robots.txt file . or where can I check code for robots.txt file . I am talking about how to generate it, as I know that. I wan to design my own robots.txt file with new syntax. Is it possible ? like we disallow certain bot or particular url, i want more options for it.
Well it is Google Local search, so i suggest you to delete the current listing and add your business again :)
Be careful if you have rankings for other keywords, do it with caution. All the best :)
nice post
great info bro
Post a Comment