Google-Googlebot not following-obeying robots.txt!

SE Optimization Articles | December 19, 2006

At last after 2 weeks, today I can see the indexing status of my website inside Google Webmaster Tools as "Googlebot last successfully accessed your home page on Dec 17, 2006."And the results are really attention-grabbing yet disappointing.

Google shows HTTP error for a page inside a directory which has already been banned using robots.txt:

User-agent: *
Disallow : /folder-main/folder-sub

Which clearly states any Search Engine Crawler or spider; including Google; not to crawl over anything inside the folders mentioned, which includes /folder-main/folder-subBut for some reason, Google or Googlebot still crawled that folder. Although it doesn’t show pages inside it in the search results but Damn! Why Google / Googlebot is not following the rules??? Do they really follow robots.txt as they say???

Conclusion:Google / Googlebot wants to grab everything within the websites either public or private. I don’t know what the reason is behind but as a result we cannot keep things in private anymore. Extremely Disgusting!!!

Source: Free Articles from ArticlesFactory.com

ABOUT THE AUTHOR

S.M.Khurram Quaseem - CEO, SK Web Technologies.

S.M.Khurram Quaseem has been involved with Internet since 1996. He and his team are now actively engaged in online website, portal, intranet & extranet development specializing in SEO, SEM and SEP with major focus on Google, Yahoo and MSN - at seo-google-search-engine-optimization-experts :: SK Web Technologies