Google-Googlebot not following-obeying robots.txt!

Dec 20
00:21

2006

S.M.Khurram Quaseem

S.M.Khurram Quaseem

  • Share this article on Facebook
  • Share this article on Twitter
  • Share this article on Linkedin

At last after 2 weeks, today I can see the indexing status of my website inside Google Webmaster Tools as "Googlebot last successfully accessed your home page on Dec 17, 2006."And the results are really attention-grabbing yet disappointing.

mediaimage

Google shows HTTP error for a page inside a directory which has already been banned using robots.txt:

User-agent: *
Disallow : /folder-main/folder-sub

Which clearly states any Search Engine Crawler or spider; including Google; not to crawl over anything inside the folders mentioned,Google-Googlebot not following-obeying robots.txt! Articles which includes /folder-main/folder-subBut for some reason, Google or Googlebot still crawled that folder. Although it doesn’t show pages inside it in the search results but Damn! Why Google / Googlebot is not following the rules??? Do they really follow robots.txt as they say???

Conclusion:Google / Googlebot wants to grab everything within the websites either public or private. I don’t know what the reason is behind but as a result we cannot keep things in private anymore. Extremely Disgusting!!!

Article "tagged" as:

Categories: