|
|
|
|
|
|
|
|
|
|
The Webmasters Five Minute Guide to the robots.txt filesIf you have a website you really need to have a robots.txt file. It gives search engine spiders specific commands and it is easy to use and easy to maintain. Here is an easy guide to a robots.txt file in five minutes. There are times when you don’t want a search engine to index a page or a folder on your website. Maybe you have some information you just don’t want to have show up in google. This may include your statistics page, a page of notes, or a dynamic page. And, importantly A robots.txt file is a simple document named robots.txt and saved in the root folder of your website. Search engines see this and follow any commands it contains. Create a simple text document using any word processor program like notepad and put these two lines it: User-agent: * Disallow: The first line tells all spiders to listen up because the following command is for you. The second line means do not index any of the following pages. And it is here you put the url of any pages you don’t want spidered. So if you wanted the spiders to skip your private page it looks like this: Disallow:/privatepage.htm If you want the spiders to skip a whole folder you put the url of that folder with a slash like this: Disallow:/privatefolder/ Simply place this text file in the root folder of your website and you are done. In the future you can add and remove commands easily. The robots.txt file is a very easy file to write and maintain and it is a very powerful tool that will help you interact successfully with search engines. This disallow command is the simplest and most used command but there are also many other commands you can use and if you have a website it is well worth your time to have a robots.txt file and even to research it a bit further. Article Tags: Don’t Want, Search Engines, Robotstxt File Source: Free Articles from ArticlesFactory.com
ABOUT THE AUTHORFor more interesting insights into being a creative webmaster and making your website work for you visit the authors site at:The Creative Webmaster – Forging the Iron of Creativity on the Anvil of a Website For more practical advice on how to earn money with your small website visit the authors tutorial website at: Earning Money with your small website 
|
|||||||||||||||||||||||||||||||||||||||||
Partners
|
