If you have a website you really need to have a robots.txt file. It gives search engine spiders specific commands and it is easy to use and easy to maintain. Here is an easy guide to a robots.txt file in five minutes.
There are times when you don’t want a search engine to index a page or a folder on your website. Maybe you have some information you just don’t want to have show up in google. This may include your statistics page, a page of notes, or a dynamic page. And, importantly, if you use google adsense and the search tool that displays search results on your website google mandates you exclude this page from search engines. Which means they mandate you having a robots.txt file.
A robots.txt file is a simple document named robots.txt and saved in the root folder of your website. Search engines see this and follow any commands it contains. Create a simple text document using any word processor program like notepad and put these two lines it:
User-agent: *
Disallow:
The first line tells all spiders to listen up because the following command is for you. The second line means do not index any of the following pages. And it is here you put the url of any pages you don’t want spidered. So if you wanted the spiders to skip your private page it looks like this:
Disallow:/privatepage.htm
If you want the spiders to skip a whole folder you put the url of that folder with a slash like this:
Disallow:/privatefolder/
Simply place this text file in the root folder of your website and you are done. In the future you can add and remove commands easily.
The robots.txt file is a very easy file to write and maintain and it is a very powerful tool that will help you interact successfully with search engines. This disallow command is the simplest and most used command but there are also many other commands you can use and if you have a website it is well worth your time to have a robots.txt file and even to research it a bit further.
Fun Summer Projects for Fathers and Sons
Summertime is the time for getting outside and having some fun. And there are lots of traditional ways to do this but this article looks at some unusual and different ways to enjoy the summer. The focus here is on projects that adults and children or teens can do together and with an eye toward learning and or making things. And although the title of this article says for fathers and sons all the projects and ideas would be great for fathers and daughters too!
The Architectural Marvels of Japan's Medieval Castles
Discover the architectural splendor and historical significance of Japan's medieval castles, which are often overshadowed by their European counterparts. Despite their distinct appearance, these fortresses played a similar protective role during Japan's extensive period of feudal warfare. This article delves into the unique design features and defensive strategies employed by these castles, highlighting their importance in Japanese history.
Epic Fantasy Book Review: Lamentation by Ken Scholes
This is an epic fantasy novel that doesn’t follow the now very over used formula that so many books in the genre follow yet it still stays comfortably within the genre. This makes a refreshing change. Add to this a heap of imagination and a talent for clear writing and you have a great book.
