Using robot.txt as web security

1. Use the robots.txt file if possible
2. ROBOTS META tag should be used if you can’t create the above file. It’s okay to use both methods if possible.
3. If you know which robots you’re trying to prevent from indexing your pages, a particular search engine for example, go to the source of the robot and remove your page if possible. In other words, many search engines are providing ways for you to remove your URLs from their indexes without having to use any of the above methods.
4. Make your page stand-alone if possible. Meaning, remove links to the page that you’re trying to keep away from robots. More links there are to your page the easier it is for a search engine robot to find your page. If your page is already in search engine indexes, it’s too late to take this preventative step.
5. If you must have absolute protection from robots, password protect those pages in question. Since all other methods are “agreements” that both parties must acknowledge in order for them to work in full, preventing the page from being served is the only way to guarantee that robots will not be able to touch your pages.
6. Robot files are good to use if some one restrict the access of certain files but a simple mistake can cost you a lot. If your site is business site and one put disallow certain directories wrongly, it means the search engines wont crawl your certain areas.

Related posts:

1. Search engines are reluctant with risky results
2. Robot learns to grasp everyday chores
3. Dubious Search Engine Advice from McAfee & Infoworld
4. Dubious Search Engine Advice from McAfee & Infoworld
5. Digg Users Revolt Over AACS Key
6. Remove search bar from IE 7

Related posts brought to you by Yet Another Related Posts Plugin.