Use the robots.txt file if possible ROBOTS META tag should be used if you can't create the above file. It's okay to use both methods if possible. If you know which robots you're trying to prevent from indexing your pages, a particular search engine for example, go to the source of the ...