Robots.txt files control search engine crawler access to certain areas of your site.

  • You may want to block off certain areas of your site from search engines
  • You may have a newsletter that you publish via that you don’t want to be found online
  • You may not want certain tags or categories appearing in search results

This is all done on the server access level – so you will need either CPANEL or FTP Access to gain access to the Robots.txt File

  • You can do it on the page level, but this would need you to do it via a Robots Meta Tag instead. I prefer the server level for a deeper level of control.