Robots.txt files control search engine crawler access to certain areas of your site.
- You may want to block off certain areas of your site from search engines
- You may have a newsletter that you publish via that you don’t want to be found online
- You may not want certain tags or categories appearing in search results
This is all done on the server access level – so you will need either CPANEL or FTP Access to gain access to the Robots.txt File
- You can do it on the page level, but this would need you to do it via a Robots Meta Tag instead. I prefer the server level for a deeper level of control.
