what is robots.txt

Robots.txt files have two key roles: Tell search engines where your XML sitemaps are located. Tell search engines which areas of your site should not be crawled. According to Google, robots.txt is a file that tells search engine crawlers which URLs they can access on your site. It is mainly used to manage crawler behavior … Read more