Robots.txt is nothing more than a plain text file that is located in the root folder of a website (for example: https://www.example.com/robots.txt) giving instructions to search engine crawlers, also called bots or spiders, on which parts of the website they are permitted to access and which parts are off-limits. It is an important mechanism for webmasters in controlling how search engines crawl and index their material. Robots.txt issues commands like Allow and Disallow to dictate which URLs or directories should be crawled and which should be barred. This could be helpful in preventing search engines from crawling duplicate content, private areas in a website, or pages that are still being developed. Robots.txt will discourage spiders from accessing particular segments of a website, but not as a security measure: Those with enough reason could always find their way around and into the blocked content if they had the URL.