An X-Robots-Tag is a directive that deals with the page treatment through a specific HTTP header to a search engine crawler. The instruction is applied for every page level on file addressing instead of being used in robots.txt because the latter works on a directory level for controlling crawling. So, some commonly supported directives, specified using the X-Robots-Tag, are noindex, nofollow, and noarchive. An X-Robots-Tag can go directly in the HTTP response header of a web server or even be included in the section of HTML pages using meta tags. This tag is significant for pages or files like thank-you pages, logins, and PDFs. It is a great tool for webmasters to command how crawling and indexing of their content is done by the search engines.