Robots.txt File Generator ​

A robots.txt file specifies which pages or files on a website can be crawled by search engine crawlers.