Robots.txt File

What is a robots.txt file?

A file that tells web crawlers which URLs should be accessed or avoided on a website, to prevent overloading servers.