A Robots.txt file is a text file that defines files on your site which are not to be scanned by web crawlers or spiders. It prevents indexing of the file by search engine crawlers so that it can keep your content private, or it can instructs crawlers to bypass accessing a file altogether.
You should have a Robots.txt file for:
- Blocking access to sensitive files (like those containing passwords or proprietary information) so everyone but the intended recipient can't see them
- Specifying portions of the site which should never be indexed, such as directories you use for temporary pages
- Preventing search engine crawlers from indexing content you don't want indexed, like product pages with sensitive prices
Robots.txt files are a convenient way to manage where your site's content is accessible to search engines and web crawlers.
Google Search Console comes with a Robots.txt file generator tool that will help you create your website's robots.txt file for use in the Google indexing system.
Before the internet existed, a robots.txt file was used to create an exclusion protocol for search engine bots and spiders. This file would prevent search engine crawlers from crawling parts of your site that you don't want them accessing.
Nowadays, a robots.txt is still very useful for preventing crawlers from indexing certain pages and directories on your site, but this is no longer the only use for it. You can also use the robots.txt file to create other directives that you want all of your crawlers to adhere to such as your indexing preference or how many times a crawled page can be indexed before it's purged from the web cache
The purpose of this article is to show how you can manually create a robots.txt file and insert into your website’