Robots.txt

1 - Assessments & Impacts

The robots.txt is a text file located at the root of a website, allowing you to control which areas an indexing robot is allowed to explore.

To fully understand what a robots.txt is, you need to understand how search engine indexing robots (also known as Web spiders, Web crawlers or Bots) work like Google, Yahoo or Bing. Here are their actions when analyzing a site such as www.mysite.com:

• They start by downloading and analyzing the file http://www.mysite.com/robots.txt.
• They analyze the rules of this file to find out which URLs they are allowed to explore
• If the robots.txt allows it, they browse the site

Robots other than Google, Bing or Yahoo may ignore your robots.txt file, especially malicious robots that crawl sites in the attempt to find vulnerabilities or email addresses.

2 - TO DO

Construction of the robots.txt file

Create and place at at the root of your website the following robots.txt:

NB: Comments can be placed after a "#' and will not be taken into account

Location of robots.txt

The robots.txt file is always at the root of your website on which it will be effective

https://www.cdkeys.com/robots.txt