Translations:Help:Robots.txt/6/en

Robots.txt supports many indexing related keywords. This includes:
 * [Required, one or more per group] The directive specifies the name of the automatic client known as search engine crawler that the rule applies to. This is the first line for any rule group. Google user agent names are listed in the [$googlelink1 Google list of user agents]. Using an asterisk will match all crawlers except the various AdsBot crawlers, which must be named explicitly.
 * [At least one or more  or   entries per rule] A directory or page, relative to the root domain, that you don't want the user agent to crawl. If the rule refers to a page, it should be the full page name as shown in the browser; if it refers to a directory, it should end in a   mark.
 * [At least one or more  or   entries per rule] A directory or page, relative to the root domain, that may be crawled by the user agent just mentioned. This is used to override a   directive to allow crawling of a subdirectory or page in a disallowed directory. For a single page, the full page name as shown in the browser should be specified. In case of a directory, the rule should end in a   mark.