Translations:Help:Robots.txt/18/en

[Required, one or more per group] The directive specifies the name of the automatic client known as search engine crawler that the rule applies to. This is the first line for any rule group. Google user agent names are listed in the [$googlelink1 Google list of user agents]. Using an asterisk will match all crawlers except the various AdsBot crawlers, which must be named explicitly.