Here’s a robots.txt file from a company whose software I’m currently evaluating:
User-agent: * Disallow: /cgi/ Disallow: /cgi-bin/ Disallow: /mantis/ Disallow: /forum/ Disallow: /stats/ Disallow: /synk/unreg.html Disallow: /synk/de/unreg.html Disallow: /synk/fr/unreg.html Disallow: /synk/it/unreg.html Disallow: /synk/email.psn Disallow: /synk/help/
This is from a small company whose main product is experiencing solid growth. In fact, they are growing so fast, they are having trouble responding to support e-mails and are consequently requesting that users check the FAQ list and read the forums before sending them e-mail. Keeping that in mind, can you tell what’s wrong with this robots.txt?