User-agent: *
Allow: *

User-agent: LinkWalker
Disallow: /
User-agent: Xenu Link Sleuth
Disallow: /
User-agent: semanticdiscovery/0.2
Disallow: /