User-agent: HTTrack Disallow: / User-agent: * Crawl-delay: 5 Disallow: /files/ Disallow: /mr Disallow: /~t