User-agent: Mediapartners-Google* Disallow: User-Agent: Googlebot #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /stories/*/comments$ Disallow: /stories/*/full_content$ User-Agent: Cogger #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax # Alexa Archver, allow them User-Agent: ia_archiver Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax # Internet Archives open source crawler # Has gone nuts on us before. User-Agent: os-heritrix Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-agent: YahooFeedSeeker Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-agent: BlogPulse Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-agent: kinjabot Disallow: /comment Disallow: /comments Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: Teoma #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: Sphere Scout #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: msnbot #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: archive.org_bot #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search # topsy.com's bot User-Agent: Butterfly #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: Yandex #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: ScoutJet #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: cdlwas_bot #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: dk-bot #Disallow: / Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: bingbot Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax Disallow: /stories/*/comments$ Disallow: /stories/*/full_content$ Disallow: /news Crawl-delay: 10 User-Agent: special_archiver Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-Agent: Diffbot Disallow: /comments Disallow: /comment Disallow: /user Disallow: /poll Disallow: /print Disallow: /search Disallow: /part Disallow: /ajax User-agent: Twitterbot Disallow: User-Agent: * Disallow: /