We can spin wheels forever with all the uglies out there, or we can list (identify) the bots to allow.
# Whitelist
User-agent: Adsbot-Google
User-agent: BaiDuSpider
User-agent: Googlebot
User-agent: Googlebot-Image
User-agent: Googlebot-Mobile
User-agent: Googlebot-News
User-agent: MSNBot
User-agent: MSNBot-Media
User-agent: MSNBot-News
User-agent: MSNPTC
User-agent: ScoutJet
User-agent: Slurp
User-agent: Teoma
User-agent: Yahoo-Blogs
User-agent: Yahoo-MMCrawler
User-agent: Yandex
Disallow: /*?
# Blacklist
User-agent: ia_archiver
Disallow: /
User-agent: *
Disallow: /