A collection of crawlies…
User-Agent: AddThis.com robot tech.support@clearspring.com User-Agent: adidxbot/2.0 (+http://search.msn.com/msnbot.htm) User-Agent: AdsBot-Google (+http://www.google.com/adsbot.html) User-Agent: AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari User-Agent: bitlybot User-Agent: BOT/0.1 (BOT for JCE) User-Agent: ClickTale bot User-Agent: ContextAd Bot 1.0 User-Agent: DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) User-Agent: Flamingo_SearchEngine (+http://www.flamingosearch.com/bot) User-Agent: Fve Nutch Spider/Nutch-1.7 User-Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html) User-Agent: Halebot (Mozilla/5.0 compatible; Halebot/2.1; http://www.tacitknowledge.com/halebot/) << http://biggeye.tacitknowledge.com/laterooms-com/profile User-Agent: Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/) User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html ) User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2) User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; Claritybot) User-Agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/) User-Agent: Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot) User-Agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html) User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) User-Agent: Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/) User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html) User-Agent: Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot) User-Agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot) User-Agent: Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com) User-Agent: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html) User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io) User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) User-Agent: Mozilla/5.0 (compatible; linkCheck) User-Agent: Mozilla/5.0 (compatible; LinkChecker/8.3; +http://wummel.github.com/linkchecker/) User-Agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/) User-Agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots) User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+) User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Selenium Bot) User-Agent: Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php) User-Agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html) User-Agent: Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html) User-Agent: Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot) User-Agent: Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/) User-Agent: Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html) User-Agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/) User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) User-Agent: Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots) User-Agent: Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; ) User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com) User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com) User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 << seen from this ip 162.242.135.149 User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36 User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24 Mozilla/3.5 (Google-HotelAdsVerifier) User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails) User-Agent: Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html) User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Selenium Bot) User-Agent: Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/) User-Agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/) User-Agent: msnbot/2.0b (+http://search.msn.com/msnbot.htm) User-Agent: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) User-Agent: msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm) User-Agent: Pingdom.com_bot_version_1.4_(http://www.pingdom.com/) User-Agent: psbot/0.1 (+http://www.picsearch.com/bot.html) User-Agent: psbot-image (+http://www.picsearch.com/bot.html) User-Agent: QuerySeekerSpider ( http://queryseeker.com/bot.html ) User-Agent: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) User-Agent: SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google User-Agent: ShowyouBot (http://showyou.com/crawler) User-Agent: SMNutchSpider/Nutch-1.7 User-Agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07) User-agent: spider User-Agent: Twitterbot/1.0 User-Agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/) User-Agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/) User-Agent: WIRE/0.22 (Linux; x86_64; Bot,Robot,Spider,Crawler) User-Agent: www.integromedb.org/Crawler User-Agent: Yahoo:LinkExpander:Slingstone User-Agent: Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)