Spider, Crawlers and Bots oh my!

A collection of crawlies…

User-Agent: AddThis.com robot tech.support@clearspring.com
User-Agent: adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
User-Agent: AdsBot-Google (+http://www.google.com/adsbot.html)
User-Agent: AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
User-Agent: bitlybot
User-Agent: BOT/0.1 (BOT for JCE)
User-Agent: ClickTale bot
User-Agent: ContextAd Bot 1.0
User-Agent: DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
User-Agent: Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
User-Agent: Fve Nutch Spider/Nutch-1.7
User-Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html)
User-Agent: Halebot (Mozilla/5.0 compatible; Halebot/2.1; http://www.tacitknowledge.com/halebot/) << http://biggeye.tacitknowledge.com/laterooms-com/profile
User-Agent: Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; Claritybot)
User-Agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
User-Agent: Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
User-Agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
User-Agent: Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
User-Agent: Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
User-Agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
User-Agent: Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
User-Agent: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
User-Agent: Mozilla/5.0 (compatible; linkCheck)
User-Agent: Mozilla/5.0 (compatible; LinkChecker/8.3; +http://wummel.github.com/linkchecker/)
User-Agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
User-Agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Selenium Bot)
User-Agent: Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
User-Agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
User-Agent: Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
User-Agent: Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
User-Agent: Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
User-Agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
User-Agent: Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
User-Agent: Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 << seen from this ip 162.242.135.149
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24 Mozilla/3.5 (Google-HotelAdsVerifier)
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
User-Agent: Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Selenium Bot)
User-Agent: Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
User-Agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
User-Agent: msnbot/2.0b (+http://search.msn.com/msnbot.htm)
User-Agent: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
User-Agent: msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
User-Agent: Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
User-Agent: psbot/0.1 (+http://www.picsearch.com/bot.html)
User-Agent: psbot-image (+http://www.picsearch.com/bot.html)
User-Agent: QuerySeekerSpider ( http://queryseeker.com/bot.html )
User-Agent: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
User-Agent: SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google
User-Agent: ShowyouBot (http://showyou.com/crawler)
User-Agent: SMNutchSpider/Nutch-1.7
User-Agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
User-agent: spider
User-Agent: Twitterbot/1.0
User-Agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)
User-Agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
User-Agent: WIRE/0.22 (Linux; x86_64; Bot,Robot,Spider,Crawler)
User-Agent: www.integromedb.org/Crawler
User-Agent: Yahoo:LinkExpander:Slingstone
User-Agent: Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)

Please Leave a Reply