Spider, Crawlers and Bots oh my!

A collection of crawlies…

User-Agent: AddThis.com robot tech.support@clearspring.com
User-Agent: adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
User-Agent: AdsBot-Google (+http://www.google.com/adsbot.html)
User-Agent: AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
User-Agent: bitlybot
User-Agent: BOT/0.1 (BOT for JCE)
User-Agent: ClickTale bot
User-Agent: ContextAd Bot 1.0
User-Agent: DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
User-Agent: Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
User-Agent: Fve Nutch Spider/Nutch-1.7
User-Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html)
User-Agent: Halebot (Mozilla/5.0 compatible; Halebot/2.1; http://www.tacitknowledge.com/halebot/) << http://biggeye.tacitknowledge.com/laterooms-com/profile
User-Agent: Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; Claritybot)
User-Agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
User-Agent: Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
User-Agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
User-Agent: Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
User-Agent: Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
User-Agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
User-Agent: Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
User-Agent: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
User-Agent: Mozilla/5.0 (compatible; linkCheck)
User-Agent: Mozilla/5.0 (compatible; LinkChecker/8.3; +http://wummel.github.com/linkchecker/)
User-Agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
User-Agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Selenium Bot)
User-Agent: Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
User-Agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
User-Agent: Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
User-Agent: Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
User-Agent: Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
User-Agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
User-Agent: Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
User-Agent: Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 << seen from this ip 162.242.135.149
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24 Mozilla/3.5 (Google-HotelAdsVerifier)
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
User-Agent: Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Selenium Bot)
User-Agent: Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
User-Agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
User-Agent: msnbot/2.0b (+http://search.msn.com/msnbot.htm)
User-Agent: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
User-Agent: msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
User-Agent: Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
User-Agent: psbot/0.1 (+http://www.picsearch.com/bot.html)
User-Agent: psbot-image (+http://www.picsearch.com/bot.html)
User-Agent: QuerySeekerSpider ( http://queryseeker.com/bot.html )
User-Agent: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
User-Agent: SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google
User-Agent: ShowyouBot (http://showyou.com/crawler)
User-Agent: SMNutchSpider/Nutch-1.7
User-Agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
User-agent: spider
User-Agent: Twitterbot/1.0
User-Agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)
User-Agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
User-Agent: WIRE/0.22 (Linux; x86_64; Bot,Robot,Spider,Crawler)
User-Agent: www.integromedb.org/Crawler
User-Agent: Yahoo:LinkExpander:Slingstone
User-Agent: Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)

Thought of the day.

Conway’s law is an adage named after computer programmer Melvin Conway, who introduced us to the idea back in 1968, yes they had computers back then kids. It states that:
“organisations which design systems … are constrained to produce designs which are copies of the communication structures of these organisations”.

What do we do about it?

DevOps. While devops is a portmanteau of development and operations, it is not another team. The last thing our IT family needs is yet another functional silo.

DevOps is a set of protocols and practice’s that stresses the importance of communication, collaboration and integration between the software development teams and the rest of our information technology professionals.

We need to start thinking beyond our IDE, we need to start thinking beyond our kanban boards and our service management systems, we need to increase our collaboration and our communication, and we need to make sure that we never create a monster like shared dependency again.

call yourself a software tester?

I have just been prompted (via some AdWords) to write a quick blog post around the subject of defect prevention. Not detection, prevention.

For a long time the software testing community has been taught that unless a defect is being managed in a defect management tool, then whatever testing process you are following is probably not fit for purpose. That is, you cant be doing serious software testing, because you are not being serious about your bugs. But it that true?…

Continue reading

Testers – we don’t need that any more.

In the past Uncle Bob and Martin Folwer have both pissed me off with their “we don’t need testers anymore” messages that they have delivered in the past. To be fair, Martin knows he pissed me off, i told him, (stick to getting more women into IT Martin); but recently Uncle Bob seems to have recognised the benefits of having career testers within the team. Perhaps he hadn’t met good testers before; after the recent discussions I had at the London Tester Gathering it seems that good testers (Agile or not) are a fairly rare breed.

So O.K, let me stand back for a moment and reflect on what XP teaches us in that a highly disciplined team should be able to achieve zero defects, so its easy to say, zero defects means we don’t need testers.
Moreover, at the SIGIST conference in June, Julian Harty (ex Google  senior test engineer) delivered a presentation that posed the question, do we need testers. What if we stop testing? After all these days people seem to want speed first and fully working functionality secondary. The future of testing?
So yes the playing field has changed, its true. But that doesn’t mean i have to agree with it.

If the software is crummy, your users will only tolerate it to a point. A great example of this is reading the feedback comments on iPhone App store. Even when a piece of software is offered for free, people leave bad feedback complaining about the uselessness or poor UI etc.

The crux of the matter for me is this. Even if we did find the magic incantation that could give us defect free code, it still wouldn’t mean the business would get quality software.

I’m signed up to Bachs ideal, but i’m embedded in an XP/Lean team. We have 90% test covereage across 903 classes and 201,000 lines of code (and counting).  Those thousands of tests run in seconds, before they make their way up the CI pipeline to run UI and performace tests. No nightly build, we test every checkin.

CI had my support 100%, because it should mitigate lots of manual testing. In fact it should mitigate lots of testing. As the product we work on has matured, the CI has formed a regression pack. Its sound great right? So why do I still have a team of 8 testers and a 15% defect rate?? Because developers are humans not robots. Because developers focus on what “Done” looks like, and not the bigger picture. Because they dont understand the context within which the story was written and thus make an assumption. Because they are testing in the small. Because they didn’t have the right data in the development environment. Because there exists two different mindsets between good developers and good testers. But non of those reasons are new, or particular to Agile. Its all old news.

But then i read this by Chief Frank C. Montagna

“Firefighters, as all humans, make mistakes. When firefighters make a mistake on the job, however, it can be life-threatening to themselves, to their coworkers, and to the public they serve. Nonetheless, firefighters will continue to make mistakes and on occasion will repeat a mistake.

Our goal should be to learn from each mistake and to try not to repeat it. We should also teach others not to make the same mistake we made. To do this, we have to admit our mistake publicly by telling others about it. This is not always easy to do. “

This Agile vs Tester friction just needs us, the testers to be pragmatic and professional; we must continue to share best practice, collaborate and talk to the developers earlier in story life-cycle. We should shout STOP! sooner rather than later, jump in feet first and embrace what may feel like an awkward discussion.

We testers are not here to simply find defects, appoint blame, or hold a “who sucks the most” post mortem. We are here to add value, and if we don’t, then why do we need testers?

A defect report is pretty meaningless. You cant sell it or market it to your customers. Worse, it has overheads. So why have one? Why not have a discussion with a developer and a product owner instead?

We (Developers and QA) both work for the same company, so why aren’t we working together, and demonstrating to Martin and Uncle Bob, that we are valuable and that testing and testers should be taken seriously.

Testers – we don’t need that any more.

In the past Uncle Bob and Martin Fowler have both pissed me off with their “we don’t need testers any more” messages that they have delivered in the past. To be fair, Martin knows he pissed me off, i told him, (stick to getting more women into IT Martin); but recently Uncle Bob seems to have recognised the benefits of having career testers within the team. Perhaps he hadn’t met good testers before; after the recent discussions I had at the London Tester Gathering it seems that good testers (Agile or not) are a fairly rare breed….
Continue reading

The Tester’s Headache: Tester Certification – my take…

Reading Simons Blog post The Tester’s Headache: Tester Certification – my take… got me simmering again.

This something I have been bending my peers ears with and blogged in passing for over a year now.

For me the value in the foundation cert. Was two fold (btw I’m old school ISEB), it gave testers a commonon taxonomy and the course provided valuable networking.

Today too many of the candidates I interview simply see the certificateas a corgi gas fiiters badge. I have the cert. Therefore I’m a tester.

Worse still many candidates have self studied, and there the only aim is  to complete practice exam papers or question banks.

I was chilled to the core when speaking with Dave Evans from SQS as I thought the Agile Tester cert. I had heard about was a joke, or tester lore. It seems istqb are serious.

All that has happened is that the cert. Has been de valued, and so I don’t make it a “must have” in the job spec anymore.

Has the calibre of applicant decreased? Nope, not one bit, it’s just increased the volume.

I still have the blisters from the ISEB practitioner exam, but I can’t think of anything I use from the sylabus now we are all out Agile.

Stuart