Spider, Crawlers and Bots oh my!

A collection of crawlies…

User-Agent: AddThis.com robot tech.support@clearspring.com
User-Agent: adidxbot/2.0 (+http://search.msn.com/msnbot.htm)
User-Agent: AdsBot-Google (+http://www.google.com/adsbot.html)
User-Agent: AdsBot-Google-Mobile (+http://www.google.com/mobile/adsbot.html) Mozilla (iPhone; U; CPU iPhone OS 3 0 like Mac OS X) AppleWebKit (KHTML, like Gecko) Mobile Safari
User-Agent: bitlybot
User-Agent: BOT/0.1 (BOT for JCE)
User-Agent: ClickTale bot
User-Agent: ContextAd Bot 1.0
User-Agent: DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
User-Agent: Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)
User-Agent: Fve Nutch Spider/Nutch-1.7
User-Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html)
User-Agent: Halebot (Mozilla/5.0 compatible; Halebot/2.1; http://www.tacitknowledge.com/halebot/) << http://biggeye.tacitknowledge.com/laterooms-com/profile
User-Agent: Mozilla/5.0 (compatible; aiHitBot/2.8; +http://endb-consolidated.aihit.com/)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2)
User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0 ; Claritybot)
User-Agent: Mozilla/5.0 (compatible; AhrefsBot/5.0; +http://ahrefs.com/robot/)
User-Agent: Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)
User-Agent: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)
User-Agent: Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
User-Agent: Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)
User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
User-Agent: Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)
User-Agent: Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)
User-Agent: Mozilla/5.0 (compatible; Ezooms/1.0; help@moz.com)
User-Agent: Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://import.io)
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
User-Agent: Mozilla/5.0 (compatible; linkCheck)
User-Agent: Mozilla/5.0 (compatible; LinkChecker/8.3; +http://wummel.github.com/linkchecker/)
User-Agent: Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)
User-Agent: Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)
User-Agent: Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Selenium Bot)
User-Agent: Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
User-Agent: Mozilla/5.0 (compatible; SEOkicks-Robot; +http://www.seokicks.de/robot.html)
User-Agent: Mozilla/5.0 (compatible; SemrushBot/0.97; +http://www.semrush.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)
User-Agent: Mozilla/5.0 (compatible; TweetmemeBot/3.0; +http://tweetmeme.com/)
User-Agent: Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)
User-Agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
User-Agent: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)
User-Agent: Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)
User-Agent: Mozilla/5.0 (compatible; YoudaoBot/1.0; http://www.youdao.com/help/webmaster/spider/; )
User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:25.0) Gecko/20100101 Firefox/25.0
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13 << seen from this ip 162.242.135.149
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.24) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24 Mozilla/3.5 (Google-HotelAdsVerifier)
User-Agent: Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like Gecko) (Exabot-Thumbnails)
User-Agent: Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)
User-Agent: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; Selenium Bot)
User-Agent: Mozilla/5.0+(compatible;+PiplBot;++http://www.pipl.com/bot/)
User-Agent: Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)
User-Agent: msnbot/2.0b (+http://search.msn.com/msnbot.htm)
User-Agent: msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)
User-Agent: msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)
User-Agent: Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)
User-Agent: psbot/0.1 (+http://www.picsearch.com/bot.html)
User-Agent: psbot-image (+http://www.picsearch.com/bot.html)
User-Agent: QuerySeekerSpider ( http://queryseeker.com/bot.html )
User-Agent: rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)
User-Agent: SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google
User-Agent: ShowyouBot (http://showyou.com/crawler)
User-Agent: SMNutchSpider/Nutch-1.7
User-Agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
User-agent: spider
User-Agent: Twitterbot/1.0
User-Agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/bot/)
User-Agent: WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)
User-Agent: WIRE/0.22 (Linux; x86_64; Bot,Robot,Spider,Crawler)
User-Agent: www.integromedb.org/Crawler
User-Agent: Yahoo:LinkExpander:Slingstone
User-Agent: Yeti/1.0 (NHN Corp.; http://help.naver.com/robots/)

Thought of the day.

Conway’s law is an adage named after computer programmer Melvin Conway, who introduced us to the idea back in 1968, yes they had computers back then kids. It states that:
“organisations which design systems … are constrained to produce designs which are copies of the communication structures of these organisations”.

What do we do about it?

DevOps. While devops is a portmanteau of development and operations, it is not another team. The last thing our IT family needs is yet another functional silo.

DevOps is a set of protocols and practice’s that stresses the importance of communication, collaboration and integration between the software development teams and the rest of our information technology professionals.

We need to start thinking beyond our IDE, we need to start thinking beyond our kanban boards and our service management systems, we need to increase our collaboration and our communication, and we need to make sure that we never create a monster like shared dependency again.

call yourself a software tester?

I have just been prompted (via some AdWords) to write a quick blog post around the subject of defect prevention. Not detection, prevention.

For a long time the software testing community has been taught that unless a defect is being managed in a defect management tool, then whatever testing process you are following is probably not fit for purpose. That is, you cant be doing serious software testing, because you are not being serious about your bugs. But it that true?…

Continue reading

Adafruit I2C RGB 16×2 LCD+Keypad & Raspberry PI

I recently bought one of these

Adafruit RGB Negative 16×2 LCD+Keypad Kit for Raspberry Pi from an Adafruit reseller in the UK (via eBay), for two reasons. 1 to use in my OBDII project, and 2 to learn how they did it (serial to parallel and some 4 bit mangle).

You see the device uses an MCP23017 16bit I2C port extender, and most 16×2 LCD displays require a 4-bit bus (4 data lines) , and an RS, RW and E lines, so i was interested to see how Adafruit were implementing the lcd display protocol over the i2c bus….

Continue reading

The wrong type of glue?

I kept on hearing the same kinds of grumbles on the floor and in retrospectives. The team were basicaly saying, we are being forced to use the wrong type of glue. The team are a pasinate lot, and so full and frank debates would flourish on which was the best type of glue, and who knew how best to apply it.

a selection of glues
Which glue is best for Baths?

Because i’d heard these grumbles a few times, i started to wonder why my awesome team were still struggling over what seemed trivial to me, just choose one and try it, right?

That is until it dawned on me.  What on earth are they trying to glue back together?

The bath, as it turns out.

The problem with gluing the bath was explained to me in great detail, some glues would leave an ugly stain, while others were very messy to handle, some other types of glue would set before the bath was stuck together again, and in most cases the glue wasnt water proof and the bath would leak. Despite this, the team were determined to get this right, they would get the bath glued together!

a bath in two halves
How to best stick the bath back together again?

Why is the bath in two halves? Seems like an obvious question now, but somehow i hadn’t asked that of the team.

“We cut the bath in half because it didn’t fit, someone ordered the wrong size bath” came the reply. So focused on delivery, the team hadnt stopped to think about telling the customer the batch was the wrong size, they just reached for the saw and cut it in half.

Obviously we haven’t stopped writing code, and invested in plumbing, i’m using a metaphor that describes what we have been doing for the last couple of months really well. In fact i can extend it a little further.

When i asked one of the dev team leads why they cut the bath in half, they suggested that they had been told to do so, but had they misunderstood what was being said?

When they had told foggy, the tech lead that they had an issue with the bath, he focused in so tightly on the problem that he only suggested that their plumbing was very complicated and that complexity was storing up technical debt that would be hard to pay down later, began to show them how they should improve and simplify it, he missed the fact that the bath was in two completely. They took his ignorance of the matter as acceptance of the fact.

To compound matters further, when JRH one of the BAs, thought he was being pragmatic in his approach, the developers actually took what he was saying as, stop crying about the glue and just do it. It turns out the BA didnt know the bath was in two either. He had made the same assumption as me, what’s so hard about choosing a glue?

How did this happen? I’m not entirly sure, but i have identified several problems that have grown like mould on a loaf of bread, slow and sure until it becomes unpalatable.

The good news is that the team obviously want to work and want to deliver, so eager to do so, that they made a decision that would allow them to continue to work. In doing so they unwittingly slipped down the rabbit hole.

In my next post i’ll explain how we are dealing with the fat end of the Business Requirement funnel, and this is causing the team to thrash.

Testers – we don’t need that any more.

In the past Uncle Bob and Martin Folwer have both pissed me off with their “we don’t need testers anymore” messages that they have delivered in the past. To be fair, Martin knows he pissed me off, i told him, (stick to getting more women into IT Martin); but recently Uncle Bob seems to have recognised the benefits of having career testers within the team. Perhaps he hadn’t met good testers before; after the recent discussions I had at the London Tester Gathering it seems that good testers (Agile or not) are a fairly rare breed.

So O.K, let me stand back for a moment and reflect on what XP teaches us in that a highly disciplined team should be able to achieve zero defects, so its easy to say, zero defects means we don’t need testers.
Moreover, at the SIGIST conference in June, Julian Harty (ex Google  senior test engineer) delivered a presentation that posed the question, do we need testers. What if we stop testing? After all these days people seem to want speed first and fully working functionality secondary. The future of testing?
So yes the playing field has changed, its true. But that doesn’t mean i have to agree with it.

If the software is crummy, your users will only tolerate it to a point. A great example of this is reading the feedback comments on iPhone App store. Even when a piece of software is offered for free, people leave bad feedback complaining about the uselessness or poor UI etc.

The crux of the matter for me is this. Even if we did find the magic incantation that could give us defect free code, it still wouldn’t mean the business would get quality software.

I’m signed up to Bachs ideal, but i’m embedded in an XP/Lean team. We have 90% test covereage across 903 classes and 201,000 lines of code (and counting).  Those thousands of tests run in seconds, before they make their way up the CI pipeline to run UI and performace tests. No nightly build, we test every checkin.

CI had my support 100%, because it should mitigate lots of manual testing. In fact it should mitigate lots of testing. As the product we work on has matured, the CI has formed a regression pack. Its sound great right? So why do I still have a team of 8 testers and a 15% defect rate?? Because developers are humans not robots. Because developers focus on what “Done” looks like, and not the bigger picture. Because they dont understand the context within which the story was written and thus make an assumption. Because they are testing in the small. Because they didn’t have the right data in the development environment. Because there exists two different mindsets between good developers and good testers. But non of those reasons are new, or particular to Agile. Its all old news.

But then i read this by Chief Frank C. Montagna

“Firefighters, as all humans, make mistakes. When firefighters make a mistake on the job, however, it can be life-threatening to themselves, to their coworkers, and to the public they serve. Nonetheless, firefighters will continue to make mistakes and on occasion will repeat a mistake.

Our goal should be to learn from each mistake and to try not to repeat it. We should also teach others not to make the same mistake we made. To do this, we have to admit our mistake publicly by telling others about it. This is not always easy to do. “

This Agile vs Tester friction just needs us, the testers to be pragmatic and professional; we must continue to share best practice, collaborate and talk to the developers earlier in story life-cycle. We should shout STOP! sooner rather than later, jump in feet first and embrace what may feel like an awkward discussion.

We testers are not here to simply find defects, appoint blame, or hold a “who sucks the most” post mortem. We are here to add value, and if we don’t, then why do we need testers?

A defect report is pretty meaningless. You cant sell it or market it to your customers. Worse, it has overheads. So why have one? Why not have a discussion with a developer and a product owner instead?

We (Developers and QA) both work for the same company, so why aren’t we working together, and demonstrating to Martin and Uncle Bob, that we are valuable and that testing and testers should be taken seriously.