Testers – we don’t need that any more.

In the past Uncle Bob and Martin Folwer have both pissed me off with their “we don’t need testers anymore” messages that they have delivered in the past. To be fair, Martin knows he pissed me off, i told him, (stick to getting more women into IT Martin); but recently Uncle Bob seems to have recognised the benefits of having career testers within the team. Perhaps he hadn’t met good testers before; after the recent discussions I had at the London Tester Gathering it seems that good testers (Agile or not) are a fairly rare breed.

So O.K, let me stand back for a moment and reflect on what XP teaches us in that a highly disciplined team should be able to achieve zero defects, so its easy to say, zero defects means we don’t need testers.
Moreover, at the SIGIST conference in June, Julian Harty (ex Google  senior test engineer) delivered a presentation that posed the question, do we need testers. What if we stop testing? After all these days people seem to want speed first and fully working functionality secondary. The future of testing?
So yes the playing field has changed, its true. But that doesn’t mean i have to agree with it.

If the software is crummy, your users will only tolerate it to a point. A great example of this is reading the feedback comments on iPhone App store. Even when a piece of software is offered for free, people leave bad feedback complaining about the uselessness or poor UI etc.

The crux of the matter for me is this. Even if we did find the magic incantation that could give us defect free code, it still wouldn’t mean the business would get quality software.

I’m signed up to Bachs ideal, but i’m embedded in an XP/Lean team. We have 90% test covereage across 903 classes and 201,000 lines of code (and counting).  Those thousands of tests run in seconds, before they make their way up the CI pipeline to run UI and performace tests. No nightly build, we test every checkin.

CI had my support 100%, because it should mitigate lots of manual testing. In fact it should mitigate lots of testing. As the product we work on has matured, the CI has formed a regression pack. Its sound great right? So why do I still have a team of 8 testers and a 15% defect rate?? Because developers are humans not robots. Because developers focus on what “Done” looks like, and not the bigger picture. Because they dont understand the context within which the story was written and thus make an assumption. Because they are testing in the small. Because they didn’t have the right data in the development environment. Because there exists two different mindsets between good developers and good testers. But non of those reasons are new, or particular to Agile. Its all old news.

But then i read this by Chief Frank C. Montagna

“Firefighters, as all humans, make mistakes. When firefighters make a mistake on the job, however, it can be life-threatening to themselves, to their coworkers, and to the public they serve. Nonetheless, firefighters will continue to make mistakes and on occasion will repeat a mistake.

Our goal should be to learn from each mistake and to try not to repeat it. We should also teach others not to make the same mistake we made. To do this, we have to admit our mistake publicly by telling others about it. This is not always easy to do. “

This Agile vs Tester friction just needs us, the testers to be pragmatic and professional; we must continue to share best practice, collaborate and talk to the developers earlier in story life-cycle. We should shout STOP! sooner rather than later, jump in feet first and embrace what may feel like an awkward discussion.

We testers are not here to simply find defects, appoint blame, or hold a “who sucks the most” post mortem. We are here to add value, and if we don’t, then why do we need testers?

A defect report is pretty meaningless. You cant sell it or market it to your customers. Worse, it has overheads. So why have one? Why not have a discussion with a developer and a product owner instead?

We (Developers and QA) both work for the same company, so why aren’t we working together, and demonstrating to Martin and Uncle Bob, that we are valuable and that testing and testers should be taken seriously.

Testers – we don’t need that any more.

In the past Uncle Bob and Martin Fowler have both pissed me off with their “we don’t need testers any more” messages that they have delivered in the past. To be fair, Martin knows he pissed me off, i told him, (stick to getting more women into IT Martin); but recently Uncle Bob seems to have recognised the benefits of having career testers within the team. Perhaps he hadn’t met good testers before; after the recent discussions I had at the London Tester Gathering it seems that good testers (Agile or not) are a fairly rare breed….
Continue reading

The Tester’s Headache: Tester Certification – my take…

Reading Simons Blog post The Tester’s Headache: Tester Certification – my take… got me simmering again.

This something I have been bending my peers ears with and blogged in passing for over a year now.

For me the value in the foundation cert. Was two fold (btw I’m old school ISEB), it gave testers a commonon taxonomy and the course provided valuable networking.

Today too many of the candidates I interview simply see the certificateas a corgi gas fiiters badge. I have the cert. Therefore I’m a tester.

Worse still many candidates have self studied, and there the only aim is  to complete practice exam papers or question banks.

I was chilled to the core when speaking with Dave Evans from SQS as I thought the Agile Tester cert. I had heard about was a joke, or tester lore. It seems istqb are serious.

All that has happened is that the cert. Has been de valued, and so I don’t make it a “must have” in the job spec anymore.

Has the calibre of applicant decreased? Nope, not one bit, it’s just increased the volume.

I still have the blisters from the ISEB practitioner exam, but I can’t think of anything I use from the sylabus now we are all out Agile.

Stuart

The Tester’s Headache: Tester Certification – my take…

Reading Simons Blog post The Tester’s Headache: Tester Certification – my take… got me simmering again.

This something I have been bending my peers ears with and blogged in passing for over a year now.

For me the value in the foundation cert. Was two fold (btw I’m old school ISEB), it gave testers a commonon taxonomy and the course provided valuable networking.

Today too many of the candidates I interview simply see the certificateas a corgi gas fiiters badge. I have the cert. Therefore I’m a tester.

Worse still many candidates have self studied, and there the only aim is  to complete practice exam papers or question banks.

I was chilled to the core when speaking with Dave Evans from SQS as I thought the Agile Tester cert. I had heard about was a joke, or tester lore. It seems istqb are serious.

All that has happened is that the cert. Has been de valued, and so I don’t make it a “must have” in the job spec anymore.

Has the calibre of applicant decreased? Nope, not one bit, it’s just increased the volume.

I still have the blisters from the ISEB practitioner exam, but I can’t think of anything I use from the sylabus now we are all out Agile.

Stuart

Making the Ferrograph SDX 63 Aurora XML aware

I mentioned in a previous post how we have used an old Ferrograph SDX 63 LED moving message display / wallboard and Robert Cowards excellent ADF firmware to give us a sign that is to be used for our Extreme Feedback device.

Once i had got my head round the Alpha Protocol, i started to knit a huge perl module together to control the sign. Essentially all it did was create an alpha protocol formated string and send it to the sign. At first this seed to be the logical way to approach the problem. I had already written an IRC bot that announces the build status into the channel and it too was written in about 10 lines of perl, so giving the bot access to the the sign module would allow the bot to relay messages to the sign.

However, once you start send messages to the sign you realise that what you actually want is an SGML way of writing message to the sign, that is to say, create your message in your mark up then send it to the sign. The sign itself only has a Z80 processor and its already overburdened with the task of displaying data on the LEDs, so making the sign understand SGML was out of the question. So how about using XML and transforming the XML through an XSLT that would result in a stream of data the sign could understand? At last i found something i actually made real use of XML transformation.

XSLT is commonly used to turn XML into html, all i had to do was create an XSL with the appropriate taxonomy for the ADF firmware and then build up my message using my own SGML.

I’ve never found a use for XML before, I’ve always found it to be a bit pointless, but then that s probably to do with my age. Once upon a time we only had 8-bit processors and 1k of RAM, and those sort of constraints make you think about how you are going to deal with your data very carefully. The last thing i would do is transmit 80 bytes of data in a 4k document, just because it can specify its contents.

SO no sooner had I finished my work when as usual i found i wasn’t the first, and software already exists that can speak the Alpha protocol and uses XML and XSLT its called bbxml (BB from the betabrite signs). Bugger!

We have statistics and charts to prove it so f**k off!

Its been a while since i’ve posted, and that has been due to partly due to the pressures of work, but but also because i didnt think i had anything significant to share. That is until this week when i had a moment of WTF!!.

I want to share with you something that is seemingly innocuous but actually very toxic; namley using velocity as a performance metric!

I have never been a big fan of metrics for metrics sake, because a lot of them a pretty meaningless. We have a focus, and that is the delivery of software. All the business wants to know is that what they have asked for is to be delievered, on time and hopefully to budget. Why bother producing Burn down or burn up charts if we dont factor in defects or Tech tasks and besides what is the point of any of it if one story can block 20 others?

The old software delivery problem still exists right?

Project Manager: How much have you got left to test?
Tester: Just one story out of the 40 we had.
Project Manager: Cool, the testing guys are nearly done.
– One week later….
Project Manager: How much have you got left to test?
Tester: Just that one story, but boy-oh-boy its a doozy.

For me metrics fall into two piles:

1. Diagnostics Metrics: These are informative indicators chosen by the team that are used to evaluate and improve the teams process. We collect diagnostic metrics to give us some insight into where we can improve, and to track whether the improvements have any effect. Simply informative, never to acertain how much value is being delieverd and of a fairly short life; if the process improves, we move on and identify other sub-optimal areas. The measurmnent is dropped a that point, its served its purpose.

2. Performance Metrics: This the measurement that details how much value is being delievered. These are used to track our performance, and are chosen carefully. There should be fewer rather than more of these metrics. Wherever possible these metrics should be outside of the teams control, that way the numbers can not be perversed. E.g User or customer feedback. “how satisfied are you with the service/product” or “how likely are you to recommend the service offered by the team to another?”.

The problem i have at the moment is that our velocity is being used as a performance metric rather than a diagnostic aid. Incredible i know, but true. Why? Because velocity cant be used as a performance measure, because it is unique to the team being measured at that point in time and only within that context. It changes, like the ebb and flow of a tide.

Not only is someone trying to use velocity as an indication of the teams performance, i have also heard the following remarks made.

  • Why is Team A slower than Team B? Wow, where do i start. Because they estimate in different scales, or maybe their iteration length is different, possibly the team composition is different? There are so many factors that can influence velocity that it’s only useful to compare it within the same team, and even then just to identify trends. The absolute value doesn’t mean anything. 
  • We only did 28 last iteration, can we push for 38, and then 58? The beginning of a project can have a naturaly low velocity, and this should be expected, however it should begin to increase after a number of iterations. However we have been asked to push and push to improve velocity when we reach a natural plateau, and we have pushed, after all,  we want to go faster, because we rock, right? Because velocity indicates the capability of our teams to deliver it will settle down itself (because our  process is stable, and we haven’t employed any creative accounting). One my heroes is Deming who noted that  in a stable process, the way to improve is to change the process.

It’s important that as journeymen (and women) of the Agile craft we remember that velocity is just a by-product of our current reality (our teams, our processes and our tools). We can only improve our process once it’s stable and we know the current capacity.
Velocity is little more than a health-check that help inform us about our team’s capability. It can not tell us about how much value is being delivered or for that matter how fast its being delivered. We can deliver a lot of story points but only through making a trade-offs with our quality which, however we measure it, will impact our ability to deliver more. As uncle Bob Martin says:

“The way to go fast, is to go well” 

Never use velocity to measure performance. After all the thing being tracked is a gumi bear, or pink elephant. Simply look at velocity as a diagnostic metric to help us improve our software delivery process.

Making the Ferrograph SDX 63 Aurora XML aware

I mentioned in a previous post how we have used an old Ferrograph SDX 63 LED moving message display / wallboard and Robert Cowards excellent ADF firmware to give us a sign that is to be used for our Extreme Feedback device.

Once i had got my head round the Alpha Protocol, i started to knit a huge perl module together to control the sign. Essentially all it did was create an alpha protocol formatted string and send it to the sign. At first this seemed to be the logical way to approach the problem….
Continue reading