The Tester’s Headache: Tester Certification – my take…

Reading Simons Blog post The Tester’s Headache: Tester Certification – my take… got me simmering again.

This something I have been bending my peers ears with and blogged in passing for over a year now.

For me the value in the foundation cert. Was two fold (btw I’m old school ISEB), it gave testers a commonon taxonomy and the course provided valuable networking.

Today too many of the candidates I interview simply see the certificateas a corgi gas fiiters badge. I have the cert. Therefore I’m a tester.

Worse still many candidates have self studied, and there the only aim is  to complete practice exam papers or question banks.

I was chilled to the core when speaking with Dave Evans from SQS as I thought the Agile Tester cert. I had heard about was a joke, or tester lore. It seems istqb are serious.

All that has happened is that the cert. Has been de valued, and so I don’t make it a “must have” in the job spec anymore.

Has the calibre of applicant decreased? Nope, not one bit, it’s just increased the volume.

I still have the blisters from the ISEB practitioner exam, but I can’t think of anything I use from the sylabus now we are all out Agile.

Stuart

We have statistics and charts to prove it so f**k off!

Its been a while since i’ve posted, and that has been due to partly due to the pressures of work, but but also because i didnt think i had anything significant to share. That is until this week when i had a moment of WTF!!.

I want to share with you something that is seemingly innocuous but actually very toxic; namley using velocity as a performance metric!

I have never been a big fan of metrics for metrics sake, because a lot of them a pretty meaningless. We have a focus, and that is the delivery of software. All the business wants to know is that what they have asked for is to be delievered, on time and hopefully to budget. Why bother producing Burn down or burn up charts if we dont factor in defects or Tech tasks and besides what is the point of any of it if one story can block 20 others?

The old software delivery problem still exists right?

Project Manager: How much have you got left to test?
Tester: Just one story out of the 40 we had.
Project Manager: Cool, the testing guys are nearly done.
– One week later….
Project Manager: How much have you got left to test?
Tester: Just that one story, but boy-oh-boy its a doozy.

For me metrics fall into two piles:

1. Diagnostics Metrics: These are informative indicators chosen by the team that are used to evaluate and improve the teams process. We collect diagnostic metrics to give us some insight into where we can improve, and to track whether the improvements have any effect. Simply informative, never to acertain how much value is being delieverd and of a fairly short life; if the process improves, we move on and identify other sub-optimal areas. The measurmnent is dropped a that point, its served its purpose.

2. Performance Metrics: This the measurement that details how much value is being delievered. These are used to track our performance, and are chosen carefully. There should be fewer rather than more of these metrics. Wherever possible these metrics should be outside of the teams control, that way the numbers can not be perversed. E.g User or customer feedback. “how satisfied are you with the service/product” or “how likely are you to recommend the service offered by the team to another?”.

The problem i have at the moment is that our velocity is being used as a performance metric rather than a diagnostic aid. Incredible i know, but true. Why? Because velocity cant be used as a performance measure, because it is unique to the team being measured at that point in time and only within that context. It changes, like the ebb and flow of a tide.

Not only is someone trying to use velocity as an indication of the teams performance, i have also heard the following remarks made.

  • Why is Team A slower than Team B? Wow, where do i start. Because they estimate in different scales, or maybe their iteration length is different, possibly the team composition is different? There are so many factors that can influence velocity that it’s only useful to compare it within the same team, and even then just to identify trends. The absolute value doesn’t mean anything. 
  • We only did 28 last iteration, can we push for 38, and then 58? The beginning of a project can have a naturaly low velocity, and this should be expected, however it should begin to increase after a number of iterations. However we have been asked to push and push to improve velocity when we reach a natural plateau, and we have pushed, after all,  we want to go faster, because we rock, right? Because velocity indicates the capability of our teams to deliver it will settle down itself (because our  process is stable, and we haven’t employed any creative accounting). One my heroes is Deming who noted that  in a stable process, the way to improve is to change the process.

It’s important that as journeymen (and women) of the Agile craft we remember that velocity is just a by-product of our current reality (our teams, our processes and our tools). We can only improve our process once it’s stable and we know the current capacity.
Velocity is little more than a health-check that help inform us about our team’s capability. It can not tell us about how much value is being delivered or for that matter how fast its being delivered. We can deliver a lot of story points but only through making a trade-offs with our quality which, however we measure it, will impact our ability to deliver more. As uncle Bob Martin says:

“The way to go fast, is to go well” 

Never use velocity to measure performance. After all the thing being tracked is a gumi bear, or pink elephant. Simply look at velocity as a diagnostic metric to help us improve our software delivery process.

Making the Ferrograph SDX 63 Aurora XML aware

I mentioned in a previous post how we have used an old Ferrograph SDX 63 LED moving message display / wallboard and Robert Cowards excellent ADF firmware to give us a sign that is to be used for our Extreme Feedback device.

Once i had got my head round the Alpha Protocol, i started to knit a huge perl module together to control the sign. Essentially all it did was create an alpha protocol formatted string and send it to the sign. At first this seemed to be the logical way to approach the problem….
Continue reading

We have statistics and charts to prove it so f**k off!

Its been a while since i’ve posted, and that has been due to partly due to the pressures of work, but but also because i didnt think i had anything significant to share. That is until this week when i had a moment of WTF!!.

I want to share with you something that is seemingly innocuous but actually very toxic; namley using velocity as a performance metric…
Continue reading

Painting over the rust – Preparation, preparation, preparation.

I thought I would follow on from my post painting over the rust.
It would be too easy for me just to say, hey our business people don’t get Agile, boo hoo. So I thought I would try and address the issue by thinking about how I felt when I was first introduced to Agile. Thinking about it this way allowed me to take a step back and understand why they don’t get it.

1. All Requirements are not needed up front.
This ones a biggie. I wrestled with it for ages, but then I would, I’m a tester and I need those requirements to do my job; after all the V-Model is nothing without upfront requirements, right?
How did I overcome this? Easy (for me at least), in that in our old way of working, even when the requirements were available early, they often didn’t reflect what was actually delivered. This is software delivery after all; we expect the customer to change its mind during the life cycle. So if I’m used to finding that the requirements documentation don’t match what’s delivered into test, why am I so worked up now? Oh wait, I’m not after all. It just felt uncomfortable because its new to me.

2. Story Points are a guess but the deadline is still firm (if not impossible).
This one is a bit of a paradox. Both are true. During estimation we size the stories with t-shirt sizes or points. The point or size at that stage is no more than an informed/educated guess. Again I had a hard time with this, until I realised that it is just a concept that allows us humans to gauge the relative size of the stories. Wrap your head around the simplicity of that device, and your home free. Sure it’s a guesstimate, but we can still meet the deadline, knowing how much work is in there. Once you get the fact that its just about relative sizing and not a unit of currency for determining performance, it becomes second nature. Its all about making sure the team gets the same amount of work passed down the conveyor belt, and that they don’t get under or over utilised. Once you know they can knock out two 4 pointers and a 6 per iteration, you know your capacity (well mostly).

3. Tasks do not get assigned at Iteration Kick off.
I didn’t struggle with this too much, in that previously a task could sit with someone that didn’t have the bandwidth to complete the task for a very long time. One important protocol is leaving things until “the last responsible moment”. By not assigning tasks we can leverage the flexibility that the last responsible moment affords. Also, during the iteration planning, the team can agree what tasks make up a story and perhaps identify the required SMEs. The whole team takes Responsibility for promises the team made. We will deliver value.

4. An increase in speed comes from maximising Quality.
It took me a while to fully understand the power of this. If the team gets it right first time, then it spend less time fixing broken code, or miss understood/poorly captured requirements. I’m sure some Kanbaners out there will be nodding sagely. But this is one of the core principals that Toyota follow, they claim to produce one brand new, fully tested, inspected and passed vehicle in the same time other manufacturers spend snag fixing at the end of their lines.
Traditionally in software delivery, code fixes are not time boxed, and a defect can loop through QA and DEV several times before its fixed. One pair can sink a load of time into fixing the defect while QA source the effort around regression. It’s a massive waste. Enforce a Zero Defect policy; get the business to sign up to it, get buy in from the operational functions (Helpdesk, System Administrators etc) and say goodbye to the defect backlog. Get it right first time and watch throughput soar. Write your tests, then your code, then apply your design.

In Summary

It was difficulty for me to adjust to being a tester in an Agile world, and it took me time. I learnt through an apprenticeship with ThoughtWorks. They may have their own flavour of Agile, regardless its not something you can learn in a book or on a course. Much of what i learnt about Agile is more about your mindset, and self discipline.

Several times along the way, I thought, this is rubbish, this is causing pain and this doesn’t work. But hey that’s Agile right? If it doesn’t work for you, you can stop, change, and monitor for improvement. Once you appreciate that, you become empowered. You can leverage the power of one.

But, we had left our business behind. We (the software delivery team) decided we had to be Agile to deliver a project, without considering the impact to the business. Worse than that, we undertook an apprenticeship with ThoughtWorks and expected the business to understand story points, velocity, waste, lean, eXtreme Programming… the list goes on.

If you haven’t taken the time to bring your business along for the ride you cant expect them just to get it.
If you use enough filler, you can paint over the rust.

Painting Over the Rust

Having Story Cards and Stand-Ups doesn’t make you Agile.

It is with a heavy heart that i watch the team descend into the latest form of chaos. Looking down on it now, i can see how we arrived here. One of the developers used the phrase “painting over the rust” to describe his dissatisfaction with the architectural decisions being made early on in the project. He was right, however what he didn’t know, was he was right on a number of levels.

I’ve struggled for some time with not being able to intelligently articulate what Agile software delivery is exactly. I hear lots of people repeating the the same old mantras, but for me, they lacked depth. Reciting the Agile manifesto to my grandmother isn’t going to help her understand what Agile software delivery is. I reason that if I don’t feel Able to describe it t my grandmother, how could i ever manage to explain it to the upper echelons of the executive?

Its not Agile, its fragile.

So the team has a story wall, and swim lanes and cards. Pretty standard stuff (actually the wall is further segmented by geographical location, because the team is split across two sites). Estimation is done planning poker style and points are points, pink elephants if you like, not days, or elapsed time (although we have worked out how much a point cost).

The team knows how many points we can get through the production line, and is comfortable managing the loss in throughput from context switching or working on new technologies etc.

We have a team made up of DEV pairs, QAs and BAs, and they understand what Agile software delivery is. They understand why we pair program. The developers are using TDD and we automate as much as we can using Continuous Integration. We have stand ups, retrospectives, iteration kick off meetings and showcases. We have brown bags and we use collaboration tools. We as a software delivery team have made the paradigm shift. But its still all buggered, why? Effectively, all we have done is painted over the rust.
The rust in our case is, (unfortunately) the executive. They don’t get it. Worse, yet i don’t think the product owners don’t get it, and in some cases i even wonder if the project managers get it. So what can we do?

We could shield the business from our internal workings. Let them treat us like a black box. They request function x, and we return a cost and an estimated delivery date. That way, they don’t have to get it, they don’t have to understand it, nothing changes for them, and the delivery team carries on irrespective. They then don’t need to learn about points or swim lanes or story cards or any other aspect of Agile. But the rust is still there, under the shiny new Agile paint, and its not going to go away, and left untreated it could get worse.

In the real world, there are only two real ways to treat rust. Cut it out, or blast it away. You can then replace the resultant hole with shiny new metal.Once the new metal is in place, its a good idea to apply a liberal coating of rust inhibitor.

Painting Over the Rust

Having Story Cards and Stand-Ups doesn’t make you Agile.

It is with a heavy heart that i watch the team descend into the latest form of chaos. Looking down on it now, i can see how we arrived here. One of the developers used the phrase “painting over the rust” to describe his dissatisfaction with the architectural decisions being made early on in the project. He was right, however what he didn’t know, was he was right on a number of levels.

I’ve struggled for some time with not being able to intelligently articulate what Agile software delivery is exactly. I hear lots of people repeating the the same old mantras, but for me, they lacked depth. Reciting the Agile manifesto to my grandmother isn’t going to help her understand what Agile software delivery is. I reason that if I don’t feel Able to describe it t my grandmother, how could i ever manage to explain it to the upper echelons of the executive?

Its not Agile, its fragile.

So the team has a story wall, and swim lanes and cards. Pretty standard stuff (actually the wall is further segmented by geographical location, because the team is split across two sites). Estimation is done planning poker style and points are points, pink elephants if you like, not days, or elapsed time (although we have worked out how much a point cost).

The team knows how many points we can get through the production line, and is comfortable managing the loss in throughput from context switching or working on new technologies etc.

We have a team made up of DEV pairs, QAs and BAs, and they understand what Agile software delivery is. They understand why we pair program. The developers are using TDD and we automate as much as we can using Continuous Integration. We have stand ups, retrospectives, iteration kick off meetings and showcases. We have brown bags and we use collaboration tools. We as a software delivery team have made the paradigm shift. But its still all buggered, why? Effectively, all we have done is painted over the rust.
The rust in our case is, (unfortunately) the executive. They don’t get it. Worse, yet i don’t think the product owners don’t get it, and in some cases i even wonder if the project managers get it. So what can we do?

We could shield the business from our internal workings. Let them treat us like a black box. They request function x, and we return a cost and an estimated delivery date. That way, they don’t have to get it, they don’t have to understand it, nothing changes for them, and the delivery team carries on irrespective. They then don’t need to learn about points or swim lanes or story cards or any other aspect of Agile. But the rust is still there, under the shiny new Agile paint, and its not going to go away, and left untreated it could get worse.

In the real world, there are only two real ways to treat rust. Cut it out, or blast it away. You can then replace the resultant hole with shiny new metal.Once the new metal is in place, its a good idea to apply a liberal coating of rust inhibitor.