DNS pre fetching

DNS pre fetch is something i have been looking into, but so far as in-browser DNS pre fetch tags go, its still a dream…

<link rel=”dns-prefetch” href=”http://working-thought.blogger.com”>

Mozilla and Chromium browsers should be able to perform DNS pre-fetching to reduce the overall page load time, however, testing this out doesn’t match the published behaviour.

  • MSIE simply does not pre fetch, as expected, perhaps IE9 will?
  • Google Chrome does pre fetch and it does seem to do it asynchronously (as per published specs), not surprising, see below.
  • Safari does not pre fetch.
  • FireFox should be able to pre fetch (see here) but i couldn’t make it go.

Meanwhile Google have re invented the DNS server with one that incorporates DNS pre fetch.

Normally when the ISPs DNS server goes back to the authoritative host to get the DNS record it just caches it without inspecting the TTL on the record. For our site that is 300 seconds, which is deliberately quite short. The next time a user requests the DNS record from their ISPs DNS server, only then does it inspect the TTL and if its expired then it will go back to the authoritative host.

Google found that their googlebot was spending more time looking up DNS records, than it was spidering pages. So they made their DNS server inspect the TTL of the record and before it expired fetch it again so the cache is kept hot.

If you give yourself a tight target like a “1 second homepage” then you may see 700ms wasted in DNS resolving. If that’s the case give the google DNS server a go. its available on 8.8.8.8 and 8.8.4.4 its quite shocking the difference it can make.

Stuart

No liars please, we’re testers.

I’ve been prompted to write this after sifting through another load of dross CVs. I say another because I was recruiting heavily last year. A recruitment drive that took me twelve months to find just two test analysts.. hold that thought.

Its not that I didn’t get many applicants the first time round I did, hundreds in fact (quite literally) and I duly read and scored every CV personally and feedback too. Not only that, I also had two of my peers review the CVs too. You cant ask for more than that. If two out of three agreed, the decision was final.

The reason it took a year was down to the quality of the CVs. At least half of the CVs we read didn’t state the daily tasks that you would expect a career tester to be doing. Only if you were playing at being a tester would you make that mistake. Unfortunately sometimes the CV would tick all the boxes and we would invite the applicant in for interview. Imagine their horror when they were faced with a simple exercise to test their SQL skills even though on their CV they had waxed lyrical about how they practically spoke to their friends in SQL because they had used it so much they were fluent.

“so I have a table called customer, with the fields ID, Name, Address. How would I get all of the records form the table?” you should see some of the answers. Its often hard for me not to shout “just give up, I know you haven’t got a bloody clue despite what it says on your CV” they muddle on “GET ALL RECORDS FROM TABLE WHERE NAME == CUSTOMERS AND ID….”

Imagine also them recoiling in their seats when I ask them to draw out the V-model and label it.
“What’s that you say, you don’t know how? But here on your CV you state you are a senior test analyst and have and ISEB foundation certificate, how do you not know the V model?” they shuffle their feet and mumble how they studied at home, well not actually studied so much as bought a guide on how to pass that’s full of example questions.

I watch in wonder as their faces contort when I ask them “so what is the difference between white box testing and black box” I let them fumble through telling me how they have used both of those “methodologies” and I follow up with “can you give me some examples where you white box tested?”

Then come the questions on web testing (its what we do after all) “so whats a cookie I ask”, the smile, easy they think “it’s a virus you get from visiting sex sites” oh my! What should I do as a tester? “never ever accept cookies, they track all your movements, like little spys in your computer”.
I follow with a simple exercise about shopping carts and sessions to see if the candidate understands why a cookie maybe important here. “the system gets all that info from the cookie” but how did it get in the cookie? “from the internet” can I see it, I want to see my cookie “oh no you cant see them, they are secret”.

I have even had to terminate several interviews because it became apparent very quickly that the applicant in the chair didn’t actually know what was on their CV because they had just copied it off a (delete as appropriate) friend / colleague / LinkedIn profile.

It was so bad in the past that we setup an online quiz. It was very simple, multiple choice, some questions around testing, some around our domain. Some very easy questions “Which of the following is a search engine” with an obvious answer “google”. We discovered a side effect of an easy question like that is that we could see how fast the candidate answered the question when they knew it straight off the cuff (about 9 seconds for that question) and compare it to a testing related question that took 3 them minutes to answer (did they have to search for the answer?). The test was very easy for a career web tester, but not so easy for an IT support person or BA or even a developer who fancied a move into testing. Its only real purpose was to filter out the complete time wasters.

So here I am again, I’m hiring and I’m inundated with CVs and again 50% are pure wasted bandwidth (I don’t give them the luxury of printing them out). But this time I don’t have the online test, and I’m gnashing my teeth at some of the incredulous stuff in these CVS. Some of them read like horrible blog postings “on this job we had this challenge and so we had to X because of Y but then Z happened and so we used plan A…” blah blah blah “then the business wanted B but I wrote the very detailed spec of C” bletch grrr spit pfftt, its all I can do to stop myself posting these fetid monologues online for no other reason than ridicule and I hate myself for it.

So faced with the prospect of interviewing a load of (lets be blunt here) bullshit artists only to show them the door at the end of it isn’t a prospect I’m overjoyed with. I don’t want to spend two hours of my life demonstrating why the candidate is a liar. I don’t want to be associated with these bottom feeders in a professional sense either. I loathe them, and I loathe the arseholes that gave them a “consultant” badge at Logica (or any other faceless body shop), because now they think they are gods gift and we should roll out the red carpet for them.

I will continue to sift through the dross, the cream always floats and that’s what I’m after, the cream, the crème de la crème.

So if you are interviewing a tester that tells you that you gave them a much easier ride than in a previous interview they attended, you know I rejected them, and that you may want to make use of that probationary period.

But if you’re a testers who’s CV isn’t straight up and down, you may want to rethink applying for a job with me.

Oh and by the way, don’t put “I have a keen eye for attention to detail” then litter your CV with spelling mistakes poor grammar and mixed styling!

No liars please, we’re testers.

I’ve been prompted to write this after sifting through another load of dross CVs. I say another because I was recruiting heavily last year. A recruitment drive that took me twelve months to find just two test analysts.. hold that thought.

Its not that I didn’t get many applicants the first time round I did, hundreds in fact (quite literally) and I duly read and scored every CV personally and feedback too. Not only that, I also had two of my peers review the CVs too. You cant ask for more than that. If two out of three agreed, the decision was final.

The reason it took a year was down to the quality of the CVs. At least half of the CVs we read didn’t state the daily tasks that you would expect a career tester to be doing. Only if you were playing at being a tester would you make that mistake. Unfortunately sometimes the CV would tick all the boxes and we would invite the applicant in for interview. Imagine their horror when they were faced with a simple exercise to test their SQL skills even though on their CV they had waxed lyrical about how they practically spoke to their friends in SQL because they had used it so much they were fluent.

“so I have a table called customer, with the fields ID, Name, Address. How would I get all of the records form the table?” you should see some of the answers. Its often hard for me not to shout “just give up, I know you haven’t got a bloody clue despite what it says on your CV” they muddle on “GET ALL RECORDS FROM TABLE WHERE NAME == CUSTOMERS AND ID….”

Imagine also them recoiling in their seats when I ask them to draw out the V-model and label it.
“What’s that you say, you don’t know how? But here on your CV you state you are a senior test analyst and have and ISEB foundation certificate, how do you not know the V model?” they shuffle their feet and mumble how they studied at home, well not actually studied so much as bought a guide on how to pass that’s full of example questions.

I watch in wonder as their faces contort when I ask them “so what is the difference between white box testing and black box” I let them fumble through telling me how they have used both of those “methodologies” and I follow up with “can you give me some examples where you white box tested?”

Then come the questions on web testing (its what we do after all) “so whats a cookie I ask”, the smile, easy they think “it’s a virus you get from visiting sex sites” oh my! What should I do as a tester? “never ever accept cookies, they track all your movements, like little spys in your computer”.
I follow with a simple exercise about shopping carts and sessions to see if the candidate understands why a cookie maybe important here. “the system gets all that info from the cookie” but how did it get in the cookie? “from the internet” can I see it, I want to see my cookie “oh no you cant see them, they are secret”.

I have even had to terminate several interviews because it became apparent very quickly that the applicant in the chair didn’t actually know what was on their CV because they had just copied it off a (delete as appropriate) friend / colleague / LinkedIn profile.

It was so bad in the past that we setup an online quiz. It was very simple, multiple choice, some questions around testing, some around our domain. Some very easy questions “Which of the following is a search engine” with an obvious answer “google”. We discovered a side effect of an easy question like that is that we could see how fast the candidate answered the question when they knew it straight off the cuff (about 9 seconds for that question) and compare it to a testing related question that took 3 them minutes to answer (did they have to search for the answer?). The test was very easy for a career web tester, but not so easy for an IT support person or BA or even a developer who fancied a move into testing. Its only real purpose was to filter out the complete time wasters.

So here I am again, I’m hiring and I’m inundated with CVs and again 50% are pure wasted bandwidth (I don’t give them the luxury of printing them out). But this time I don’t have the online test, and I’m gnashing my teeth at some of the incredulous stuff in these CVS. Some of them read like horrible blog postings “on this job we had this challenge and so we had to X because of Y but then Z happened and so we used plan A…” blah blah blah “then the business wanted B but I wrote the very detailed spec of C” bletch grrr spit pfftt, its all I can do to stop myself posting these fetid monologues online for no other reason than ridicule and I hate myself for it.

So faced with the prospect of interviewing a load of (lets be blunt here) bullshit artists only to show them the door at the end of it isn’t a prospect I’m overjoyed with. I don’t want to spend two hours of my life demonstrating why the candidate is a liar. I don’t want to be associated with these bottom feeders in a professional sense either. I loathe them, and I loathe the arseholes that gave them a “consultant” badge at Logica (or any other faceless body shop), because now they think they are gods gift and we should roll out the red carpet for them.

I will continue to sift through the dross, the cream always floats and that’s what I’m after, the cream, the crème de la crème.

So if you are interviewing a tester that tells you that you gave them a much easier ride than in a previous interview they attended, you know I rejected them, and that you may want to make use of that probationary period.

But if you’re a testers who’s CV isn’t straight up and down, you may want to rethink applying for a job with me.

Oh and by the way, don’t put “I have a keen eye for attention to detail” then litter your CV with spelling mistakes poor grammar and mixed styling!

Hokey Cokey or Hocus Pocus

Back in September 2007 we released a new version of our search application.
The new version was step change for us. At that time we were powering the core of our search offering with an Oracle database, and a Java application that returned flat html. It was all very web 1.0, and we had begun to see issues with the performance of the site and discovered that throwing 8 more servers into the oracle grid didn’t gives 8 x more power. We took the oracle database out of the mix and brought in Endeca search.

The Endeca API allowed us to show the visitors how many of the things they were searching for were available before they submitted the search from. For example, if you were searching for a BMW 5-Series, the fuel type drop down on the search form would list the number available next to the drop down [Petrol (5), LPG (2)]. So a big change to the “build your search and submit it and hope it returns results” model we had previously used. To be able to allow this feature to work we had to use Ajax, or more specifically JSon. So as the user changed their criterion the relevant drop down were updated without refreshing the form. So like I said a step change for the front end, the back end and user behaviour.

The new version was released in stages, inviting visitors to the site to try the new version. This tactic has its own associated problems (for example only a certain type of person will follow a “try our new X” link, so your new application doesn’t get exposure to a good representation of your audience), once the visitors had interacted with the new search form, we invited them to give us some feedback, so that we could improve on what we had we had done. Below is a selection of that feedback:

It crashed 5 times and slow.
Takes longer, too complicated, should be kept simple
Too slow!!
Not as easy to use
Very slow to bring up menus, Spent time waiting.
It doesn’t work – my searches keep spontaneously disapearing (Cars)
is slow. maybe is my broadband problem.
I don’t want to know how many cars in the uk, I just want to know how many in my local area
It’s silly to have to click ‘show your results’ it was better on the previous version where it showed the results.
Too slow in uploads.
More criteria = more time.
Too many details to put in .
More options, as not 100% encyclopaedic knowledge of cars, the sub model option was difficult .


So, pretty damming stuff. But something didn’t make any sense. We had rigorously tested the performance of the system and were confident that it was faster than the old system. The Market leading browse back then was IE6 and given that we had engineered or built it for IE6 it positively flew in Opera or FireFox. So we were perplexed. That is until we did some usability testing (I wont discuss the fact that the usability testing was too late in the project to be really beneficial).

The usability testing did allow us to understand why we got so many “slow” comments on the feedback. Faced with all the feedback the new search form gave the users as they refined their search, they believed two things, 1. That they had to fill in all of the options and 2, that they couldn’t interact with the form until the animated counters stopped moving.

Manufacturer, Model, Variant, Trim, Colour, Fuel Type, Mileage, Age, Min Price, Max Price, Distance from the visitor. As the user slowly changed each of the drop down controls on the search form, some options would become unavailable (greyed out). This was because the back end had contracted them out of the possible results. If no Red BMWs were available, the colour Red would not be available for choice on the drop down control. So the user would change say model to 3-Series and find there wasn’t any Red available on the drop down, so they would back up and change 3-Series to 5-Series and so on. They didn’t realise you could just search for all the Red cars within 20 miles of their house, and drill down from there. To some extent they still don’t 2 years on.

It reminds me a little bit of when I was working on a project with BT and the then new System-X exchanges. The switches could support loads of (then) new features (things we take for granted today, like 1471 in the UK). Being a geek I was amazed at what I could do with a DTMF (touch tone) phone, and went out immediately and bought one. The next day I asked why BT hadn’t publicised any of the features and capabilities. Their response was immediate, dry and serious. “Our users won’t understand them”. I can still remember how I felt, almost like I had stumbled into some great conspiracy. BT wanted to keep people in the dark, and protect them from the nasty technology that might confuse them.
It was several years later that I received a booklet with my phone bill that explained the features and how to access them. Having used the features for sometime at hat point, I had great difficulty in understanding the booklet. Maybe BT were right, maybe it was all too confusing.

Fast forward to now, and my current project. Again another release and another step change. This time the look and feel of the site has been overhauled. The backend is still Endeca powered, but the Java app has been completely rewritten. And in the rewriting of the application we have taken the opportunity to bake testing in from the start. The JavaScript, cascading style sheets and html are all tested automatically. Regression should be a thing of the past (but that’s another blog post), the application has unit testing, functional and non functional testing applied at every check in. The functional testing has been expanded into “User Journey” testing, in which likely user scenarios are played out. All of this happens automatically before the application reaches QA. Then the QA team goes to town, with time to exercise their real skill, exploratory testing. So there you have it, never in the history of our company has a product been so well inspected. So we felt pretty confident when we were ready for Beta.

This time round, instead of us inviting user to try out our new site, we employed AB testing. 5% of our traffic was diverted to the new site, and once again, users were invited to leave feedback. I took the opportunity to setup a Google alert for spotting the use of the beta url in forum or blog posts, so I could keep track of what the community was saying.
Once again the feedback came in…

The used car search, the old one is much clearer to use and a lot better, . The new “improved” one is poor.
Preffered old site looked more proffesional and was easier to use.
The search criteria should be your main focus and keep that in a clear box format like your old site and allow people to search quickly but also as specifically as they want.
The old site is much better the new site is more complicated to use in the end I shut it down and went on to ebay please change back.
It looks much better than the previous website, but since I dont live in UK, I usually have to copy and paste the London postcode from the FAQ page. Unfortunately, I cannot find the page.
Bad design. Not as easy to use and selct options, not as clear and concise. the old one was perfect.

Erm, what? The old one was better? Perfect? Now we are confused.

So again we tackle the perceived issues of our users. We keep seeing comments of missing images, and we start pulling apart the application, the infrastructure and network. It turns out it was an Ad Blocker, that has decided that the way we format our image urls (cache busting) means they are adverts and blocks them.
People complain of slow loading time, so I begin to conduct some testing around that. I conclude they maybe right, so we engage with Gomez to find out for sure. Gomez shows something alarming. Users on a half decent (2Mb and above) broadband connection will get a decent experience. People on anything less are going to be pulling their hair out. The digital Britain report suggests that most of the UK has 3Mb broadband, so do our users just have slow connections? Regardless I have begun some work into improving the perceived page load times, and will roll those requirements into cross cutting requirements in the same way as we do for SEO and DDA compliance. We are going to lighten the page weight and strip out the heavy JQuery that is only used to titillate. We are going to build our own analytics into the front end that will allow us to see in real time what the users experience (current render times etc), we are moving some of the content so that it resides under a new host allowing it to be fetched in parallel by the browser. All of this should help the usrs with slow connections.

But what about the “it crashes my browser” comment? Our in page analytics trap JavaScript errors and reports them. And while our users suffer at the hands of errant JavaScript squirted into the page by 3rd party advert traffickers our own code is solid, so what’s this crash?

We contacted a customer who had left his details and asked him if he could walk us through the “crash” and we would follow along step by step in the office. At the point where he claimed his browser had crashed, we were watching the light box “melt away”, something we had designed in. His expectation was that the light box would work like a tab, and that he could tab between the photos and the detailed specification of the vehicle. Not melt away to the bottom of the screen. So now we will remove the animations on the light boxes (and other objects).

What have I learnt?

Three things:

1. Next project, I’m running the usability testing, with real scenarios and everything.
2. Perceived performance is more damaging than actual performance.
3. BT may have been right…

Hokey Cokey or Hocus Pocus

Back in September 2007 we released a new version of our search application.
The new version was step change for us. At that time we were powering the core of our search offering with an Oracle database, and a Java application that returned flat html. It was all very web 1.0, and we had begun to see issues with the performance of the site and discovered that throwing 8 more servers into the oracle grid didn’t gives 8 x more power. We took the oracle database out of the mix and brought in Endeca search.

The Endeca API allowed us to show the visitors how many of the things they were searching for were available before they submitted the search from. For example, if you were searching for a BMW 5-Series, the fuel type drop down on the search form would list the number available next to the drop down [Petrol (5), LPG (2)]. So a big change to the “build your search and submit it and hope it returns results” model we had previously used. To be able to allow this feature to work we had to use Ajax, or more specifically JSon. So as the user changed their criterion the relevant drop down were updated without refreshing the form. So like I said a step change for the front end, the back end and user behaviour.

The new version was released in stages, inviting visitors to the site to try the new version. This tactic has its own associated problems (for example only a certain type of person will follow a “try our new X” link, so your new application doesn’t get exposure to a good representation of your audience), once the visitors had interacted with the new search form, we invited them to give us some feedback, so that we could improve on what we had we had done. Below is a selection of that feedback:

It crashed 5 times and slow.
Takes longer, too complicated, should be kept simple
Too slow!!
Not as easy to use
Very slow to bring up menus, Spent time waiting.
It doesn’t work – my searches keep spontaneously disapearing (Cars)
is slow. maybe is my broadband problem.
I don’t want to know how many cars in the uk, I just want to know how many in my local area
It’s silly to have to click ‘show your results’ it was better on the previous version where it showed the results.
Too slow in uploads.
More criteria = more time.
Too many details to put in .
More options, as not 100% encyclopaedic knowledge of cars, the sub model option was difficult .


So, pretty damming stuff. But something didn’t make any sense. We had rigorously tested the performance of the system and were confident that it was faster than the old system. The Market leading browse back then was IE6 and given that we had engineered or built it for IE6 it positively flew in Opera or FireFox. So we were perplexed. That is until we did some usability testing (I wont discuss the fact that the usability testing was too late in the project to be really beneficial).

The usability testing did allow us to understand why we got so many “slow” comments on the feedback. Faced with all the feedback the new search form gave the users as they refined their search, they believed two things, 1. That they had to fill in all of the options and 2, that they couldn’t interact with the form until the animated counters stopped moving.

Manufacturer, Model, Variant, Trim, Colour, Fuel Type, Mileage, Age, Min Price, Max Price, Distance from the visitor. As the user slowly changed each of the drop down controls on the search form, some options would become unavailable (greyed out). This was because the back end had contracted them out of the possible results. If no Red BMWs were available, the colour Red would not be available for choice on the drop down control. So the user would change say model to 3-Series and find there wasn’t any Red available on the drop down, so they would back up and change 3-Series to 5-Series and so on. They didn’t realise you could just search for all the Red cars within 20 miles of their house, and drill down from there. To some extent they still don’t 2 years on.

It reminds me a little bit of when I was working on a project with BT and the then new System-X exchanges. The switches could support loads of (then) new features (things we take for granted today, like 1471 in the UK). Being a geek I was amazed at what I could do with a DTMF (touch tone) phone, and went out immediately and bought one. The next day I asked why BT hadn’t publicised any of the features and capabilities. Their response was immediate, dry and serious. “Our users won’t understand them”. I can still remember how I felt, almost like I had stumbled into some great conspiracy. BT wanted to keep people in the dark, and protect them from the nasty technology that might confuse them.
It was several years later that I received a booklet with my phone bill that explained the features and how to access them. Having used the features for sometime at hat point, I had great difficulty in understanding the booklet. Maybe BT were right, maybe it was all too confusing.

Fast forward to now, and my current project. Again another release and another step change. This time the look and feel of the site has been overhauled. The backend is still Endeca powered, but the Java app has been completely rewritten. And in the rewriting of the application we have taken the opportunity to bake testing in from the start. The JavaScript, cascading style sheets and html are all tested automatically. Regression should be a thing of the past (but that’s another blog post), the application has unit testing, functional and non functional testing applied at every check in. The functional testing has been expanded into “User Journey” testing, in which likely user scenarios are played out. All of this happens automatically before the application reaches QA. Then the QA team goes to town, with time to exercise their real skill, exploratory testing. So there you have it, never in the history of our company has a product been so well inspected. So we felt pretty confident when we were ready for Beta.

This time round, instead of us inviting user to try out our new site, we employed AB testing. 5% of our traffic was diverted to the new site, and once again, users were invited to leave feedback. I took the opportunity to setup a Google alert for spotting the use of the beta url in forum or blog posts, so I could keep track of what the community was saying.
Once again the feedback came in…

The used car search, the old one is much clearer to use and a lot better, . The new “improved” one is poor.
Preffered old site looked more proffesional and was easier to use.
The search criteria should be your main focus and keep that in a clear box format like your old site and allow people to search quickly but also as specifically as they want.
The old site is much better the new site is more complicated to use in the end I shut it down and went on to ebay please change back.
It looks much better than the previous website, but since I dont live in UK, I usually have to copy and paste the London postcode from the FAQ page. Unfortunately, I cannot find the page.
Bad design. Not as easy to use and selct options, not as clear and concise. the old one was perfect.

Erm, what? The old one was better? Perfect? Now we are confused.

So again we tackle the perceived issues of our users. We keep seeing comments of missing images, and we start pulling apart the application, the infrastructure and network. It turns out it was an Ad Blocker, that has decided that the way we format our image urls (cache busting) means they are adverts and blocks them.
People complain of slow loading time, so I begin to conduct some testing around that. I conclude they maybe right, so we engage with Gomez to find out for sure. Gomez shows something alarming. Users on a half decent (2Mb and above) broadband connection will get a decent experience. People on anything less are going to be pulling their hair out. The digital Britain report suggests that most of the UK has 3Mb broadband, so do our users just have slow connections? Regardless I have begun some work into improving the perceived page load times, and will roll those requirements into cross cutting requirements in the same way as we do for SEO and DDA compliance. We are going to lighten the page weight and strip out the heavy JQuery that is only used to titillate. We are going to build our own analytics into the front end that will allow us to see in real time what the users experience (current render times etc), we are moving some of the content so that it resides under a new host allowing it to be fetched in parallel by the browser. All of this should help the usrs with slow connections.

But what about the “it crashes my browser” comment? Our in page analytics trap JavaScript errors and reports them. And while our users suffer at the hands of errant JavaScript squirted into the page by 3rd party advert traffickers our own code is solid, so what’s this crash?

We contacted a customer who had left his details and asked him if he could walk us through the “crash” and we would follow along step by step in the office. At the point where he claimed his browser had crashed, we were watching the light box “melt away”, something we had designed in. His expectation was that the light box would work like a tab, and that he could tab between the photos and the detailed specification of the vehicle. Not melt away to the bottom of the screen. So now we will remove the animations on the light boxes (and other objects).

What have I learnt?

Three things:

1. Next project, I’m running the usability testing, with real scenarios and everything.
2. Perceived performance is more damaging than actual performance.
3. BT may have been right…

How will we preserve Twitter, Facebook or LinkedIn?

On Tuesday i attended a talk by Doron Swade at the MOSI for the Computer Conservation Society.

Doron Swade is an engineer, historian, and museum professional, internationally recognized as the authority on the life and work of Charles Babbage, the 19th-century English mathematician and computer pioneer. He was Senior Curator of Computing at the Science Museum, London, for fourteen years and during this time he masterminded the eighteen-year construction of the first Babbage Calculating Engine built to original 19th-century designs. The Engine, was completed in 2002.

Doron was talking about the historical and cultural issues in the history of computing that he faced at the Computer History Museum in Silicon Valley, California. When something struck me.

The big kids on the block today are not the computers but the programmes they run, the software. There hasn’t been any significant advancement in computing hardware for some time. However the internet is changing the way we communicate and socialise and somehow we will need to preserve it for historical interest.

But how on earth will we preserve software like Google, Twitter or FaceBook or MySpace? Because the software that powers these sites is only part of the puzzle, because with these sites its the content that makes these sites what they are. Terabytes of user generated content. How can we preserve that so that in 60 years we can look back with the same fondness we look back at the Manchester Baby, the UNIVAC or the IBM360??

Once you have wrapped your head round that task, who will test such a system? and how will the ensure that its a true representation of what those sites look like today?

While i sat there among some of the early pioneers of British computing who were gently dozing off i wondered if one day, i would be sat in that room, while tomorrows Doron tells me the problems faced with ranking the websites, facebook before twitter, or linkedin before Myspace?

How will we preserve Twitter, Facebook or LinkedIn?

On Tuesday i attended a talk by Doron Swade at the MOSI for the Computer Conservation Society.

Doron Swade is an engineer, historian, and museum professional, internationally recognized as the authority on the life and work of Charles Babbage, the 19th-century English mathematician and computer pioneer. He was Senior Curator of Computing at the Science Museum, London, for fourteen years and during this time he masterminded the eighteen-year construction of the first Babbage Calculating Engine built to original 19th-century designs. The Engine, was completed in 2002.

Doron was talking about the historical and cultural issues in the history of computing that he faced at the Computer History Museum in Silicon Valley, California. When something struck me.

The big kids on the block today are not the computers but the programmes they run, the software. There hasn’t been any significant advancement in computing hardware for some time. However the internet is changing the way we communicate and socialise and somehow we will need to preserve it for historical interest.

But how on earth will we preserve software like Google, Twitter or FaceBook or MySpace? Because the software that powers these sites is only part of the puzzle, because with these sites its the content that makes these sites what they are. Terabytes of user generated content. How can we preserve that so that in 60 years we can look back with the same fondness we look back at the Manchester Baby, the UNIVAC or the IBM360??

Once you have wrapped your head round that task, who will test such a system? and how will the ensure that its a true representation of what those sites look like today?

While i sat there among some of the early pioneers of British computing who were gently dozing off i wondered if one day, i would be sat in that room, while tomorrows Doron tells me the problems faced with ranking the websites, facebook before twitter, or linkedin before Myspace?

If you are not thinking about perfomrance, you are not thinking

Performance testing is a funny old thing, in that whenever the subject comes up, people get all hot and bothered about it. The thing that really tickles my fancy is when developers suddenly get righteous about testing!

Testers and developers have a totally different view of the world. The best testers i have worked with have a real need to dig into systems. Even with black box testing they find a way to work out what a systems does way beyond its simple inputs and outputs. They cant help themselves. It is almost like they cant pass go if they don’t break the system, almost an addiction (or is that affliction).

Now that the developers find themselves writing unit tests, integration tests and acceptance tests they think that overnight they have learnt everything there is to know about testing, right? Wrong!

Yes, sure a developer can write a test, but they often struggle with the intent of the test an more so with non functional testing like performance testing. let me shake it down.

Ok, so the business wants to monetise their existing data by presenting it in a new way, for example “Email Alerts”, you know the sort of thing. You create a search, and when your criteria are met you get sent an email.

The developer sits down to think about performance testing, and thinks about how the system works. In our example here, the system will fire the searches every night, when the database is relatively quiet so that we don’t overload the system during peak hours.

So the developer thinks OK I’ll create a load of these “alerts” using SQL inserts and fire up the system and see how fast it can work through them.

They do just that and get back some statistics like, number of threads, amount of memory the JVM consumed, how many connections to the DB were needed, how many searches were executed, how long it took to execute a search, that sort of thing. They call meetings and stroke their chins in a sage like way. The figures look about right.

But in real life the database would never have the alerts inserted into it in that way. Its probable that users would be inserting data at the same time as it was being read out. Also the product isn’t likely to go live and have 100% take up over night. Its more probable that the take up would be slower, perhaps taking weeks or months never achieving 100% take up. Old alerts would be expiring and some users would renew those while new ones are being created while some other are being edited (change of email address etc).

The crux of the matter is mindset. The tester sits down and thinks, what could go wrong? What happens if the DB is unavailable at the time the batch job runs? What happens if the DB needs to be taken down for maintenance during a batch run, will the batch pick up where it left off? Can the batch job complete before the off peak period comes to an end? Can the mail server handle the number of emails to be sent? What happens to email that bounces? In other words the tester takes a step back and looks at the system holistically. Because a user doesn’t give a damn if your search engine can execute a query in 33ms if they don’t get the email until 12hours after it was relevant.

Now on the current project we have completely rewritten the platform. New version of Java, new styles of writing code, new infrastructure etc etc. The search engine technology is the same, however during the life of the project the API has been updated and a particular feature enabled. This feature allows us to search in a related way. Generally speaking it allows us to do the Amazon type thing; “People who search for X also search for Y”, but it comes at a cost, it takes longer to return the result set (of course it would its got to do another search under the hood).

Again during “testing” the developers just threw as much load as they could muster at the application. But guess what, now its live the figures they got during testing don’t match, not even close.

It isn’t like hadn’t been bleating on about performance for months. I even printed out posters that read “If you are not thinking about performance, you are not thinking” and stuck them above the urinals in the toilets.

Its only now that the team are in the spotlight (its so bright it burns) that they have come to us to ask for our help. Once again we are trying to polish a rough product instead of building the quality in from the start. Once again we cant.

It doesn’t matter a damn that we went all out Agile, TDD and XP if the thing doesn’t perform. The end user doesn’t care that we have continuous integration, they know its damn slow and they vote with their feet (or keyboards/mouse).

If you are not thinking about perfomrance, you are not thinking

Performance testing is a funny old thing, in that whenever the subject comes up, people get all hot and bothered about it. The thing that really tickles my fancy is when developers suddenly get righteous about testing!

Testers and developers have a totally different view of the world. The best testers i have worked with have a real need to dig into systems. Even with black box testing they find a way to work out what a systems does way beyond its simple inputs and outputs. They cant help themselves. It is almost like they cant pass go if they don’t break the system, almost an addiction (or is that affliction).

Now that the developers find themselves writing unit tests, integration tests and acceptance tests they think that overnight they have learnt everything there is to know about testing, right? Wrong!

Yes, sure a developer can write a test, but they often struggle with the intent of the test an more so with non functional testing like performance testing. let me shake it down.

Ok, so the business wants to monetise their existing data by presenting it in a new way, for example “Email Alerts”, you know the sort of thing. You create a search, and when your criteria are met you get sent an email.

The developer sits down to think about performance testing, and thinks about how the system works. In our example here, the system will fire the searches every night, when the database is relatively quiet so that we don’t overload the system during peak hours.

So the developer thinks OK I’ll create a load of these “alerts” using SQL inserts and fire up the system and see how fast it can work through them.

They do just that and get back some statistics like, number of threads, amount of memory the JVM consumed, how many connections to the DB were needed, how many searches were executed, how long it took to execute a search, that sort of thing. They call meetings and stroke their chins in a sage like way. The figures look about right.

But in real life the database would never have the alerts inserted into it in that way. Its probable that users would be inserting data at the same time as it was being read out. Also the product isn’t likely to go live and have 100% take up over night. Its more probable that the take up would be slower, perhaps taking weeks or months never achieving 100% take up. Old alerts would be expiring and some users would renew those while new ones are being created while some other are being edited (change of email address etc).

The crux of the matter is mindset. The tester sits down and thinks, what could go wrong? What happens if the DB is unavailable at the time the batch job runs? What happens if the DB needs to be taken down for maintenance during a batch run, will the batch pick up where it left off? Can the batch job complete before the off peak period comes to an end? Can the mail server handle the number of emails to be sent? What happens to email that bounces? In other words the tester takes a step back and looks at the system holistically. Because a user doesn’t give a damn if your search engine can execute a query in 33ms if they don’t get the email until 12hours after it was relevant.

Now on the current project we have completely rewritten the platform. New version of Java, new styles of writing code, new infrastructure etc etc. The search engine technology is the same, however during the life of the project the API has been updated and a particular feature enabled. This feature allows us to search in a related way. Generally speaking it allows us to do the Amazon type thing; “People who search for X also search for Y”, but it comes at a cost, it takes longer to return the result set (of course it would its got to do another search under the hood).

Again during “testing” the developers just threw as much load as they could muster at the application. But guess what, now its live the figures they got during testing don’t match, not even close.

It isn’t like hadn’t been bleating on about performance for months. I even printed out posters that read “If you are not thinking about performance, you are not thinking” and stuck them above the urinals in the toilets.

Its only now that the team are in the spotlight (its so bright it burns) that they have come to us to ask for our help. Once again we are trying to polish a rough product instead of building the quality in from the start. Once again we cant.

It doesn’t matter a damn that we went all out Agile, TDD and XP if the thing doesn’t perform. The end user doesn’t care that we have continuous integration, they know its damn slow and they vote with their feet (or keyboards/mouse).

Life after ThoughtWorks?

After working with ThoughtWorks I’m completely sold on Agile software development for the type of product we deliver. Which if you know me may be a bit of a surprise as I have a reputation for being a cynical old curmudgeon, and a staunch doubter to boot.

While I am still unsure if agile would work for large scale software projects (think air traffic control) I am certain that there are agile practices (lean for example) that could be pilfered and applied to those lumbering projects too. If you are reading this and you haven’t had any exposure to Agile, I guess the biggest thing I should tell you is that this isn’t some process you just pick up and run with, nor is it a methodology to apply. You can’t learn it from a book, or by a project template; it is quite simply a mindset and as such it presents the practitioner with quite a shift in paradigm. Simply put, you can’t flick a big agile switch and hey presto you are Agile; it is more subtle than that.

I think before I continue it’s important that I clarify something. The term agile is applied to a wide variety of processes, techniques, methods, tools, practices, projects, and phases of the development life cycle; it has become a buzzword used by people trying to paint their work in a new light (who doesn’t want to be known as being agile?). It’s important, therefore, to set out some basic definitions and context for the use of the term “agile,” especially as i will use it constantly throughout this article.

Within the context of software development, the term “agile” (with a small “a”) is meant to imply that the development team is nimble, flexible and responsive to the business needs, and that it is able to adopt new technologies and techniques that can improve software delivery. The term “Agile” (with a capital “A”) refers to a very specific set of processes (and i use the term process as more of a place holder) applied to software development that have evolved over the past fifteen years or so; including some you have probably heard of, like eXtreme Programming (XP), Scrum, Feature-driven Development (FDD), Crystal, Dynamic Systems Development Method (DSDM) and Lean Software Development. A non-profit organisation The Agile Alliance was created by the ideas people behind most of the Agile processes. The Agile Alliance promotes a set of core values that a process must follow to be called Agile:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

So to be Agile then, a process must support these values (and more), albeit in diverse ways. Some, processes for example like Scrum, address team management, while others such as XP or DSDM, address development activities or other activities of the software development life cycle. It’s worth making a mental note that users of Agile processes do not have to follow all its Agile practices, and neither does the use of one process preclude the use of any other. One thing I learnt is that Agile supports such process change, if a particular way of working is not working then change. In fact many I found that many of the Agile practices are complimentary.

No matter your preference, all of the different flavours of Agile will deliver working functionality in short time-boxed iterations. They implement early and frequent testing. They require lots of involvement from the customer on a frequent if not full-time basis and they assume that the customers requirements will continually change.

So why all the fuss about big “A” little “a” when talking about Agile? There are two main reasons:

  1. Companies who adopt Agile processes have to be prepared to completely change the way they not only develop software but how they think, and this change is so big it is very hard to do. However companies that have achieved this shift will appreciate the significant benefits they can reap in productivity, quality and value of the software that they deliver. Notice i didn’t say it would be faster. These companies are Agile with a capital A.
  2. Companies that are not able to embark on this level of change can still become more reactive and flexible in the ways that they build software. They become more agile and begin to realise the advantage that Agile can deliver. This toe dipping exercise can lead to a true Agile team, however the company must understand that it’s not the prefferable way, and they would benefit from the gung-ho approach to Agile adoption.

To reiterate what i’ve said, at a minimum, every Agile process delivers working functionality in short, time-boxed iterations. Agile implements early and frequent testing, and it involves the customer on a frequent if not full-time basis; and assumes that requirements cannot be fully defined at the start of a project, and that the requirements will continually change. The ethos is simple that by using these practices, the development teams will be able to respond quickly to the ever changing customer priorities and feedback, and deliver value to the business. This is often missunderstood as being quicker, I would prefer to describe it as improved time to benefits.

As a tester I have become quite accustomed to being involved late in a project, often right before delivery. I have adapted and learnt how to cope with shortened time frames for testing, and receiving specifications and requirement documentation that don’t actually match the product delivered in QA. Typically the business (or for that matter the project team) usually has little interest in the test teams input, that is until they call us a bottle neck.

Feeling loved and appreciated.

Agile software development for a tester is radically different from the traditional PRINCE(2)/waterfall software development lifecycle (SDLC), because it throws QA right into the heart of project on day one. As Agile testers we suddenly found ourselves being involved in the analysis and design of the product. We became heavily involved in decision-making throughout the whole project and because the delivery of the software is incremental we found that we began testing at the very beginning of the project and that we had to maintain pace and keep step with development to prevent any delay. This is a far cry from waiting for a software deliverable (possibly unfit for purpose) to be thrown over the fence a few weeks before go live.

The paradigm shift I have mentioned is a large one, not just for QA but the whole project team, because our QA team now drives the entire software development process.

Quality assurance, with its focus on preventing defects, is translated into the agile practice of having committed QA resources on the development team that participate in decision-making on a daily basis, throughout the life cycle of the project.

Their input during elaboration and design helps developers write better code. More “what-if” scenarios are considered and planned for, as the collaboration between coders and testers gives the coders more insights than if they were to have planned the work on their own.
Likewise, the testers gain added insight into the expected functionality from the coders and the product owner, and are able to write more effective test cases for the product.
Quality control places its emphasis on finding defects that have already slipped into the system, and working with developers to eliminate those defects. This bug-checking is done within the iteration, using such techniques such as daily builds and smoke tests, automated regression testing, unit testing, functional and exploratory testing, and acceptance testing. Everyone participates – no one is exempt from the tasks of ensuring that the feature coded meets the customer’s expectations.

Your role morphs and evolves.

Through a methodology known as “Story Test Driven Development” the test requirements (aka the acceptance tests) are captured (we currently use Twist) in a test like format and they are then augmented to make them into automated tests. Nothing unusual in that, lots of test teams create automated tests, however here the automated tests are being executed by the development team not the test team and the tests exist before the development team have even created the code for the software the team is delivering. The real beauty of this method is that the development team can then integrate the automated tests into a Continuous Integration environment, where the newly checked in code is built and tested automatically. So the QA teams test suite is run against the code every time a change gets checked in. But wait, that’s not all. The delivery of code into QA can not happen until it has passed all of our tests, which means that we have built the quality in from the very start. Let me state that again, in case you missed it. The development team can not deliver code into QA until it has passed the QA teams tests. That’s a statement that should raise a lot of internal debate with any passionate tester who hasn’t worked in this way before, and it did with us. Its worth me making the distinction here that Test Driven Development (TDD) is not a testing methodology it’s a design methodology. Being testers we have a very different view of the world to a developer, its in very our nature. Working this way felt like we had harnessed our power and put it where it belongs, under the smelting pot of code.
I should also mention here that the developers were also working in a new way, because they worked in pairs (pair programming) and also used TDD at unit level. This means that a developer has to write a unit test before he writes the code for the unit being developed. That means then, that before its delivered to QA, our code has been tested twice. That’s two times tested.

So if development is running the QA teams tests, and the QA team are writing the automated tests (and its blurry line between coding and writing an automated test), and the code is tested before its delivered to QA, a whole load of question begin to bubble up.

Some of early questions we had:

  1. Who tests the testers’ tests?
  2. This is agile, so what happens if the requirements change?
  3. So what do QA test if it’s already passed the QA tests?

On a typical PRINCE(2)/waterfall project the testing team plans as far in advance as it can. We normally try to follow the IEEE 829-1983 standard and document as much as we can.
The documentation covers how we approach the testing and detail the testing activities. These documents are usually created in isolation from the project team and may be published for approval. However in my experience the documents are rarely scrutinised and any feedback given only serves to pay lip service to the processes. Another checkbox checked, and another Gant chart milestone met.

Working in an agile testing environment also has a requirement to define which tools and methods will be used for writing, executing, and reporting tests and determine the best approach to testing, and the scope of that testing. The big difference is that the whole team is engaged in this deterministic process and we found that it was important to engage the developers in this definition, because they would be executing our tests and writing their own unit tests. Moreover thought had to be given to automating the regression testing, something that would happen as part of the continuous integration process.
The business stakeholders were also involved in this process (unfortunately only by proxy through the business analysts, as they are physically located in a different part of the country) as they would help to define and run the acceptance tests. In agile, we (the whole team) all test, but the business accepts.

In short within agile practices, everyone has a contributory part in defining, upholding, and improving the quality of the product.

One of the gotchas was that we found that we needed to become more technical. We had thought that we were already more technically skilled than your average tester, however as we endeavoured to automate our testing we found that we had to skill up and learn not just how to write code (Java for us), but compile the code and version control it. They were steep, steep learning curves, which now we have overcome them, have empowered us, giving us the tools for a brighter, faster and more accurate future. Having said that, it does appear that some of this coding effort could be considered a one off setup task, because we now have a framework of “tools” that cover all of the tasks we need to help us execute an automated test. Couldn’t we have asked the development team to do the tech tasks for us? We did, and they didn’t have any resource free to accommodate our needs as well as those of the agile project. Regardless, we have skills now that we have been able to share with the wider QA team, and they too are seeing the fruits of our initial labour.

Traditional Tools Solve Traditional Problems in Traditional Contexts. Agile Is Not Traditional.

Traditional, heavyweight, record-and-playback tools (like Quality Center) address the challenges faced by teams operating in a traditional context with specialisms. They address the challenge of having non-programmers automate tests by having record-and-playback features, a simplified editing environment, and a simplified programming language.

But Agile teams don’t need tools like these (optimised for non-programmers). What Agile test teams need are tools to solve an entirely different set of challenges that are related to collaborating, communicating, reducing the waste (Muda), and decrease the feedback loop. Ergo, traditional (long standing) test automation tools just don’t cut the mustard in an Agile context because they are designed to solve traditional problems, in traditional contexts and those really are quite different to the challenges faced by Agile test teams. To make it clear, QC & TD isn’t going to cut it in Agile.

At the Google Tech Talks December 9, 2005 Elisabeth Hendrickson gave a talk on how as more teams are adopting Agile practices such as XP and Scrum, software testing teams are being asked to become “Agile” as well… View it here