This is a summary about Rex Black’s keynote about Testing and Quality management at Nordic Testing Days 2012. This was the first keynote on the second day. People were still tired from the evening activities and some even skipped the keynote altogether. I was keen to see Rex’s track as I had some… opinions about ISTQB and the certification system as a whole. So, to be there and be able to challenge him was an opportunity I didn’t want to miss.
On Monday we had "argued" with Rex about metrics and coverage. Rex commented Sami Söderbloms statement about not measuring coverage by giving an example. The discussion went something like this: “With all things being equal, the first tester returns his test report with 90% coverage and another with 15% coverage. Which one would you trust more?” Rex commented. I raised my hand (the inner voice screaming "Me, me, me! Let me answer!". Sami then said that I could answer the question. I tried to question the true coverage by asking about what has been found. Rex replied by pleading to the all-things-being-equal statement. All things that were found were supposed to be equal (so as important bugs, issues, etc.) “But all things are never equal in software testing”, I said. Later I realized that if all things were equal, how can there be a difference in the coverage?
If we have all things equal, isn’t the coverage supposed to be equal too? If two clone testers are put to test a product in exactly the same way, don’t they both produce the same amount of coverage in any way you measure? Change anything and all things are not equal. So if in an all-things-being-equal-situation we have difference in coverage, our model to measure testing might be faulty, the person interpreting the results may be biased, etc. So how can one make an example from a situation that is faulty by default?
The keynote however began with Rex warning about using the c-word (coverage) and the m-word (metrics). I immediately knew this wasn’t going to be pretty. ;)
Rex gave a few cases where a manager was oblivious about the need for skilled testers. One manager had supposedly said something like this: “The requirements of a tester: heartbeat, respiration.” That struck home right then. I knew these managers and product owners exist! I had been in a project where the solution to testing was “ask people around if they have the time”. I then asked around and gathered as good a bunch of people I could find… and I tried to train them to be adequate testers. Some became good testers, some didn’t, but I had a team.
Feels like the requirements for a tester in fact are the aforementioned heartbeat and respiration. In some cases, though, also a certification might be squeezed. If we find a tester with a certification, one has to be a good tester! Right? Rex mentioned in the talk (which I already knew from previous experience with the ISTQB) that people who certify themselves with the foundation level certification do not need any skills from testing before the certification. It is “an entry-level certification”, Rex says. So how come people with no skills are certified as testing professionals? This leads to a situation where unskilled, untalented testers that look better to managers than people with REAL skills and attitude towards testing. I that what it is suppose to be doing?
The certification thing got me so winded up that I almost missed the rest of the keynote while trying to douse my building rage. (Just kidding! ;) ) Rex had some good thoughts about how to veer the Agile testing more towards intelligent testing – away from “Unit testing + Acceptance testing = Proper testing” an more towards testing throughout the life cycle. Automated testing is there only to support manual testing*. Usually Agile testing is a Waterfall within the Agile cycle where testing is done at the end of the cycle (if we have the time). Testing earlier, reviewing the deliverables in time and early enough, challenging designs, etc. – not just automating unit tests.
Rex also talked about the cost of bugs, but there was no ground breaking new information about that. I have read that from the ISTQB syllabus a while ago. And I still believe that counting bugs and measuring them against each other will lead into biased view that “all bugs are equal”. James Bach said it well once that “bugs should be thought as unicorns” How many unicorns fit into a cubicle? Two?) How many bugs are in a product? I know this is generalization from Rex’s part but I think it dangerous to make such claims as (like I mentioned earlier about the decision-making skills of a manager**) people WILL make the tools out of the metrics to compare PEOPLE against each other. "I found 4 bugs and he only found 3. I'm better, right?"
Rex also talked about the quality management. He mentioned that good testing doesn’t make good quality in its own. I agree. We need to be able to start making good quality from the top, starting with directors. We also need to be able to improve the processes, tools, skills and communication to make better quality. We can do it all the time while we work by challenging bad behavior and encouraging good behavior.
Anyway, I feel that the foundation level certification is not currently building the community towards good testing. It has become a magic medallion that is bought from the wandering gipsy saleswoman. And when the kings and queens see the medallion they say “This must be a skillful knight as he has the Badge of Knighthood”. And yes, the medallion is bought, not earned. The training courses for the foundation level solely aim to give answers to the exam questions. They do not encourage people to learn, but to memorize (the worst thing is that they might say that one doesn’t need this information in practice). Rex said that when his consultancy company holds these courses they focus on practical use of the things trained. I wish I could once attend one of his company’s courses to see how it is done.*** The whole certification issue might need a blogpost of it's own, but it will have to wait. Some people have already covered that for me.****
You might get a message that I didn’t like Rex or his keynote. I did in fact, and I encourage all of you to read Rex’s blog and the books he has written. I don’t say I agree with all he writes but they give you a lot to think about. Rex as a person is (to me) "a happy giant"*****. What I liked most about him is that he really listens to what you have to say. We had inspiring conversation with Rex after the keynote although I challenged him about the ISTQB. I hope to hear him talk again at some other event. And I was bummed that he couldn't join my workshop... :(
*) And to build confidence, some say. But, like Michael Bolton said in the Rapid Software Testing course: “Were not in the confidence building industry – we’re in the false confidence demolition industry!”
**) Says a guy who spent two years as a test manager and is currently a maintenance manager deputy.
***) Do you dare to invite me to a course, Rex? ;)
****) Jari Laakso's post http://jarilaakso.blogspot.fi/2012/02/test-is-dead-vs-istqb-kills-people.html
*****) He's like 7 feet tall.
