The magic medallion of testing - summary of a keynote by Rex Black

This is a summary about Rex Black’s keynote about Testing and Quality management at Nordic Testing Days 2012. This was the first keynote on the second day. People were still tired from the evening activities and some even skipped the keynote altogether. I was keen to see Rex’s track as I had some… opinions about ISTQB and the certification system as a whole. So, to be there and be able to challenge him was an opportunity I didn’t want to miss.

On Monday we had "argued" with Rex about metrics and coverage. Rex commented Sami Söderbloms statement about not measuring coverage by giving an example. The discussion went something like this: “With all things being equal, the first tester returns his test report with 90% coverage and another with 15% coverage. Which one would you trust more?” Rex commented. I raised my hand (the inner voice screaming "Me, me, me! Let me answer!". Sami then said that I could answer the question. I tried to question the true coverage by asking about what has been found. Rex replied by pleading to the all-things-being-equal statement. All things that were found were supposed to be equal (so as important bugs, issues, etc.) “But all things are never equal in software testing”, I said. Later I realized that if all things were equal, how can there be a difference in the coverage?

If we have all things equal, isn’t the coverage supposed to be equal too? If two clone testers are put to test a product in exactly the same way, don’t they both produce the same amount of coverage in any way you measure? Change anything and all things are not equal. So if in an all-things-being-equal-situation we have difference in coverage, our model to measure testing might be faulty, the person interpreting the results may be biased, etc. So how can one make an example from a situation that is faulty by default?

The keynote however began with Rex warning about using the c-word (coverage) and the m-word (metrics). I immediately knew this wasn’t going to be pretty. ;)

Rex gave a few cases where a manager was oblivious about the need for skilled testers. One manager had supposedly said something like this: “The requirements of a tester: heartbeat, respiration.” That struck home right then. I knew these managers and product owners exist! I had been in a project where the solution to testing was “ask people around if they have the time”. I then asked around and gathered as good a bunch of people I could find… and I tried to train them to be adequate testers. Some became good testers, some didn’t, but I had a team.

Feels like the requirements for a tester in fact are the aforementioned heartbeat and respiration. In some cases, though, also a certification might be squeezed. If we find a tester with a certification, one has to be a good tester! Right? Rex mentioned in the talk (which I already knew from previous experience with the ISTQB) that people who certify themselves with the foundation level certification do not need any skills from testing before the certification. It is “an entry-level certification”, Rex says. So how come people with no skills are certified as testing professionals? This leads to a situation where unskilled, untalented testers that look better to managers than people with REAL skills and attitude towards testing. I that what it is suppose to be doing?

The certification thing got me so winded up that I almost missed the rest of the keynote while trying to douse my building rage. (Just kidding! ;) ) Rex had some good thoughts about how to veer the Agile testing more towards intelligent testing – away from “Unit testing + Acceptance testing = Proper testing” an more towards testing throughout the life cycle. Automated testing is there only to support manual testing*. Usually Agile testing is a Waterfall within the Agile cycle where testing is done at the end of the cycle (if we have the time). Testing earlier, reviewing the deliverables in time and early enough, challenging designs, etc. – not just automating unit tests.

Rex also talked about the cost of bugs, but there was no ground breaking new information about that. I have read that from the ISTQB syllabus a while ago. And I still believe that counting bugs and measuring them against each other will lead into biased view that “all bugs are equal”. James Bach said it well once that “bugs should be thought as unicorns” How many unicorns fit into a cubicle? Two?) How many bugs are in a product? I know this is generalization from Rex’s part but I think it dangerous to make such claims as (like I mentioned earlier about the decision-making skills of a manager**) people WILL make the tools out of the metrics to compare PEOPLE against each other. "I found 4 bugs and he only found 3. I'm better, right?"

Rex also talked about the quality management. He mentioned that good testing doesn’t make good quality in its own. I agree. We need to be able to start making good quality from the top, starting with directors. We also need to be able to improve the processes, tools, skills and communication to make better quality. We can do it all the time while we work by challenging bad behavior and encouraging good behavior.

Anyway, I feel that the foundation level certification is not currently building the community towards good testing. It has become a magic medallion that is bought from the wandering gipsy saleswoman. And when the kings and queens see the medallion they say “This must be a skillful knight as he has the Badge of Knighthood”. And yes, the medallion is bought, not earned. The training courses for the foundation level solely aim to give answers to the exam questions. They do not encourage people to learn, but to memorize (the worst thing is that they might say that one doesn’t need this information in practice). Rex said that when his consultancy company holds these courses they focus on practical use of the things trained. I wish I could once attend one of his company’s courses to see how it is done.*** The whole certification issue might need a blogpost of it's own, but it will have to wait. Some people have already covered that for me.****

You might get a message that I didn’t like Rex or his keynote. I did in fact, and I encourage all of you to read Rex’s blog and the books he has written. I don’t say I agree with all he writes but they give you a lot to think about. Rex as a person is (to me) "a happy giant"*****. What I liked most about him is that he really listens to what you have to say. We had inspiring conversation with Rex after the keynote although I challenged him about the ISTQB. I hope to hear him talk again at some other event. And I was bummed that he couldn't join my workshop... :(


*) And to build confidence, some say. But, like Michael Bolton said in the Rapid Software Testing course: “Were not in the confidence building industry – we’re in the false confidence demolition industry!”
**) Says a guy who spent two years as a test manager and is currently a maintenance manager deputy.
***) Do you dare to invite me to a course, Rex? ;)
****) Jari Laakso's post http://jarilaakso.blogspot.fi/2012/02/test-is-dead-vs-istqb-kills-people.html
*****) He's like 7 feet tall.

Porridge is bad for you!

"What?! What is he talking about?" This blog post is about challenging one's criticism and trying to make it more effective using simple methods. This post is about moving away from unhealthy criticism into constructive critique and feedback. This is about finding the right mind-set.

Why do people criticize?

"You're doing it all wrong!" "Your tie is hideous." You've seen it. You've heard it. Probably you've even done it. I know I have. And in most occasions I have let my self-criticism be clouded by someone else’s opinion about the same subject. By not having a concrete opinion of one's own it's easy to adapt to criticism of someone else.

Let's take an example of a guru that has strict criticism against one subject - a porridge (seems like a neutral enough). We all know that porridge is good for us, right? What if a nutrition guru says that porridge is bad? What do we do? If we have huge respect on that guru's thoughts and doings, might we blind our own judgment with the upward gaze towards the guru? We might just take the guru's opinion granted and start proclaiming that porridge is bad for you.

One way out of this bad equation is to step back and challenge one’s own critique. Is it justified? Are you making an assumption? (Great blog post about defeating assumptions by Ilari Henrik Aegerter) What if the guru was wrong? After making sense of what do YOU think, then you should back the critique up with facts, not opinions. By finding the facts, you might be looking for biased facts, but at least you have something to back it up. It's obvious (to some) that a biased opinion should stay as an opinion, but they rarely do. Instead of being opinions they become statements supported by chosen facts.

Doing basic critique, source checking, challenging, context projecting, you can find the root cause of the guru's opinion about the porridge. Does he (our guru is a he today) have an agenda of his own? Are there hidden meanings in the critique itself? Does it provoke thinking instead of criticizing the product?

How do people criticize?

"When giving feedback, do it like so: Always give good feedback in public and be precise about what was done well. Always give negative feedback in private and be precise. Try to find the solution instead of the one to blame." This was said by my father who has decades of experience in management and leading people. I have always thought this as the fundamental guideline of critique. I think most people know this and agree with this, but how come most people don't act accordingly?

Let's say that the guru had discovered some facts that "Ye olde bran porridge" has all sort of chemicals in it that disable some growth hormones on a child. Obviously that's a statement to be told to the public, right? And as we hold the guru in high regard, he is mandated to present his opinion (possibly supported by facts) in some public media. There are channels in which you can present a complaint about food (health inspector or some kind of an agency) and they will take the necessary precautions to tell the public that "porridge is bad for you". Possibly they have first discussed with the porridge company, who might have taken the product off the market.

The guru might give criticism about porridge in public and have the wrath of the porridge company on his shoulders. He might not care as he's a nutrition guru and has an agenda of his own (hoes he?). Is the guru doing the right thing expressing his opinion so loudly in public? Is the guru promoting himself instead of giving critique?  Was the bad thing in the porridge, in the chemical, or in the company making the porridge?

Where's the difference in the approach between the two models of critique? Was the guru able to achieve the goal of his criticism through a public channel (which ever the goal might have been)? Was the "behind closed doors" critique more efficient than the "in your face" critique? They all depend on the context, obviously. What was the goal of the critique?

Feed-forward

Some people think critique is feedback. Well it kinda is in some extent. Feedback however can be constructive even when the feedback is negative. Feedback is given when someone asks for it; critique is given when ever. Feedback is not trying to make one feel happy/sad but to make them improve; critique is about making a statement. When giving feedback don't sugarcoat it, instead say what YOU like and you'd like to see improved. "I liked the taste of porridge and how my stomach feels afterwards. To make it even more healthy I would not put in the chemicals that prohibit my growth."

As the feedback is a kind of a thing to be asked for, critique is the kind of a thing you just blurb out. Feedback has a purpose and it is meant to improve the one receiving the feedback. Critique has the tendency to provoke something. Conversation, debate, hatred, etc. Challenging can be more effective a way than critique. Challenging the critique itself can become the most valuable feedback there is!

Is the content self-justifying or do we need to empathize to support the critique?

There are tons of guides in how to give feedback without being critical. I know a dozen occasions where I have let my judgment be clouded by numerous things that have lead into bad critique and undesired results. Here's one:

I try to promote intelligent testing and intelligent approach to quality in general. I also believe that certifications that focus on the certificate itself are no good. A certificate that focuses on skills in a field that requires skills is a good thing; artificial certificate concentrating on a narrow view about best-practices (and only knowledge thereof) is a bad thing. This is what I thought and still think. I was having a discussion with people I think highly of about “what is your opinion about ISTQB-provided series of certificates”. By making a comment that the certificate looks good on paper (with some unflattering spices), I provoked a series of questions about “how do I back up my statement” and "do you even know what you're speaking of".

The questions stuck home and I started to think of how I really thought about the issue. The fundamental thought behind the issue remains the same, but as I have not delved deep enough into the syllabus, the history thereof, the initial goal behind the syllabus. Am I eligible to make claims about the issue? Was I repeating what the other people were saying and them making myself feel important about myself by making a rash claim? Does my opinion really matter in this case and could I do some good without being so loud about it? Is there a possibility to raise conversation about the issue within the certificate organization without sounding like a zealot?

With the comment I made (which was criticism at its worst) I thought the content of the comment was self-justifying. "Obviously all the people were thinking the same so I just said it out loud." Even though some of them were, they rightfully challenged my comment and forced me to think about it. What was my goal when stating something like that? What was the desired outcome? Praises to me? More Twitter followers? To raise conversation? To sound like a dumb-ass?

What I did achieve with the comment was for me to be able to criticize my own behavior and claims. I once claimed (in Finnish) that one way to achieve the best quality of an end-product is to "Murder you darlings" - by finding the most direct route from the current point into the desired point. By removing all the excess and self-promotion from the content. To go directly towards results. In making the comment I a was focusing on "sounding cool" instead of trying to use the words as a tool to achieve a goal (which apparently was shrouded).

Did I hurt someone in the process? Can't tell. Not directly, I assume (Pekka, you're assuming things).

Did I achieve the goal? Can't tell. I wasn't aware that there was a goal.

Did I learn something? Oh boy, did I!? ;)