Wednesday, 4 July 2012

The magic medallion of testing - summary of a keynote by Rex Black

This is a summary about Rex Black’s keynote about Testing and Quality management at Nordic Testing Days 2012. This was the first keynote on the second day. People were still tired from the evening activities and some even skipped the keynote altogether. I was keen to see Rex’s track as I had some… opinions about ISTQB and the certification system as a whole. So, to be there and be able to challenge him was an opportunity I didn’t want to miss.

On Monday we had "argued" with Rex about metrics and coverage. Rex commented Sami Söderbloms statement about not measuring coverage by giving an example. The discussion went something like this: “With all things being equal, the first tester returns his test report with 90% coverage and another with 15% coverage. Which one would you trust more?” Rex commented. I raised my hand (the inner voice screaming "Me, me, me! Let me answer!". Sami then said that I could answer the question. I tried to question the true coverage by asking about what has been found. Rex replied by pleading to the all-things-being-equal statement. All things that were found were supposed to be equal (so as important bugs, issues, etc.) “But all things are never equal in software testing”, I said. Later I realized that if all things were equal, how can there be a difference in the coverage?

If we have all things equal, isn’t the coverage supposed to be equal too? If two clone testers are put to test a product in exactly the same way, don’t they both produce the same amount of coverage in any way you measure? Change anything and all things are not equal. So if in an all-things-being-equal-situation we have difference in coverage, our model to measure testing might be faulty, the person interpreting the results may be biased, etc. So how can one make an example from a situation that is faulty by default?

The keynote however began with Rex warning about using the c-word (coverage) and the m-word (metrics). I immediately knew this wasn’t going to be pretty. ;)

Rex gave a few cases where a manager was oblivious about the need for skilled testers. One manager had supposedly said something like this: “The requirements of a tester: heartbeat, respiration.” That struck home right then. I knew these managers and product owners exist! I had been in a project where the solution to testing was “ask people around if they have the time”. I then asked around and gathered as good a bunch of people I could find… and I tried to train them to be adequate testers. Some became good testers, some didn’t, but I had a team.

Feels like the requirements for a tester in fact are the aforementioned heartbeat and respiration. In some cases, though, also a certification might be squeezed. If we find a tester with a certification, one has to be a good tester! Right? Rex mentioned in the talk (which I already knew from previous experience with the ISTQB) that people who certify themselves with the foundation level certification do not need any skills from testing before the certification. It is “an entry-level certification”, Rex says. So how come people with no skills are certified as testing professionals? This leads to a situation where unskilled, untalented testers that look better to managers than people with REAL skills and attitude towards testing. I that what it is suppose to be doing?

The certification thing got me so winded up that I almost missed the rest of the keynote while trying to douse my building rage. (Just kidding! ;) ) Rex had some good thoughts about how to veer the Agile testing more towards intelligent testing – away from “Unit testing + Acceptance testing = Proper testing” an more towards testing throughout the life cycle. Automated testing is there only to support manual testing*. Usually Agile testing is a Waterfall within the Agile cycle where testing is done at the end of the cycle (if we have the time). Testing earlier, reviewing the deliverables in time and early enough, challenging designs, etc. – not just automating unit tests.

Rex also talked about the cost of bugs, but there was no ground breaking new information about that. I have read that from the ISTQB syllabus a while ago. And I still believe that counting bugs and measuring them against each other will lead into biased view that “all bugs are equal”. James Bach said it well once that “bugs should be thought as unicorns” How many unicorns fit into a cubicle? Two?) How many bugs are in a product? I know this is generalization from Rex’s part but I think it dangerous to make such claims as (like I mentioned earlier about the decision-making skills of a manager**) people WILL make the tools out of the metrics to compare PEOPLE against each other. "I found 4 bugs and he only found 3. I'm better, right?"

Rex also talked about the quality management. He mentioned that good testing doesn’t make good quality in its own. I agree. We need to be able to start making good quality from the top, starting with directors. We also need to be able to improve the processes, tools, skills and communication to make better quality. We can do it all the time while we work by challenging bad behavior and encouraging good behavior.

Anyway, I feel that the foundation level certification is not currently building the community towards good testing. It has become a magic medallion that is bought from the wandering gipsy saleswoman. And when the kings and queens see the medallion they say “This must be a skillful knight as he has the Badge of Knighthood”. And yes, the medallion is bought, not earned. The training courses for the foundation level solely aim to give answers to the exam questions. They do not encourage people to learn, but to memorize (the worst thing is that they might say that one doesn’t need this information in practice). Rex said that when his consultancy company holds these courses they focus on practical use of the things trained. I wish I could once attend one of his company’s courses to see how it is done.*** The whole certification issue might need a blogpost of it's own, but it will have to wait. Some people have already covered that for me.****

You might get a message that I didn’t like Rex or his keynote. I did in fact, and I encourage all of you to read Rex’s blog and the books he has written. I don’t say I agree with all he writes but they give you a lot to think about. Rex as a person is (to me) "a happy giant"*****. What I liked most about him is that he really listens to what you have to say. We had inspiring conversation with Rex after the keynote although I challenged him about the ISTQB. I hope to hear him talk again at some other event. And I was bummed that he couldn't join my workshop... :(

*) And to build confidence, some say. But, like Michael Bolton said in the Rapid Software Testing course: “Were not in the confidence building industry – we’re in the false confidence demolition industry!”
**) Says a guy who spent two years as a test manager and is currently a maintenance manager deputy.
***) Do you dare to invite me to a course, Rex? ;)
****) Jari Laakso's post
*****) He's like 7 feet tall.


Sami Söderblom said...

Hi Peksi,

I think it was the best choice for me to sleep a bit longer and enjoy those delicious breakfast bacons. I don't want to start my day, especially a bit hangover infested one, by building up my rage...

Firstly, a gigantic thanks for blocking Rex's daggers for me! Even though I know and feel it in my heart that all-things-being-equal is complete bullshit, I wasn't prepared with convincing arguments. I'm glad you were. So thank you!

Secondly, nice analysis on the dangers of certifications! Even though I can dodge them, I still have to use so much energy to them. The reason I'm certified in ISTQB and TMap is that I can pass some sort of quality gate and get to do things I'm good at and which actually create added value. It's like using Lindsay's stealth mechanics to divert and distract the ignorant management so that work can be done. But again, too much energy has to be put into it.

But luckily this isn't a permanent state. People are growing weary of the things that don't work and are seeking alternatives. Even managers do. We just have to endure a little while longer and continue to do our work, and be loud. The more we write blogs/articles/books, speak at various, create communities, share our knowledge and of course produce the more we dilute the dark side from the equation.

May the force be with you! :D

Pekka Marjamäki said...

Hi Sami!

I have grown to use the ISTQB as a test for those who try to recruit me. If they say things like "it's good that you're certified" I know they don't have enough knowledge in recruiting skilled testers. One headhunter from some British company started the phone interview with "I see that you have the ISTQB certificate, which is a good thing". Nothing good came from that. :D

I do think that the Advanced Software Testing books are worth reading as you might get some pointers from those books and processes that can be used in some context. If you try to implement the whole ISTQB model to a project/company you are heading towards the edge of the canyon - a little while it's all good and fun, but then comes the drop. So do things that support the context you're working in!

- Peksi

Rex Black said...

Greetings Pekka (and Sami)--

First off, I enjoyed talking with you both at the Nordic Testing Days conference. Even though we disagree on a few topics, it's good to find people who can separate professional opinion differences from personal differences. I enjoyed discussing those professional opinion differences with you, too, because, as you said, Pekka, we were able to talk to each other, and listen to each other, rather than talking past each other, as is all too often the case in our profession.

Now, to the substance, I want to briefly address two points. The first involves both the "c-word" (coverage) and the "m-word" (metrics). :-) Let me expand on my "all things being equal" comment at the conference, so you can see the point I'm making.

Scenario 1: Suppose that a professional tester spends forty hours testing an application. He applies all appropriate black box, white box, defect based, and experience based testing techniques. He finds 100 bugs, which he describes in appropriate and detailed bug reports. He goes to his manager and tells her, "I am finished testing, here are my results, here are the bugs I found, and the following is my conclusion."

Scenario 2: Now, let's repeat scenario 1 exactly, but this time suppose that the tester keeps track of which tests have been run against each of the requirements spcification elements (or user stories if you're in an Agile world). And this time, when the tester goes to the manager, he also delivers coverage analysis that shows 100% coverage of the requirements.

In which scenario can the manager have a higher level of confidence that the testing is complete, that the test results are complete, and that the tester's conclusion is correct? Clearly, in scenario 2, yes?

Second, regarding certification. It's important to separate what the ISTQB is trying to accomplish from some of the mistakes that a few people might be making in their use of the program. At the Foundation level, the ISTQB is setting down basic terms, techniques, and concepts that everyone involved in testing should know and understand. In our (RBCS's) courses, we go beyond that and have people actually apply those concepts to a realistic sample project. I wish all ISTQB training providers would do the same thing.

At the Advanced level, the focus is much more on analyzing situations, planning, creating test cases, and other such hands-on skills. The Expert level goes even beyond that. These certifications require demonstrated experience as testers.

What the ISTQB is doing is building a framework, a structure, for what knowledge, skills, and abilities people should have at various stages in their testing careers. Someone who has the Foundation certification has taken a first step--an important step, but only a first step--on that path. To build on your metaphor, Pekka, the tester with a Foundation certification is not a knight; she is a squire. The tester with an Advanced certification is a knight.

Is there misuse of the certification by lazy recruiters and hiring managers? Yes, Pekka and Sami, I've heard stories like yours, and I believe them. But there is also appropriate use of the certification by recruiters and managers, too, and I've seen that myself.

Ultimately, the ISTQB program is constructed by people--just like software--and, as such, it has a few bugs in it. However, the overall intent is to advance the profession of software testing by collecting useful and interesting ideas and techniques, sequencing them in a career path, and supporting an infrastructure of books, articles, blogs, conferences, webinars, training, and exams. As such, it is on balance a good thing which moves our profession forward. I'd personally like to see smart people like you (and Sami) get involved to help us make it even better.

Huib Schoots said...

Hi Rex,

Your example of the coverage maybe sounds okay the first time you hear it, but what does 100% requirement coverage /really/ mean? The requirements are NOT the requirements document. So 100% requirement coverage, to me, doesn't give a higher level of confidence. It very much depends on the mission, the risks involved, etc. Let's take a extreme situation to clarify what I mean: who cares about 100% requirements coverage when 5% of the requirements can cause people to get injured or even die and the other 95% is just reports and other unimportant stuff?

Second, regarding certification. You say: "the tester with a Foundation certification is not a knight; she is a squire. The tester with an Advanced certification is a knight". Really?? I think there is MUCH more to study, train and practise to become a knight! As Bach, Kaner and Pettichord say in their excellent book "Lessons Learned in Software Testing", lesson 272: “If you can get a black belt in only two weeks, avoid fights”. So I hope your knights avoid fights! I rather believe in the 10.000 hour rule (Malcolm Gladwell in his book Outliers) to become an expert.

What do you call appropriate use of the certification by recruiters and managers? What is the added value for these certificates? I have been a trainer for ISTQB and I have the foundation and advance test manager certificated myself. The courses are full of theory and lectures. Not once did we test actual software! Sure, basic terminoligy and theory has /some/ value. But I think we should value practise and experience over theory and lecture. Don't you?

At the Foundation level, the ISTQB is setting down basic terms, techniques, and concepts that everyone involved in testing should know and understand. In other words: you are trying to set a common language, yes? Maybe you should read the blog Common Languages Ain’t So Common
by Michael Bolton about this. I hope you understand that I do not agree with the goal for a common language. It will only create more assumptions and less communication. Context matters so let's talk about that in stead of assuming we know what others are trying to tell us.

ISTQB claims the overall intent is to advance the profession of software testing. I would love to believe that, but what I see doesn't advance the profession. Millions are being spend on exams that do not have much value. And fake-testers are being trained for managers who believe the owner of advanced ISTQB certificate is a knight.

Sami Söderblom said...

Hi all,

I wonder if developing ISTQB towards context-driven world is the right way to do it? It may require too much, an overhaul that changes everything. Why ISTQB needs to be the standard of testing under which everything falls? There is also TMap, which is so dear to us at Sogeti. They are basically the same thing, but with different basic language. What?! How can basics be different?! Oh no, I've gone cross-eyed.

Note: People who invented ISTQB did not invent testing!

Do we really need an umbrella? What if we just accept ISTQB, TMap, SAMI (just created that ;), etc. as heuristics among many? While at it, we could improve our thinking, which helps us to be better, diverse testers. Advanced thinking is basically all the umbrellas, medallions, belt grades and steak dinners we need.

I practise an old Japanese samurai fighting craft called Hontai Yoshin Ryu. We practise certain techniques that have been proven useful (someone has killed someone using them ;), we build our strength, coordination, stamina, etc. but what hopefully makes us samurai some day, and what has helped samurai back in the day to survive the fights, is the special way of thinking. Japanese call it "fudoshin" ( I hope I'll find my "fudoshin" in testing some day... :)

When it comes to being expert, knight or whatever... Our grand master or more appropriately "head of the family" has trained martial arts since 3 years old. As he's now over 60 years old, he still never refers himself as "master", "expert" or such nonsense. Instead he likes to call himself an "oldest student". There are no belt grades at his level. No medallions. No steak dinners. Just an honor of keeping over 400 year old tradition alive and being the one who's the closest to the original idea of being a samurai.

That is quite beautiful.

Rex Black said...

Hi all--

I'd like to address Huib's points, because they bring up excellent issues.

Regarding coverage, you have put your finger on one of the main issues with requirements based testing as a strategy, which is that the requirements are not usually complete. This is why we (my consulting company) recommend to our clients the use of multiple dimensions of coverage, including risk (based on a quality risk analysis), types of data, system configurations, and other critical attributes of the system. The issue of coverage is more complex than just requirements. I was using that to provide a concise example, which I think still stands.

Now, as for knighthood and expertise, it's funny that you bring up Gladwell, because I quote that 10,000 hour rule all the time. That's why we recommend that people taking the Advanced courses have at least 5 years (which would be 10,000 hours) of testing experience. What we find is that people without that level of experience find it difficult to understand--much less apply--the techniques we teach in our courses.

I'm sorry to hear that you've been involved in Advanced training courses that are all theory and lecture. I can understand why teaching such a course would not be a fulfilling experience for you or the students. I can't speak for our competitors' courses, but I am confident that if you were teaching using our courses, you'd have a different opinion.