Porridge is bad for you!

"What?! What is he talking about?" This blog post is about challenging one's criticism and trying to make it more effective using simple methods. This post is about moving away from unhealthy criticism into constructive critique and feedback. This is about finding the right mind-set.

Why do people criticize?

"You're doing it all wrong!" "Your tie is hideous." You've seen it. You've heard it. Probably you've even done it. I know I have. And in most occasions I have let my self-criticism be clouded by someone else’s opinion about the same subject. By not having a concrete opinion of one's own it's easy to adapt to criticism of someone else.

Let's take an example of a guru that has strict criticism against one subject - a porridge (seems like a neutral enough). We all know that porridge is good for us, right? What if a nutrition guru says that porridge is bad? What do we do? If we have huge respect on that guru's thoughts and doings, might we blind our own judgment with the upward gaze towards the guru? We might just take the guru's opinion granted and start proclaiming that porridge is bad for you.

One way out of this bad equation is to step back and challenge one’s own critique. Is it justified? Are you making an assumption? (Great blog post about defeating assumptions by Ilari Henrik Aegerter) What if the guru was wrong? After making sense of what do YOU think, then you should back the critique up with facts, not opinions. By finding the facts, you might be looking for biased facts, but at least you have something to back it up. It's obvious (to some) that a biased opinion should stay as an opinion, but they rarely do. Instead of being opinions they become statements supported by chosen facts.

Doing basic critique, source checking, challenging, context projecting, you can find the root cause of the guru's opinion about the porridge. Does he (our guru is a he today) have an agenda of his own? Are there hidden meanings in the critique itself? Does it provoke thinking instead of criticizing the product?

How do people criticize?

"When giving feedback, do it like so: Always give good feedback in public and be precise about what was done well. Always give negative feedback in private and be precise. Try to find the solution instead of the one to blame." This was said by my father who has decades of experience in management and leading people. I have always thought this as the fundamental guideline of critique. I think most people know this and agree with this, but how come most people don't act accordingly?

Let's say that the guru had discovered some facts that "Ye olde bran porridge" has all sort of chemicals in it that disable some growth hormones on a child. Obviously that's a statement to be told to the public, right? And as we hold the guru in high regard, he is mandated to present his opinion (possibly supported by facts) in some public media. There are channels in which you can present a complaint about food (health inspector or some kind of an agency) and they will take the necessary precautions to tell the public that "porridge is bad for you". Possibly they have first discussed with the porridge company, who might have taken the product off the market.

The guru might give criticism about porridge in public and have the wrath of the porridge company on his shoulders. He might not care as he's a nutrition guru and has an agenda of his own (hoes he?). Is the guru doing the right thing expressing his opinion so loudly in public? Is the guru promoting himself instead of giving critique?  Was the bad thing in the porridge, in the chemical, or in the company making the porridge?

Where's the difference in the approach between the two models of critique? Was the guru able to achieve the goal of his criticism through a public channel (which ever the goal might have been)? Was the "behind closed doors" critique more efficient than the "in your face" critique? They all depend on the context, obviously. What was the goal of the critique?

Feed-forward

Some people think critique is feedback. Well it kinda is in some extent. Feedback however can be constructive even when the feedback is negative. Feedback is given when someone asks for it; critique is given when ever. Feedback is not trying to make one feel happy/sad but to make them improve; critique is about making a statement. When giving feedback don't sugarcoat it, instead say what YOU like and you'd like to see improved. "I liked the taste of porridge and how my stomach feels afterwards. To make it even more healthy I would not put in the chemicals that prohibit my growth."

As the feedback is a kind of a thing to be asked for, critique is the kind of a thing you just blurb out. Feedback has a purpose and it is meant to improve the one receiving the feedback. Critique has the tendency to provoke something. Conversation, debate, hatred, etc. Challenging can be more effective a way than critique. Challenging the critique itself can become the most valuable feedback there is!

Is the content self-justifying or do we need to empathize to support the critique?

There are tons of guides in how to give feedback without being critical. I know a dozen occasions where I have let my judgment be clouded by numerous things that have lead into bad critique and undesired results. Here's one:

I try to promote intelligent testing and intelligent approach to quality in general. I also believe that certifications that focus on the certificate itself are no good. A certificate that focuses on skills in a field that requires skills is a good thing; artificial certificate concentrating on a narrow view about best-practices (and only knowledge thereof) is a bad thing. This is what I thought and still think. I was having a discussion with people I think highly of about “what is your opinion about ISTQB-provided series of certificates”. By making a comment that the certificate looks good on paper (with some unflattering spices), I provoked a series of questions about “how do I back up my statement” and "do you even know what you're speaking of".

The questions stuck home and I started to think of how I really thought about the issue. The fundamental thought behind the issue remains the same, but as I have not delved deep enough into the syllabus, the history thereof, the initial goal behind the syllabus. Am I eligible to make claims about the issue? Was I repeating what the other people were saying and them making myself feel important about myself by making a rash claim? Does my opinion really matter in this case and could I do some good without being so loud about it? Is there a possibility to raise conversation about the issue within the certificate organization without sounding like a zealot?

With the comment I made (which was criticism at its worst) I thought the content of the comment was self-justifying. "Obviously all the people were thinking the same so I just said it out loud." Even though some of them were, they rightfully challenged my comment and forced me to think about it. What was my goal when stating something like that? What was the desired outcome? Praises to me? More Twitter followers? To raise conversation? To sound like a dumb-ass?

What I did achieve with the comment was for me to be able to criticize my own behavior and claims. I once claimed (in Finnish) that one way to achieve the best quality of an end-product is to "Murder you darlings" - by finding the most direct route from the current point into the desired point. By removing all the excess and self-promotion from the content. To go directly towards results. In making the comment I a was focusing on "sounding cool" instead of trying to use the words as a tool to achieve a goal (which apparently was shrouded).

Did I hurt someone in the process? Can't tell. Not directly, I assume (Pekka, you're assuming things).

Did I achieve the goal? Can't tell. I wasn't aware that there was a goal.

Did I learn something? Oh boy, did I!? ;)

I don't know

What's this post about?

I don't know. That's right; it's about "not knowing".

I spend the two days at a coaching course by TNM Coaching and we had a great time there. There were really good conversations about topics regarding coaching and I will delve more deeply into the ones that had most impact on me. Before I go deeper I will share some insights that I learned at the course about questions and the answer "I don't know".

The fact of not knowing

We all get asked questions that we may not have the answer. Some questions are just too complex for us to understand or we may not have the skills to answer that question. For example someone asks you about the amount of stars on the Northern sky, so you'll probably answer "I don't know". In this case you may not have acquired the necessary knowledge base on the topic and thus lack the ability to know the answer.

Could you answer differently?

When we answer by saying "I don't know" we subconsciously diminish ourselves. We give ourselves the impression that the knowledge is required to be "something" and by lacking the knowledge we are lacking as humans. It's a human behavior thingy of some kind, I think, and it comes naturally if we don't have the answer thought up.

If we could spend some time to think about the answer we might be able to avoid the "not knowing -trap". A difficult question requires a bit of analyzing. Do I need that knowledge right now? Could I check it up somewhere? Is there a reason why I have now acquired the required knowledge to answer this question? Do I possess the knowledge already but effectively forgotten it? By answering a question with a question (be it mental or verbal) you may find a better answer than "I don't know".

I don't know if I want to answer this question

There are different situations where you answer a question with an "I don't know". Sometimes the answer is to avoid answering truthfully. In a coaching session when another person asks questions and helps you solve your problem the answer "I don't know" may come in up. This poses interesting issue with the coach as there may be answers behind the "I don't know".

At this moment comes a point where the situation needs to be evaluated. Is it reasonable to try to find the answer behind the dodge or is it better to let it be. If you decide to let the issue untouched it should be stated. You may agree to speak of the topic at later times.

If you however mutually agree to go deeper behind the dodge, one good way is to eliminate the psychic lock (a Jedi Mind trick) by a simple question: "If you knew the answer, what would it be?" or "If a situation presents itself, what would you or someone else do to solve the situation?" The wording reaches behind the barrier and encourages the person to use the capabilities he or she has and start seeing the solutions. This may not however work always, but it is one part of the probing and facilitating the coachee to be able to verbalize his or hers issues and find the solutions.

More coaching stuff coming up!

This is my insight onto the subject of "not knowing". I got the inspiration to this from the coaching coach Vivienne Ladommatou who spent two days with us at F-secure to help us be better coaches. I really admire her wisdom about Genuine Interest and I think that could be the next post regarding the coaching.

I will be making at least some kind of a blog post series about coaching, especially the things that matter to me the most in coaching. I will do some practicing to hone my skills so if someone is interested to help me with my quest to become a better coach, feel free to contact me and we'll figure out some time slot and issue that we start discussing about using the coaching process.

So just tweet or Skype me, or comment under here, and I'll arrange a coaching session that best suits the context. When will be a good time to coach? Well... I don't know. ;)

Testing with the stars

On 15th and 16th May there was the coolest conference I have ever been to: Turku Agile Days 2012!!! There were great speakers from around the world - Lisa Crispin, Elisabeth Hendrickson, Linda Rising and many many others. But in addition all the talking and workshopping, there was this competition called Testing with the Stars. This was the first of its kind and I had the privilige to be invited to be the star tester in the competition: I also had the most amazing pair from F-secure St. Petersburgh office, Sergey Moshnikov (@dan_dandelion in twitter). He was a great sport to tolerate all my rantings about exploratory testing and he did an amazing job with all the sessions, especially on the subject of BDD in which he appeared to be a huge talent! It was an honor to be paired up with a talent like that.

You can read more from here but here's a quick summary of how it went.

The competition had originally 6 pairs of testers consisting of a star tester and the guest tester. The teams were

• Ru Cindrea & Ulrika Malmgren
• Anssi Lehtelä & Tom Leppälahti
• Petteri Lyytinen & Jenna-Riia Karhunen
• Pekka Marjamäki & Sergey Moshnikov
• Maaret Pyhäjärvi & Henrik Juvonen
• Juha Rantanen & Anssi Kinnunen

The first day was spent doing 3 ten minutes sessions each about Testing strategy, BDD and Exploratory testing. After the ten minutes session our "dance" (as the concept was borrowed from the Dancing with the Stars TV-program) was judged by Lisa Crispin and Elisabeth Hendrickson (who were the guest judges of the competition) and finally also by the audience. So basically there were 18 dances from all the six teams and three got to the semi-finals on the second day. Me and Sergey were lucky enough to get into the semi-finals along with Juha Rantanen, Anssi Kinnunen, Anssi Lehtelä and Tom Leppälahti. The first day was great fun and the contestant did an amazing job with the little time they were given to prepare to the show.

The second day began with a dance free of choise. Me and Sergey did a BDD-dance on how to introduce value to the BDD process. We had help from Jenna-Riia (thanks for all the help we got!) and we managed to pull together an act with the score of 21. Juha Rantanen and Anssi Kinnunen dazzled us all with a BDD dance where thay demonstrated the story building process and included an example of how it could have been done using Robot framework. They got a score of 26. Anssi Lehtelä and Tom Leppälahti did an amazing job visualizing test strategy building using Anssi himself as the test object! It was hugely entertaining but got a score of 16. After the audience had voted the teams' performance Anssi's team and ours were tied with 29 points each (Juha and Anssi had 34 so they were going to the finals anyhow). As the audience was the ... urm... audience of the dances, it was their word that settled the tie and Anssi and Tom went into finals.

Anssi was however not pleased with this as he kinda liked our show, se he insisted that we would team up and do a double dance! So we did a testing competition within a testing competition and we let the audience decide which team won the exploratory testing duel. It sounded like the audience was leaning against Anssi's team but we had a blast doing the duel. Then Juha and Anssi came with their exploratory testing dance and blew up the bank! Dancing robot testing to the beat of Scooter's "Move your a**". With which they won the competition.

I really enjoyed myself during the conference even though I didn't get to see too many talks. I however got to hear the great Linda Rising do a keynote on Agile Mindset and a talk from Elisabeth about test automation. I am definitely going to attend the next TwtS that is going to be arranged as it was so entertaning and good learning to all the participants also. If any of you want to see the video recordings from the sessions they can be found from Bambuser but be warned. ;)

As the Linda Rising's talk was so inspiring, I'll finish this time with a quote she mentioned in her keynote:

"Ever tried. Ever failed. No matter. Try again. Fail again. Fail better." -Samuel Beckett

In the face of failure - part 2

Sometimes we fail to achieve what we have set our minds to achieve. That is not always a bad thing as we might learn something about the process of failing. The failure itself may require us to change our perspective or approach to the task at hand. In the first part I covered what happens when a test fails. In the second part I look into people's failure in effort to do something. All along I try to offer insight and learning possibilities to the subjects.

When a person fails (or doesn’t fail)…

Tests can fail. They reveal information when they do so. Even a passed test may reveal new information. Same thing basically apply to human failures. Why did my effort fail? What are the critical points that lead into the failure? Is there a possibility that the failure was evident and there was nothing that could be done to prevent it? If not, how can I see the critical points in the future?

Are we the cause of failure?

There are hundreds of self flagellation books that motivate you not to fail. If we are to attain a critical mind and keep it, we need to realize that we are fallible. We make mistakes, ok? There's nothing you can do to rid yourself of all failures. By keeping in mind that you are allowed to fail, you may prioritize with more precision, plan important thing more carefully, and learn from events that may lead into the failure.

The worst enemy of a person (who fails or succeeds) is ignorance. The ignorance itself may be the cause of failure, but by ignoring the root cause for a failure prevents all learning. Ignorance leading to failure is the worst cause, but still a person can learn from ignoring as much as from anything else. A person may have a reason to ignore something due to scheduling, required effort/skill/resources, insufficient knowledge, or a memory lapse. This is a kind of benign ignorance, as the ignorance is used as a tool towards the goal. If a person chooses to neglect, belittle, condescend either other people directly or indirectly, it is malign ignorance. A person failing to deliver on time because of procrastination or lack of interest in the subject falls to the same malign category.

People can learn from ignorance, more easily from benign ignorance than from malign. You can always choose to take note on the thing you previously ignored and decrease the risk to fail. Even attitude of ignoring can be changed, but it could require huge effort to do so. It is encouraged that people rid themselves of ignorance to important matters. They may prioritize some things lower than others and thus they do not ignore them completely. Asking "am I ignoring important things?" could result in fewer failures. If a failure occurs, what was ignored and why?

We may fail also because of choosing the wrong tools, approach or methods. For example making a presentation about some scientific thingy and structuring it poorly, may result in failure. Choosing wrong viewpoints (or too few) can result in biased view and thus result in failure. We may even choose the right tool and fail because of lack of skills to use it. If I don't have the sufficient skills to do this, who has? Can I utilize that person’s skills in this task? Do I have to adjust my views to reach a better result or to suit the given task?

We may make a efficient contribution towards succeeding in the task but it may still fail. We may have overlooked some important information that could drastically change the outcome of our task, or we could have made the wrong interpretation. The material or the baseline itself on which we built our effort upon may be faulty. Have we enough information to successfully complete the task? How do we determine that? Is the opinion I have conclusive enough to lead into success?

"It's their fault!" ;)

All things leading into failure are not reliant upon the person doing the task. The task mey require effort from multiple people or organizations and all cannot be controlled. To succeed in a multi-person, multi-venue task we need to establish a knowledge which person is holding what information / is responsible of which deliverance / responsible form which sub-task. If keys to our success lie in another person's hands, we need to make sure those keys open the doors for us.

It could be that the venue where you were to hold the presentation is accidentally overbooked and your effort fails. They may have suffered from a Force Majeure or something completely irrelevant to the task at hand but preventing from succeeding in it. There are some cases where one can find learning opportunities regarding the Force Majeure incidents. Can I prevent the task from failing due to unexpected events? Does a failure due to those events cause failure to retry? Can I take precautionary actions to prevent those things affecting my task?

If the environment and the nature (a stampeding elephant horde is considered a force of nature. "Jumanji!") don't cause a failure you still have take into account the audience of your task. This may include the people taking part to a presentation, the stakeholders of a software project, etc. The person doing the task has little power to effect the audience. The stakeholders may have beliefs, reasons, attitudes, etc. that may hinder the successful execution of a task. Audience may choose to ignore your topic and perform a no-show (or have more important thing to do at the time). What can I do to ensure that the task fulfills the goals regarding the audience/stakeholders? Do I need to adjust my approach so that the stakeholders don't have to struggle to relate to me or my topic? Do I need to do research on the receiving end of my task?

By asking important questions about the task and the surroundings can drastically decrease the possibility of a failure. And even if you fail, you will have a solid base for learning from your observations and the information received from the analysis of a failure. Accepting that you are fallible increases the ability to think critically. By anchoring ourselves to the opinion that we can't fail, makes us vulnerable to being biased and blind to important details and information.

Don’t think failing as a bad thing...

As we know, we all fail at something. There's nothing wrong with that. Succeeding might result in fewer learning opportunities than failing in the same task. Still, we can use the same techniques we use to analyze failures to apply to analyzing success. Did I prioritize efficiently my time and tasks? Was there a pit that I fell into thus decreasing my chances of succeeding? How can I remove that deficiency in my future doings?

Attitude towards failure is one of the biggest obstacles towards success, as we think it makes us losers or bad people (at least some people do that). Every attempt is a learning opportunity, pass or fail. Inquisitive mind is the best tool to enable learning in failure as in success.

If you found this piece of blog post inspirational, please comment that it was inspirational. If not, then comment that it wasn't. What ever the response, tell me why. If we (failing or succeeding in our doings) do not get feedback on from our effort, we cannot tell whether it is a success or not. By saying so, I don't encourage people to praise without justification or mock unnecessarily. The task for commenting is to succeed in commenting. Would you rather fail or succeed in that?

In the face of failure - part 1

Sometimes we fail to achieve what we have set our minds to achieve. That is not always a bad thing as we might learn something about the process of failing. The failure itself may require us to change our perspective or approach to the task at hand. in the first part I cover what happens when a test fails. In the second part I'll look into people's failure in effort to do something. All along I try to offer insight and learning possibilities to the subjects.

When a test fails (or doesn’t fail)…

As testers we do tests (DUH!). This may be a mission, a scenario, a flow, a check, basically of any proportion of work effort done in order to achieve some testing goal. This may also be a check conducted by a machine of some kind (test automation script, you name it). The point is that there is a test and we have or might not have some assertions (assumptions, expectations) regarding the outcome of the test. There may be tests that we have no idea what the outcome is (“What happens if I press this blank button at GUI?”) but it may result in a future assertion regarding the same test object.

Test may fail or they may pass. That is the binary nature of a test. They may however trigger a whole another result, for example “indefinable”, “false negative”, “false positive”, "what the bloody hell is that?". What happens if a test results in a “test passed”? What does it mean? Can it still fail? Can it mean something more?

Test fails because the test object doesn’t pass the assertion

This is what we want to happen, when a test fails: The test fails because there is something broken in the test object. By wanting this to be the case we might close our eyes from something important. The case simply states that the test object does not pass the assertions for reason unknown.

We think that the problem lies in the test object but are we sure? We need to eliminate the "think" and go towards the "know". We must analyse the test itself to determine whether the result is infact correct. The result may be a false-negative as where we need to determine if the test itself is not faulty. We may be testing the wrong thing or asserting the wrong things.

After we have determined that the test object infact is not built to match the test, we must ask: "have we built our test incorrectly?" This may result in information about the behavior of the test object just as well as the behavior of the TEST! The behaviour may be incorrect in both situations, but the newly found behaviour may be the thing customer/user/stakeholder (that is important enought) wanted or truly needed. The current behavior may also be a better solution than the intended/planned. In any case, this information must be revealed with more testing and analysing.

The test result may also uncover a risk that was not taken into account or was ignored before. This rises important questions about the the test object and the processes of development and testing. If there is a hole in our risk mitigation strategy could there be a need to revise the process or processes? Could there be more areas that we have not yet covered?

Lastly we may require to re-test some parts of the test object as the unwanted behavior mey be required to be fixed (or the test if chosen so). In any case the test requires revising, possibly some fixing and definitely more analysing. Do we need more tests to cover the area where the behaviour was found? Do we need to revise MORE tests? Was the test extensive enough to be feasible after the fix?

False negatives and positives

In a false positive/positive case we have already done some analysing to reach the conclusion that the test gave faulty information. Both cases may reveal something important about the test object but more than that they tell that there is something wrong with the testing. Do we have enough information about the test object to be making statements by the test results? Do we need to do more research on the test object to make our tests better? Have we missed something important in the process? These results always require analysis on why the tests give false results in the first place.

False positive is basically a situtation, where we think that the test object is behaving the way we think it is. This result is corrupted by the defects in the test itself and thus the test gives a "pass" result. it could be that the test is concentrating on the wrong area or function, thus giving false information. Therein could lie a risk that we have not covered a critical part of the product with sufficient testing. The assertion in the test itself could be missing or not strict enough ("Check if there is a response of any kind." -> Soap error message.) To get this situation corrected, usually both the test and the teset object need to be analysed and possibly fixed.

False negative is a situation where the test gives a falsely prompted negative result to a test. Almost all the situations apply to this as in the false positive case. We may need to back up the result with additional tests to rule out the possibility that the test object is behaving incorrectly. We may be lacking in skill, we may have ignored something or simply there is a defect in the assertions.

Tests always reveal something, and it may be important

By doing testing we uncover information. The goal of testing is to give enough knowledge for the deciding-people to make the right desicions. Even a poor test can reveal important information. If critical information is revealed at this stage (early or late), have we done something wrong in the previous stages? Is the poorly constructed test a waste? Do we need to construct better tests or are we aiming for fast results at this stage?

It could also be that because of some flaw in some process we have stumbled upon the wrong area to test. It could be that there are communication problems, ignorance or some other reason that we do testing to an area we are not supported to do that. It could be that the feature is under developments still, the environment is not ready, etc. Does the testing we did vale any value or was it waste? Do we value learning? Did we learn anything about the test object?

Even if we have the most beautifully constructed tests and the test object is in good shape, we may encounter problems if we do not know how to interpret the results of the tests. Results contain sometimes false results (positive/negative) that must be uncovered and examined so that we have correct information about the test object at all times. The one interpreting the results (tester during testing, test automation specialist, etc.) should have enough competence and criticism to question the results. We may ignore all "pass" results and just focus on the "fail" leading to false information.

We may think we know enough about defect. We did find it, didn't we? By analyzing the failure itself and the root cause we can uncover more information about the test object and its surroundings. By questioning the test results just as we question the test object, we may reveal some information about the testing methods, processes, tools, etc. and be able improve our testing.

Read also the part 2.

Visualizing the testing effort

There has been some downtime with updating the blog, but that's about to change. The previous three months have been quite intense and I have learned quite a bit about software testing in a testing team. As some people already know, I work as a legacy system tester. The two months that have passed I have been learning the product and domain in a client team (Yay! Go team Raid!) with a bunch of expert testers. The team has three testers (four when I was in there), two test automation specialists and a Scrum master / manual tester. I was there to reinforce the manual testing and to learn as much as I can during the way.

While I was in there my goal was to hone my skills to perfection in exploratory testing and especially in scenario and flow testing. The features I was working with were part of a complex basic functionality that had been growing since the first product was out. This addition to that functionality pool was a much needed feature and affected basically every part of the functionality.

As the testing was being done we had to forgo the automation as we needed fast and tangible results, as in severe bugs as fast as possible and large coverage of the ground functionality. This resulted however in a pile of bugs and lots of rework. At this point we had no stable ground as all bugs were being fixed one by one and there were built coming twice, three times a day. Result: instability.

The decision to do all testing manually resulted in a HUGE amount of re-testing. We could have made a decision to write a basic set of tests to be run automatically and thus decrease the amount of checking done manually. On hindsight this would have been a great thing but it would have decreased the coverage achieved by manual testing.

To find the golden road with manual testing and automated testing is probably the hardest possible thing to achieve. Managers tend to veer towards automated testing as the tests are re-runnable and documented. Developers have the same approach as the script more easily presents the coverage of testing done. Is it possible to create the efficiency of a manual testing with the coverage visualization of automated testing?

Visualization

When we do automated testing its coverage is measured in different ways to sate the needs of visualization. We have the percentage of code covered by low level testing and we have some number of tests run through different scenarios through GUI. These high level tests are usually run through APIs or some engines that replicate the use of GUI. Both ways we measure have different ways to interpret the results and some pits they fall into when visualizing the results.

Code coverage tells you how much of the code is covered (DUH!) but more importantly it tells you how much is NOT covered. We can make assumptions about the coverage, but as the coverage doesn't cover the user using the product but the code using the code, this can be deceptive. It is however a way to build confidence to the product and to lessen the risk of regression (and we all hate regression, right?).

From tickbytick.co.uk

It is possible to take this to the next level by measuring branch and module coverage. This eases the visualization as you can point out the components within the flow that have not been tested. By taking the "untested" parts into a context we can see whether the coverage in that component is critical or trivial. Instead of relying on numbers (what does 49,21% percent of branch coverage tell you?) we can visualize the coverage and make decisions about the enoughness of coverage more easily and more accurately.

After the low level coverage we move to high level testing, system testing, acceptance testing, you name it (literally). When automating testing on a user experience level we need to make a choice whether to make tests easy to create or them to be realistic. Easy tests mean using APIs and direct calls to back end to replicate a situation in a GUI. This is great way to give false confidence in GUI. As the test engine cannot see the client, the GUI may be unusable but still pass all tests. The buttons, bars, boxes, whatever may be unreachable, unusable or plain ugly, and the machine doesn't see that as it "simulates" the use. This kinda testing is comparable to the code coverage. We have the knowledge of what has been tested but there is no knowing if the critical parts have been tested (or that they work in actual use case).

When doing high level testing and taking it closer to user, we lose some of the speed that we rely on when automating testing. By clicking artificially the elements that the test machine (program or script) sees we can be fairly sure that the user sees them too. This approach however comes with a load of updating the scripts and screenshots, but when done modularity this approach can solve lots of problems in client side test automation. When visualizing the latter GUI automation case we can draw lows of what forms and dialogs have been browsed and used. This tells the story of different scenarios that have been tested.

So we have possibilities to create a greater visualization of the coverage we have. By displaying these results in a radiator we can spread the information and more importantly raise questions about the results. "Why is that area uncovered by tests?" "Why is that part covered so vastly as it's only a supporting function?" From that point on we can make the test automation work for you instead the other way round.

So how can we visualize manual testing by using what we already have in automated testing? Visualizing code coverage is quite futile as it requires some sophisticated debugging/sniffing tools that should be run at the same time. And it may not give any value to the testing and/or visualization of manual testing. There may be some benefits with special cases like prototype testing when the code is not yet built into dll. This may have security problems and require a strong business case to be considered.

More reasonable way to approach is visualizing the scenario coverage of manual testing. If we visualize the testing we do manually it should be as easy as possible. If a larger team does testing and should record the coverage, the way should be standardized or at least agreed. Visualization through mind maps is a great and easy way to do this either with individual testers or with testing teams. As I wrote before on mind maps, taking the visualization a step further by adding the whole teams effort on a single mind map can visualize the total coverage. Using radiators as a way to spread information the coverage can be real time and dynamic.

If the aspect of using manual labor to create coverage charts is daunting, there is a possibility to record the testing done with a tool however and make it visualize the coverage. By recording the testing done manually, say with a screen recorder (Adobe Captivate, Bluberry FlashBack recorder) we record what we have tested and run it though a picture recognition program. It then lists the steps in a scenario (with possible side steps) and draws a graph of testing coverage. There is not yet known tools to do that but a simple Python application might do the job (I may have to return this eventually, but this should suffice at the moment).

As most of the information about testing lies in the brains of a tester, that information needs to be pulled out and visualized. Using radiators and simplifications we can provide the information the managers need to make decisions about the product. If we can make them veer away from the number-driven into the information-driven decision making, the products' quality will (or might) be increased.

Any way, I have gotten some good ideas from the client team to improve my work as a tester and hopefully I can start forcing these ideas to other so that visualization comes naturally with all aspects of testing. By being able to visualize the quality in both manual and automated testing they both become important to decision making and less ambiguous.

Managing Heuristic Exploratory Testing Based on MindMap

The FreeMind mind map template that I use.

The MindMap allows me to create a structured testing quickly and efficiently so that the testing is accountable and efficient. The mind map functions as requirements document, test plan, test report, bug report and heuristics cheat sheet for my testing. The mind map functions simultaneously as the tool and the product of testing thus supporting the values of agile and lean development. I’ll explain how.

Gathering Information

As the picture above states that there are multiple heuristics to choose from and when starting to test the best way to start is to gather information. The “Project” heuristic is a good way to start developing a sense of what to test, how and in what order. Using the applicable heuristics one can form the basic knowledge about the test object and the surrounding project.

This phase may be as quick or as time consuming as necessary. The key is that you’ll establish the needed knowledge about the test object. This may include (but not restricting to) referring to requirements documentation or other documents, talking to product/project/solution owners/architects/managers, requesting consulting from developer/other tester or the like; anything you see helpful to your task at hand.

At this stage you’ll start also forming you testing plan. This may include equipments and resource scouting or gathering, scheduling and possibly limiting the scope of testing. By limiting the scope into testing that can be done in 90-120 minutes sessions the testing is more manageable and less tedious. Forming the testing into specific missions also makes your testing more motivating as you’ll achieve lesser goals on your way to larger goal.

At this point you have a mind map that looks something like this:

Project heuristics during Information gathering

You should have sought answers to the questions or decided they do not need answering. This forms the basics of your test planning. At this point you’ll also have a good grasp of what is important and should be tested first. It is possible that the priorities change during testing (and it should) as does the scope. This is however the framework in which the testing starts.

The gathered information is context-dependent and varies across domains. The mind map may also be tweaked to suit the domains special needs. Also the items/tokens may be chosen to best suit the purpose either increasing or decreasing the amount. What I use are Question (“Someone needs to tell me something regarding this topic”), Exclamation (“There is something in this topic that needs to be looked into in detail”), Idea (“Possible solution, test idea or a thought”), Blocker (“This topic might cause a block to testing now or at some point”), OK (“This topic is tested and considered not needing further testing”) and NOK (“This area requires more testing at some point”). The most important thing is that the choice of items supports the cause instead becoming the cause itself.

How to measure coverage?

As we have now established the basic knowledge behind the testing task we may start to sketch the coverage and depth of the testing. This is done by modeling the test object. Modeling is a simple way to map or… hmmmm… model the test object. This is done by creating a representation of the test area using the modeling heuristics. The models may be as detailed as needed or as vague as they can be, again what bests suits the context.

I myself start the modeling by mapping the structure of the test object. In case of GUI testing I try to sketch the basic layout of the GUI forms and deconstruct the components within the GUI. For example some text fields can be tested with same inputs (possibly). This also gives a good representation of the depth of the GUI (as if there are any forms /behind/ the previous). Menus are a good guide in deconstructing the GUI as they also may hint onto what functions the system has to offer.

The structure may consist of anything “physical” or fairly easily recognizable objects of the test object. The operations of a web service are easily modeled as are the fields within the operation. There are possibly even tools (spiders) that will form the structure model for you by minimal work. This is the first coverage model that I use and frankly the most simple. The coverage may even be derived into X/Y –coverage to sate the managers’ need for figures, but this is also to remind you of what has been done and what needs to be done.

Functional coverage is the list or the model of what functions the test object has to offer. This may also be somewhat easily derived from visual/physical things but also the hidden/hard-to-come-by functions should be stated. The function may however be used in a different way that it was intended so user/admin/automatic operations should be stated also either as their own or where they differ from the intended functionality.

The data coverage is important to some and should be deconstructed into a heuristic list. Easiest way is to find out what kind of data the structure requires and what is in the interface requirements or in database. Also analysis of what kind of data is it possible to insert into system and HOW! There may be data security issues in public interfaces or GUIs that need to be tested with different approach or depth.

Platform may affect all tests and if there are any platform dependencies they should be stated. Sometimes it is not seasonable to test everything in every platform so the criticality of the testing should be estimated on each platform. Sometimes test automation and changing the test approach may come handy if testing multiple platforms (i.e. mobile phone testing on different phone models).

Time-related testing is also a way to measure coverage of the test object. If there are time-related testing to do (performance testing, stress testing, concurrency testing, “24/7 testing”, etc.) these are also a way to measure coverage. Performance coverage may be important in a SOA world where critical components require consistent and good performance.

These form into a map of models of which we can inspect coverage and progress of testing.

The coverage break down using Models heurisrics

Heuristics support testing

The use of heuristics enables you to think of multiple ways to test something. One can apply many different techniques onto ones testing to dig up different kinds of information about the product. They can be test techniques (remember that exploratory testing is an approach not a technique), quick tests, tours, you name it. The point is that the heuristics mind map functions as a checklist (if that is the way you want to use it, for example coverage reasons) or as a heuristic in itself.

Whatever the approach to testing is the heuristics are there to help you in your thought process. Using the mind map to record your thoughts about some aspect of testing is advisable and easy, as you can connect techniques to models and parts thereof. For example using quick tests (“clicking frenzy”, “banging the keyboard”, etc.) to stroll through a GUI or flow testing to do some E2E testing, the mind map is a good tool to design testing on the go and to report any findings into the map.

The mind map can be used to report bugs using the video clips generated from test sessions in addition to possible description in the mind map. These help communication to developers, managers and clients, and writing the defect reports. The issues and questions that testing uncovers can be linked to oracle heuristics to make more case to the defect.

For example a basic stress test may indeed pass within the NFR description but the memory usage or the amount of handles may not even be described in the NFR. Of course it is up to the tester to choose the best way of describing his/hers findings. The rule of thumb is that there should be enough information to help the developers to fix the problem AND to make an appealing case to fix it! (No bug report is good enough if the bug doesn’t get fixed!)

Heuristic testing skills (for example Rapid Software Testing skills) are also important here as one can rely too much on the check list and forget to do the testing in their heads (where most testing is done). Heuristics should be applied (if applicable) not followed. There’s no harm trying to apply some heuristic in a context, but using the heuristic as a hammer and trying to smack everything is not advisable (it might even be stupid). They are there to help you reach your testing goal and provide tools to your brain.

If stuck, the heuristic mind map can give you a good boost into proper direction: just look at the heuristics (if you can’t remember them) and think of what to do next and how. Heuristics work also as guides in tweaking your plan. You may have gotten false information during your info-gathering or bugs prevent you from doing something. Heuristics can provide a solution to a changed situation when a new mission needs to be formed.

The testing heuristics applied

A product and a tool of testing

The agile principle describes unnecessary documents as bad (or that is how I have interpreted the manifesto). Michael Bolton talked about using “memos” in testing to help the brain to interpret all info pushed into it. Instead of being a document (a product which you produce) the memo is used as a tool to help testing. The mind map has a similar value to testing: it helps make notes about things you see, feel, and observe, but also enables you to use these notes as a report of what you have done (if someone asks).

The mind map serves as a tool to facilitate the testing but also as a product of testing. It serves as a platform for communication when describing what went wrong and how. It serves as a report of the testing to increase accountability. It serves as a session report of which the Test manager or other testers can fathom what you did, how and what important knowledge there is that they need to consider.

A finished mind map looks something like this:

A finished mind map with references to other heuristics and links to screen caps and video clips.

The questions should be answered, if there were any. The bugs, issues, ideas, etc. should be discussed and possibly closed. New testing sessions should be organized to cover areas that were not covered.

I used FreeMind 0.9.0 in my example and in my work daily. The example mind map was done to illustrate the possibilities of using mind maps. Normally my mind maps are a bit more restricted to certain type of testing.

The web is full of mind map software from browsed based (like mindmeister) into downloadable tools (like XMind). Just choose the one best suit you (or ask someone for a recommendation) and create a mind map that suits your style of testing. And get testing!

If you have had good experiences using mind maps, please share them. :)