Testing technique workshop

The last part of the Testing Tuesday’s “Test Pistols Tour” was a workshop about testing techniques. The original plan was to have a list of techniques and then exercise to learn those techniques. The scheduling caused us to change our approach because we had no time to create environments for exercises.

So, I turned to the community.

When I dragged my but to the conference room I was expecting just a handful of people, 3 or 4, but eventually we had 7 people. I think there was a bit of a tour fatigue in the air, since this was the seventh workshop. I had seven brave soldiers at the meeting room.

“I have changed the rules!” I said. “I ain’t gonna tell you about testing techniques. You’re gonna tell me about testing techniques.”

Now the plan was the following: Pair people up, make them test something and describe their testing. Then discuss what kind of a problem they were trying to solve with their chosen approach. Sounds simple enough. I was a bit uncertain if people could describe their testing to a level from which I could derive a technique. The challenge was thrown.

I told them to open Word. The assignment was to test the “Find and replace” functionality and describe to you pair what you did and why. I asked some questions from the teams during the 10 minutes of testing we did and made them focus on actually telling why they chose to do something. After the ten minutes, we started talking about how the testing was done. These are the key points we came up.

Hot-key testing

The first team started to describe what they did by explaining how they searched for the functionality. They were trying to find different ways to access the functionality. They found out that on different operating systems the hot keys vary. More so, the hot keys are customizable thus enabling different combinations. “Ctrl+F” was the easiest way to find the function, because it happens to be the same on many other software also (comparable product and familiarity to user). On Mac there wasn’t a “Ctrl+F” so the hot key was a bit difficult to find.

Based on their approach to using the hot keys we gathered that the technique can be used on many Windows based software (and why not Mac based, but I don’t have the experience to use hot keys quite yet). The commonly known hot keys like “Ctrl+C / V / X / Z“ etc. are quite easy to test. The tests are quick and cheap, very generic thus making the technique quite useful.

Premise variance testing

When the group was trying to find different ways of accessing the functionality (hot keys, context menus, sidebars, ribbons, etc.) I asked if it changes the behavior of the functionality when you access it from different origin points. If you change the premise, can the functionality change?

We started to think if we could apply it to various other solutions and products, and we came up with “premise variance testing”. When one changes the premise condition to a function, there might be changes in the behavior. This kind of technique can be derived also into a “step variance testing” where you mutate a single or many elements within the process.

Help testing

When one team was trying to figure out how the function worked, they pulled up the manual. The help can be quite simple for experienced user but it acts as an oracle on many occasions. During help testing one can testing the help itself against the product and test the product against the help. In either case, one acts as the test object and the other as the oracle.

This technique could be derived into all kinds of oracle material testing. We can test against oracles that are used by various stakeholders, e.g. requirements or design documentation. We test the product and ask “is this ok?” and we then try to solve the problem by referring to the oracle. We might have an oracle (e.g. human oracle telling how it should work) and then test the other oracle based on the new knowledge (the human oracle disputes the written document). The “Help testing” might become “Oracle testing”, but the name doesn’t give me good vibes. ;) A help could actually be any material that helps us do testing.

Data roundtrip testing

A team was testing replacing a word with gibberish and then replacing that back to the original value (“Pekka” -> “ASDFGH” -> “Pekka”) and they wanted to know if the same amount of entries are changed. So basically the idea was to revert the original data without actually reverting the state. Mathematically I think this is called “inverse function”. First we apply the normal function followed by the inverse function. Roundtrip actually means that you return to where you started from.

We had a discussion if the “roundtrip testing” is actually a generic thing that can be done to a state also. It is possible to revert the system to previous state without any information whatsoever about the state that was visited. This might actually be a problem in itself, but we chose to narrow our testing technique to mere data.

Minimum data

We did find some testing ideas while describing techniques and I think this was worth mentioning. A team wanted to test with as little data as possible. That is a one variable of premise variance testing where we solely focus on varying the data instead of the states. This testing can be the defaults form testing, testing without any inputs (NULL, n/a, whitespaces, etc.), removing metadata, etc. and it can find bugs in the exception handling logic.

Conclusion

All in all, the testing techniques we found have already been described in other sources, but these made sense to us and felt important. The terms are more tangible than “product tours” or some techniques found in books. We defined the terms and we learned how to describe them in a language that suits our context.

I know that at least the premise variance testing stuck. I have used it a few times now to describe what I do. It makes sense to repeat this exercise again with a different depth. Then uncover new, undescribed techniques and make them part of our tool box. After there has been a handful of these sessions, we might have enough skills to describe our testing to any stakeholder in a language we share and understand.

Sadly that was the last of the Testing Tuesday workshops on this tour. There will be another tour in Helsinki, and I shall write up as much as possible from those sessions.

- Peksi

Bug handling workshop

I am running a thing called "Testing Tuesday" at the office. The concept is simple: Sanctify Tuesdays to software testing. This comes in form of helping project teams to test by helping them solve their testing related problems, promoting testing in every possible way. And to top it all, an hour workshop on some testing related topic. I will do a proper write-up on the subject later, but I wanted to share the coolest thing that happened during the 6th (out of 7) Testing Tuesday. The topic was "Bug handling" and the results were really awesome!

A week before this workshop we had a testing oracle related workshop, which I then promoted on twitter. I had classified three bugs and I mentioned those in my tweet. I then had a tweet exchange with Michael Bolton about classification.

That discussion made me want to redefine my Bug handling workshop, since I saw that people I work with, me included, might have quite a different approach to handling the observation we make and receive about the product we work with. So after talking to Michael on Skype I decided to do the following:

Have people define a bug handling process from the very beginning to the very end. Then plot it out, draw diagrams, etc. to explain it. Then focus on the difficult parts and try to enhance the process.

So we started by defining where does "bug handling" start. I started by saying that it starts from the moment there is code, but I was corrected. Bug handling, or observation handling, starts with the first indication or deliverable of work. That might be the requirements documentation, project plan, or whatever tool that is used to run the project. It can be unwritten requirements. It can even be an idea! From the very beginning we start testing and observing the subject. It is those observations that might require handling.

Based on the purpose and the need, we define the way we report, write down, take notes, etc. If we are talking about testing ideas, the observations could be about the idea or the repercussions thereof as statements voiced out. When testing a software, observation may be something you see, hear or feel, that you write down or record. A bug report is a description of your observation, which is then used in various ways to help understand the observation.

https://commons.wikimedia.org/wiki/File:Magnifying_glass_icon_mgx2.svg

"Observation is the active acquisition of information from a primary source. In living beings, observation employs the senses. In science, observation can also involve the recording of data via the use of instruments. The term may also refer to any data collected during the scientific activity." - Wikipedia (Observation)

It is these observations that we then start to analyze. It can be done in many ways. An observation as a bug report can then be inspected to its validity. Analysis might require communication with the stakeholders, tools, classification algorithms, etc. It is these actions that we employ to analyze the observation. It can be a snap decision or statistical analysis. Whatever is done during the analysis, there is an outcome. The outcome might be to trash the bug report, invalidate the observation, classification of the issue, pigeon holing an inference, describing a behavior in a more concise way, etc. Analysis creates something out of the observation.

Based on the analysis, there might be an action to deal with the observation. It can be a change in code, adding something to a document, building a new tool, fixing a leaking pipe, redefining an argument, etc. There might not be any actions towards the original subject of observation, but perhaps to the process with which we test and challenge. There might even be some actions to make the observation or the analysis different. Maybe a process improvement, learning a new skill, etc. Actions might require on the sub processes and further actions. In the end, however there is a follow up on the actions.

The follow up usually happens after the action. The follow up depends on the observation in a sense that there might be a need to reconstruct the situation in which the observation happened. There might be a need to refer to the earlier version of the subject under test. There might have even been a shift on the subject based on the analysis. And the action itself dictates the follow-up and the magnitude and the nature of it. The follow-up might require regression testing on a bug finding, another round of reviews, rerunning the test automation suite, rethinking, etc.

These four basic action became the guiding principle in all our testing processes.

Observation - Analysis - Action - Follow-up

But it wasn't enough. Every single bit of these actions required supporting activities. Observation required note taking, testing skills, tools, etc. Analysis requires processes, practices, domain knowledge, etc. During our workshop discussion I picked up some key words that were used. I then generated two clouds: The core activities & the supporting activities. The core activities are not enough on their own. The context states what kind of supporting activities are needed to make the core valuable. The supporting activities are of no value without the core, but the core loses value without any supporting activities.

Here are some of the things we came up.

Like I mentioned, all this is useless without the context. Every scenario requires a context that states the most useful way to approach the "bug" handling. Pair review requires different supporting activities than Beta testing, but they both have the core activities. The tools that are used might differ: you can use post-its, JIRA, QC, email, surveypal, etc. to communicate your observations. During the analysis those observation might be enveloped by a tool to create virtual stickers and notes. There might even be a template that is used to report an observation. Those observations might be classified, prioritized, trash or whatever. Based on the analysis at some point in time, something might be done. When I say might, it means that it is possible that an observation is lost and not acted upon. You can call it an "action" if an observation gets lost, but it is philosophical. Let us assume that every observation has an action. The action might require communication, changing something, tools, practices, processes, people, etc. Those actions then have follow-ups. That follow-up can be enveloped in the same tool that it was when it entered as an observation. It can even have a process of its own.

To conclude, there is no best practice to handling observations. Not every observation is a bug. Not every bug needs to be handled the same way. What was the most valuable thing I got out of my workshop was "mind the context!" Think of the value of your process to stakeholders. Think of the needs that need fulfilling. Think of the feedback loop. Think of the people involved in different tasks.

That is all today.

- Peksi

Let’s Test 2015 – conference at a glance (from a distance)

Just to get up again and write something, I decided I’d do something I’ve done before. I really want to join the Let’s Test 2015, but it seems I cannot. This blog post is similar to those I’ve done before for it is a “Conference at a Glance” kinda thing. So this is

Let’s Test 2015 – conference at a glance (from a distance)

It seems the conference is a 3 day thing with 2 key notes, 33 sessions or various kind. Since I’m a lazy person, I shall do the following:

• Tackle both key notes
• Choose one session from each day based on my familiarity of the speaker
• Choose one session from each day based on my interest in the title
• Choose one session from each day that I pick randomly

That should bring me to a total of 11 sessions that I try and grade. If you (as a conference speaker or as anyone else) feel like I should do more, just ask me in the comments section.  I try and keep it simple.

I will use a heuristic grading system (introduced here) to determine what would be the best session for me. I will grade the stuff with Angry Birds ™ grade – 0-3 stars per area – on five areas:

• Person-to-person (How will the person and his/hers work affect/inspire me or the people I know?), 
• Session value – short time span (How much can I get out of the session tomorrow – next year?), 
• Session value – longs time span (How much can I implement o my work and teach to my colleagues, my community?), 
• Steal-ability (How much of it am I willing to borrow and further develop to make it better and, more importantly, mine?), and 
• Challenge-ability (My past knowledge on the topic and my willingness to challenge the session contents.)

Ben Simo’s “There was not a breach – There was a blog”

Based on the description on the webpage, I quickly came to think I should have done something like that. I should have started blogging about something that is affecting major public. Like the Finnish Railway renewal or something else of a similar matter. Having not thought of the idea, I shall keep my eyes open the next time something big comes up. So “Thanks Ben for giving me a great idea. Don’t mind me copying it in the future!”

As we’re talking about a keynote here, I don’t think during-session-challenging will occur. As this is kind of an experience report, I feel I can absorb huge amount of wisdom and ideas from it. If only reading through the description gave me so many ideas already, attending the keynote might blow my mind. Alas, it might not happen. So my head is safe for now. Challenge-ability might be a bit low, but if you think the challenging as in self-challenging, things I can challenge in my own thinking processes, ways of working, how I present myself to others. On those parameters I see a lot possibility to challenge.

On that note, the steal-ability just went through the roof. As for long term value, I believe that short term value comes from bringing the conversation to the coffee tables for those who are not American nor have the exposure to HealthCare.gov. The long term effects lie in the the steal-ability I mentioned before. If I can introduce some public service kind of attitude towards my own behavior, it’ll really make a difference.

P2P-level is a bit tricky. Ben has been on my radar for years. I’ve been following him and reading his blog sporadically, but I never really came to realize how much of an influence he has been. There hasn’t been too much communication between him and me save for a few tweets now and then, or some random facebook comments to each other’s posts. I must say I should have been more in his face about stuff. I shall change that and get to know him better!

As for score (should I be at the conference I would attend the keynote whatever it might be):

• Person-to-person: * (I wanted it to be ***)
• Short time value: *
• Long time value: ***
• Steal-ability: ***
• Challenge-ability: ** 
• Total: 10/15 stars

Antti Karjalainen’s “Detecting the Heartbleed Vulnerability”

The description might not provoke my vastest interest, but I was part of the Heartbleed scene in a sense. While working at F-secure I heard about it weekly during the “hip season”. I didn’t particularly have anything to do with fixing nor working with it, but I was there to spread knowledge about it.

The thing is, I am the kinda guy who shies away from über cool tools, fancy technology and protocols. I have some interest in security testing, fuzzing and analytics, but I’m more comfortable to leave those to the people who know them better. When it comes to knowing what fuzzing does, I’m comfortable in what the Wikipedia says. That is to say with no disrespect towards the fine men and women who dapple with such technologies. Hats off to all of you!

P2P is a challenge for me, since Antti is a Finn and I might have run into him at some other conference. I can’t put my finger on it, however. I have never talked with him about fuzzing nor about tools (“the shying” and other excuses). The thing is, I don’t think I could talk with him about the fancy stuff. I might have a few cool comments like “that looks cool” or “whaddaya know”. The thing is, however, that I cannot say for sure. I wish there would be a common ground on which we could build a conversation and then work from that.

The short term value might be in form of an interest into fuzzing tools. Or to some yet unknown aspect of fuzzing I could use in my testing. I feel that the topic is so tool/technology oriented, I might not get enough. It also affects the long term value for I don’t think I am capable of transferring that knowledge to my community or colleagues. I don’t say there won’t be any inspiration during the keynote – I might change my way to think testing and tools.

And since my knowledge on the technology and the tools are so diminutive, I feel the steal-ability and challenge-ability are scored low also. This isn’t to say that Antti doesn’t rock most of the listeners’ world! I believe that he is the kinda guy we need more of. Just like Ben, helping regular people with their daily lives is what counts!

• Person-to-person: *
• Short time value: *
• Long time value: *
• Steal-ability: **
• Challenge-ability: *
• Total: 6/15 stars

1st day sessions:

Ilari Henrik Aegerter’s ”A Tester's Walk in the Park” (based on familiarity to speaker)

Ok. Having read the description, I’m torn in half: A session without a clear outtake or a fountain of good ideas. I don’t know quite yet. I do appreciate that the problem solving is the key to it, but are we talking about artificial, abstract problems like “where is testing going”, or concise practical problems like “how can I convince my manager to pay my trip to Sweden”? The uncertainty intrigues me in a way that I wouldn’t dare miss this session. If we can coax people like Michael Bolton or Ben Simo to join in with their problems, it can be a hoot. It could be a hoot with just me, Ilari and three guys I don’t know. The thing is “I don’t know”.

I have spent some time chatting with Ilari. He has coached me on different things, latest today (the April 1st 2015) on how to approach a testing communication problem. I know this guy and I like him. I have to admit I haven’t paid too much attention on his whereabouts the last 18 months, but I reckon that all will be changed.

The value of the sessions is a tricky one. I cannot say what value I can get for I don’t know the contents that well. On short term it might arouse good conversations and comradery between the people attending. Experience reports in the form of problem solving might be a good take-away. In longer time span, the technique itself might be a good thing to learn. The introduction of philosophical thinking and approach to software testing is actually quite intriguing. Challenging the session might be a tough one, since it feels like an experience report of a sort.

• Person-to-person: ***
• Short time value: **
• Long time value: **
• Steal-ability: **
• Challenge-ability: *
• Total: 10/15 stars

John Stevenson’s ”A Journey Towards SelfLearning” (based on interest in the title)

WOW! This sure sounds cool! A sessions in a form of a game show! Count me in! I am interested in learning and how people learn. I am a continuous learner myself and I think it is time to up that interest in me once again. If this session could make my learning more structured, I would get more out of the time I have at hand.

I know next to nothing about John Stevenson, but lately I have spotted some of his tweets. He could be one to chat more deeply about learning. Him and James Bach might be a good dinner guests if I wanted to talk about learning and teaching. Since I hope to be a teacher of a sort someday, I think they could give me valuable ideas.

The value I see from this session is vast, both long and short term. While stirring me short term, it could make me think about my life on a longer run, my education and my striving towards being a teacher (though Finnish teachers are paid really poorly). Since I know something about learning, I think I might even be able to challenge John on his ideas. Though I don’t think I can introduce any particularly new and fancy to his curriculum, I might be able to increase the value of the session by making him express thing in different ways to be more easily approachable.

• Person-to-person: *
• Short time value: ***
• Long time value: ***
• Steal-ability: **
• Challenge-ability: **
• Total: 11/15 stars

Louise Perold ’s "Non-violent Communication" (random pick)

Well well well. NVC. I must say I had need for those skills today when I argued with a developer on why the test cases written for manual execution are a poor excuse of a test automation. I did seek some coaching from Ilari (like mentioned before) but I wasn’t able to convey my point to him in a way that would have left both him (the dev) and me in a mutually enjoyable place of mind. Having said that, Louise’s session might be the one for me.

I don’t know her at all, though I might have traded tweets in the past. I’m intrigued meeting her, though. Should I not be able to come to conference, I really want to talk to her about Mortimer J. Adler’s book “How to Speak How to Listen” which I’m reading sporadically now and then. Also what I found out from Daniel Pink’s book “To Sell is Human” might contribute to the Non-violent communication.

Values from this session might be really high in short term and long term. I might not be able to transfer them to my community in a way Louise could, but I bet the example and behavior might influence my co-workers and other people as well. My knowledge on listening and conversation skills might enhance the challenge and steal –abilities, but I would have to let the time indicate what’s beneficial and what’s not.

Non-violent starring would be as follows:

• Person-to-person: *
• Short time value: ***
• Long time value: **
• Steal-ability: **
• Challenge-ability: **
• Total: 10/15 stars

2nd day sessions:

Laurent Bossavit & Michael Bolton’s ” "Defense Against The Dark Arts” (based on familiarity to speakers)

Critical thinking, they say. Very well. I would expect something like this from Bolton and Bossavit. Given Laurent’s book “Leprechauns of Software Engineering” it is about time to teach us Earth dwellers them skills to tackle possibly harmful (and perhaps even rigged) information.

I know both of the speakers and I love spending time with them. Both incredibly intelligent fellas with excellent ideas and views of the world. It seems almost a loss that I might have to miss their workshop. But I shan’t weep! I find their session really intriguing so I might badger for a coaching sessions should I need to miss the event.

As for value of the session, I see high value altogether. Critical thinking skills and the practical application of it are worth gold in the testing industry. Whatever I can bring home from that session would be valuable material for my colleagues in ways to think critically.

Since I have some former knowledge on critical thinking, I feel the challenge-ability is high. Also I see that the whole session is about challenging, I feel compelled to challenge much of their material. Steal-ability is high in a sense that I want to educate my colleagues on that particular subject.

My god, I’m looking at quite a session:

• Person-to-person: ***
• Short time value: ***
• Long time value: ** (it’s really 3 stars, but I cannot give the highest score yet)
• Steal-ability: ***
• Challenge-ability: ***
• Total: 14/15 stars

Huib Schoots’ “How to be an Explorer of Software” (based on interest in the title)

Mr. Schoots talks about creativity? The guy who had a conference sessions in which he mostly played music talks about creativity? I think this is the stuff everyone needs to see. The title pushed me in a quite different assumption on the contents of the session. I thought that it would be about hands on exploring something, but since the description kinda gave the impression that it’s about how we document and observe, it actually fulfills the title quite well. Interested I surely am.

Huib is a great fella and he could be one of the top 3 reasons for me to attend. He is the bloke that makes people smile on their worst day. All that and a world-class tester! Say no more! I have seen Sami “the Monkey” Söderblom talking about exploring, read (the beginning) Elisabeth Hendrickson’s book on exploring, so it would be nice to see another take on the subject. The value is hard to define, since I feel the values are more personal than community wide spreadable thoughts. To be able to more concisely test and document a software is a great skill to have.

To challenge Huib is something I think he would enjoy more than anything, so I think I must get in touch with him and get my hands on the material if I can’t attend. I might also be able to steal something that I later present something as my own, but I ain’t gonna let him know. It’s hard to put a finger on the stuff that I’d like to steal, but I bet there are loads of stuff.

The exploration of stars…

• Person-to-person: ***
• Short time value: ***
• Long time value: **
• Steal-ability: **
• Challenge-ability: ***
• Total: 13/15 stars

Erik Davis’ "Effective Practice Manager" (random pick)

A practice manager. That’s a new term for me. Based on the description I didn’t get what is a practice manager. Is it like a skill coach or a competence manager? Either way I feel this is an experience report with some educational features. Oh it’s an experience report with discussion. Maybe we have some cool practice manager problems we can help solve for him. Challenging might come to play while there’s a discussion in the session, but I do know so very little about the subject beforehand.

I was intrigued by the “increase their own impact at work” bullet point. That is something I want. I want autonomy and to see my own passions and skills realize in my daily work. In that sense this session might be a good catch. It would definitely have long term effects but I couldn’t get the short term value out of the description. Maybe it lies in the “whatever else comes out of the discussion”.

I don’t know Erik at all. He might be one of those blokes who have avoided my radar so far. Based on his personal description I think I he would be the guy to get to know. Maybe I’ll ask him for a beer at the evening activities. His interest in educating testers and building their skill set is something I want to do also.

• Person-to-person: **
• Short time value: *
• Long time value: **
• Steal-ability: **
• Challenge-ability: *
• Total: 8/15 stars

3rd day sessions:

Jari Laakso’s ”Security Testing” (based on familiarity to speakers)

Before I have even checked the description on Jari’s session, I must say I am intrigued by this. He is my first touch on security testing on my Finnish blog years back. Unfortunately, I haven’t met the guy. I know he’s intelligent and capable tester, but I haven’t been in contact him for some time. Maybe I have to rekindle that connection to get the latest.

Ok. The description. Very well! Hands on security testing! Where do I sign? This is like one of the essential stuff that people are asked to when doing exploratory testing. SQL injections are in every text book on testing, but this guy actually shows how to do that. Nice! I can see the value skyrocket! The thing is, to be able to share this knowledge I should know even more about the subject. I might lack the interest in security testing in general, so I might not be able to teach these skills to others (I’d ask Jari to do it for me).

To challenge him might be difficult due to the fact that we are talking about technical stuff once again. Things like cross-site scripting are founded on protocols, REST-calls and whatnot, and I don’t feel I can challenge him on those. I might wanna try, tho. ;)

• Person-to-person: ***
• Short time value: ***
• Long time value: *
• Steal-ability: *
• Challenge-ability: *
• Total: 9/15 stars

Alexandra Casapu’s “Examine Your Testing Skills” (based on interest in the title)

Hands on testing and discovering our testing skills. I like it. In EuroSTAR test lab I had the chance to dapple with Mr. Lyndsay’s machines and I really want to have another go! Besides I’m interested in how my skills map out. I don’t know Alexandra, but I’ve heard of her. She is one of those people I want on my “meet these people in the Testing Scene” list.

The idea of having various non-technical skills to help you with testing is interesting. I would like to know other peoples’ skills and if I can borrow their passions and learn what they know. I’d be able to steal ideas from the participants and from Alexandra. Very nice! Since I have been doing some reading on skills, I might be able to challenge her and my previous views quite easily. And skills practicing in practice is always something to take home to.

• Person-to-person: **
• Short time value: **
• Long time value: **
• Steal-ability: **
• Challenge-ability: **
• Total: 10/15 stars

Scott Barber’s  ”Experiencing Product Owners”  (random pick)

Random in deed… Scott, if you’re reading this, you might want to check the description on the website. ;)

A big star on the effort, though.

Conclusion

Having surpassed the pain of starting to write again, I feel joyous on how I managed to rate 10 (Scott’s doesn’t count) sessions on five heuristic parameters. The toughest thing is ahead of me, though. I need to convince someone to pay my trip to Sweden.

Vi ses!
Peksi

You gotta fight for your right to test!

I am terribly sorry for the soon to be rant and biased output that is going to happen. I have did this once, but the conversation resulting from that was rather pleasant and constructive. This is a comment / response / rant about a blog page I stumbled upon today. You can find the original post here. The page may have been outdated since it was created in 2010 but it seems that it was updated half a year ago.

My first though was “not another test case” when I read the article. I’m currently trying to figure out what I will write to my next column on the “Testaus ja Laatu” –magazine, and since the theme is juxtaposing, I thought I’d try to write some blog-stuff first. One of the topics could be “test cases vs. no test cases” and I shy away from that strict way of thinking. Exploratory testing, to me, is utilizing all the available tools and gimmicks to get to the best results. Black and white –world vision narrows too much my take on testing.

Having said that, I will use my recently found taste to logic to point out what I found could be wrong in the article. I do not know the state of mind in which the article was written or the context in which it is supposed to be fitted, so I will project it to my own workplace where that is necessary.

You are warned…

The first sentence goes like this: “A significant factor in a successful testing effort is how well the tests are written.”  I have no idea how much significant is this case. I would guess that it is either the largest part or the one after that. When I do testing, the significance of the test cases are miniscule, and I tend to do successful testing. Not all the time, but I know a certain cases where properly written test cases did not contribute to the successful testing effort. Also I don’t exactly know what qualifies as successful testing effort. It could mean having run all the tests within the schedule (which doesn’t quality for successful for many reasons), having written all the test cases (do I need to say more about this), the product is shipped to end user (lots of products have been shipped to customers and later fixed due to poor quality), etc. So I would say that well written test cases /may/ be a factor in the successful testing effort just as well as poorly or not-at-all-written test cases.

“They have to be effective in verifying that approved requirements have been met AND the test cases themselves are understandable so the tester can run the test as intended by the test author.” 

So the test cases have to be effective in verifying requirements. Granted. Do the /have to be/ written. No. Even poorly written test cases can be effective at verifying requirements. I think the ability to verify something comes not from the writing of a test case but from the skill of a tester. If a tester is skilled there may not be need for any written test cases to get the job done. “--approved requirements have been met --” So anything that is not written to requirements doesn’t get tested? We also have expectations about the product that don’t really qualify as requirements. Testers still need be aware of anything that might threaten the quality of the product.

“The 3C’s: Clear, Concise, Complete. Test Cases are not open to multiple interpretations by qualified testers. Ambiguous or nonspecific Test Cases are difficult to manage.” 

What if we do not know about the product enough at the time we write the test case? Should we wait until the product is finished and then write the cases? Isn’t that just huge waste of time and money? And when it comes to managing test cases, I think lines of text are the easiest to manage, it’s the people that might require managing. It is true that estimating coverage and depth may be difficult if the test case is ambiguous. I also think that it is difficult to estimate those with good and precise test cases. People are best to estimate the depth of their testing and the confidence in their testing. ASK IF SOMETHING IS AMBIGUOUS. -> Fight ambiguity with openness and communication

“Test Cases are easily understood.  They provide just enough detail to enable testers familiar with the functional area under test to run the test case. Note:  Detailed click by click steps are only useful for automated tests.” 

This is something that I agree! Test case can be easily understood even if it’s like this: “Play around with the product and describe the person next to you what it does.” It is both easily to understand and you already have enough familiarity because the base requirement for that is none! This could be the baseline for any testing ever to be done at any context. FAMILIARISE YOURSELF WITH THE PRODUCT.-> Fight ignorance with eagerness.

“Test cases include setup requirements (environment & data), expected results, any dependencies on other tests and tracing to the requirements they validate. Are traced to the results of each test run, including defects (bugs) discovered, test platform run on, software build version used.” 

For the data and expected results, I recommend all you read Michel Bolton’s and James Bach’s conversation and decide yourself if the test case is complete with expected results. It is necessary to document stuff that mentioned in the text so to avoid unnecessary overlapping. But traceability to test cases? Is that possible for bugs that are found during the test case execution but outside the intended observation area? BE PREPARED FOR THE UNEXPETED RESULTS WITH UNEXPETED INPUTS. -> Fight patters with chaos and vice versa.

“Measurable: For each test case, there must be a way to objectively measure that the test case has either passed or failed. All test cases must be linked to the requirements that they verify to enable impact of changes made to the requirements or solution designs to be identified (measured).” 

OK. Let’s say that the test is executed perfectly and the result is exactly as expected. A minute after that the computer crashes. Does that constitute as failed? How much do we actually know about the product to say that a test has passed or finished. And if we want to find bugs, isn’t the test passed only if it finds bugs? I think test only fails if you don’t run the test, because it didn’t test anything. TEST IS ONLY FAILED IF IT IS NOT RUN. -> Fight unnecessary documentation with rightly timed planning. I.e. keep the time between planning and doing minimum. Preferably do them at the same time.

“The test case must have been approved, prior to being run, by the key stakeholders of the requirements that the test case verifies. Any changes made to test cases caused by requirements or solution designs changes must also be approved.” 

There is ABSOLUTELY NO POINT IN THIS! Why the hell do we need approving for our testing? Only thing that requires approving is the results of our testing. If the stakeholders do not trust us to test the software, we could record everything we do and ask them to audit the test material. I would think they are happier to audit actual results than worthless, constantly changing trivial documentation. IF WE NEED EVERY TEST APPROVED, WE LOSE MONEY AND TEST COVERAGE. -> Fight über control with proper* documentation.

“Realistic test cases DO: Verify how the product will be used or could be misused, eg positive and negative tests. Verify functions and features that implement approved product requirements. Can be run using the platforms and software configurations available in the test environment.
Realistic test cases DO NOT: Verify out of scope functions and features. Verify unapproved product requirements.”

I agree with the first sentence – a good test could test how the product could be used/misused. Also how it should/will be used. How much does the product actually differ from what the customer actually wants? I also agree with the second sentence, but it should be broader. It should test the features, platforms, integrations, data integrity, security, performance, etc. I don’t understand the third sentence for if we don’t have the proper environment or setup, we should acquire them in time. If we do have a scope (referring to the fourth sentence) then I agree that we should not spend too much time on out of scope elements. They could be mutated or broken due to changes made somewhere else so regression/smoke testing should be implemented to out of scope areas. If the fifth sentence I don’t understand what is meant by “unapproved”. Unapproved by who? By the testers, client, developers, managers? It really depends, so making a claim like that is just nonsense. A REALISTIC TESTCASE IS REALISTIC BUT FLEXIBLE. -> Fight rigid descriptions with autonomy, critical thinking and challenging.

“Test Cases must be able to be successfully completed within the project scope and schedule.” 

There is no point on writing test cases that are not run. But “must”? And why do they have to be “successfully run” and is a failed test a “successfully run” test? If a test finds three hundred bugs, is it successfully run if it doesn’t reach the expected result? WE CAN ONLY RUN AS MUCH TEST CASES. -> Fight the quantification of test cases with the amount of time spent doing testing. Count time, not test cases.



To summarize, I think the whole concept of SMART test cases is wrong excluding few things that I agreed with. I also encourage to keep the writing of test cases in minimum and the amount of testing done in maximum. Use the time wisely and appropriately! But if you do, you should consider these instead of the SMART way:

• Fight ambiguity with openness and communication
• Fight ignorance with eagerness.
• Fight patters with chaos and vice versa.
• Fight unnecessary documentation with rightly timed planning. I.e. keep the time between planning and doing minimum. Preferably do them at the same time.
• Fight über control with proper documentation.
• Fight rigid descriptions with autonomy, critical thinking and challenging.
• Fight the quantification of test cases with the amount of time spent doing testing. Count time, not test cases.

I will leave you to this. I promise I will be back with more about test cases (if I have the time). I’m not saying “don’t write test cases”, but use your head! It is not smart (bun intended) to follow some rigid procedure for all context, but to adapt to situation and to make most of the time you have to test.

- Peksi

* Proper: Appropriate as automated as possible. Including video recording, automated logs, scans from scribbles, session sheets, statements, etc. What is required to get enough information to the decision makers without hindering the job of a tester.

Reading Practice

I’m reading currently a book called ”How to read a book?” by Mortimer J Adler and Charles van Doren. I have just started it but I talked about it in the office the other day. One of my colleagues said that one way to read a book is to first read the table of contents and the read the first page of every chapter. This post is an experience report on what can I learn by using a different reading strategy. This blog post will act also as platform for my thoughts when I read my target book. It was inspired by Erik Brickarp's blog post on taking notes.

The first pages of  “How to read a book?” book states that there are two ways to learn from books: to illicit information, and to understand the book.  My goal here is to run this exercise to illicit information. I choose a book called “Being logical” by D.Q. McInerny as my target book.

Preface

I decided to read the preface to be more in tune with what is going to be written about the subject. Forewords usually try to tell the reason behind why the book is written and to inspire the reader to go on. So when I got to the end of page 1, I decided to go on.

I had read the first paragraph of the book earlier (doesn’t count as cheating) and didn’t continue because of eldritch terms and words. Having read the preface, I the realized they will be explained. The book says it will concentrate to leave as little room for assumptions as possible thus trying to be explicit rather than implicit. Also it is not a text book for a class but a practical guide. This made me interested even more.

The preface addresses the structure of the book. It seems to be in five parts; each part building on the previous one. I’m interested if the parts contain what they were supposed to contain. The first part is told to be preparatory, the second one to lay the foundations, the third one to the meat and potatoes of thinking logically, fourth part discusses attitudes and fifth discusses fallacies. I know that I am fallacious, even more than regular people, so it is interesting to see what can I learn about myself while reading.

The table of contents

First thing that came to mind was that never before have I really read through the table of contents. This is the first time I really look into them. I usually skip the pages altogether to get cracking with the reading. Truth be told, I was quite excited to see how the table of contents was structured. It felt like a story was already told with the items on the list.

I pretty soon realized that every chapter had 8-30 sub chapters. Each chapter was about 1-5 pages long. I was starting to question my decision to read only first page from each chapter. For most of the sub chapters it would get the whole chapter done already. So I decided to read only the first *subchapter* of each chapter. They were, according to the table of content, almost always the introduction or foundation for the chapter.

Also I noticed that as for this book, it makes a great checklist for practicing critical thinking. By looking into the titles of each sub chapter, you would get a coherent guide to practice logical thinking. I could create a powerful mindmap out of those and print it on my cubicle wall. (And I will share it with you! Tadaa!)

Feel free to print this also!

Also this was a great stage to look for words and phrases you don’t understand that are essential to the understanding the content of the chapter. “Syllogistic”, “agnosticism”, “antecedent”, “equivocation”, “Post Hoc Ergo Propter Hoc” and “expediency” were new to me. Now that I checked them, I know better.

1st Part – Preparing the Mind for the Logic

I tackled it with my usual gusto; get cracking with it and read it again if I need to. And again I needed to read it twice for me to understand the message. I think I shouldn’t as this type of reading is just to get information. Sadly I didn’t /read/ it, I skimmed it. And after the second reading, I realized that the first sub chapter was exactly about that: Paying attention. And I had to pay, for it took my time and time is money. The book encourages you to listen instead of hearing, to see instead of looking. It gives great insight to the foundation of logical behavior.

The table of contents told that topics like “Effective Communication” and “Truth” were in latter parts of the chapter. This is the problem of this kind of quick reading, I see interesting stuff and I don’t want to stop reading. But I must refrain from reading more as I was to read this thing quickly, so on to the next chapter.

2nd Part – The Basic Principle of Logic

At this point, the technique I chose was beginning to bother me. I did read it fast, but this was only a glimpse of what it could have been. The principles could have been more thoroughly read but I chose to stick to my plan. The 1st subchapter was about the first principle of logic. Before the 1st subchapter was a brief piece of text regarding logic as three separate but linked entities; as science, as art and as skill. This did raise some questions, but I think they will be answered if I read the book with thought.

The first subchapter tries to focus on practical side of things. The theory will be discussed elsewhere, but I was intrigued by the the notion of reading even more about the theory of logic. This subchapter proved some examples which clarify the link between logic and human reasoning. Again I was tempted to read on, but managed to jump to the next chapter.

3rd part – Argument: The Language of Logic

The subchapter was about founding an argument. I did have some experience in the practical side of forming an argument, as I had been practicing that with Michael Bolton on Skype, so I knew the elements to build an argument. Those were “premise” and “conclusion”. With the help of examples the book showed some good uses of supporting and supported arguments, giving me a good ground to build upon – in fact just like the preface promised.

As this on itself might have been a good starting point to a person not knowing anything about logic and was just seeking a quick 15 minutes intense session on logic, this only made me hungrier to read through the whole chapter. I noticed that, if not the writing style or the content, my eagerness to read on was huge. I had to struggle every time to quit reading further.

4th part – The Sources of Illogical Thinking

I was a bit taken aback with the concept of writing about things that made you illogical. When writing about negative stuff some might absorb the “bad habits” instead of seeing them as negative. After reading the subchapter, I felt that it only made me stronger thinker and I was trying to spot those weaknesses – you might say - and try to find ways to mitigate them.

Skepticism was the topic of the first subchapter and it was really informative. It gave insight on selective skepticism and to behavioral skepticism – and gave some concrete examples. I began to think immediately about exercises I could to using skepticism in my coaching. I may have already done that but unintentionally. Not I could be even better at forming an argument. The book made some examples about damaging skepticism but tried to enforce the power of healthy doubt – doubt as a catalyst for learning.

5th part – The Principal Forms of Illogical Thinking

This 5th part was about fallacies. They were explained to be the typical patters people do/use to act illogically and to build illogical arguments. Usually fallacies appeal to emotions so they are more powerful than sound logic. Again by recognizing the patterns I might be better at building an argument that is based on sound logic.

The lure of using a fallacious argument is high, because it might be quicker and more effective than using sound logic. Even though you’re right and have a sound logic behind the conclusion, basing that on fallacious argument is almost never a good thing – it might bite you back when you least expect it. I had read something about the fallacies so this area in particular was really tempting.
Afterword
The afterword was a good reflection through the book. They stressed the concentration when arguing so you focus on using your logic in building an argument.  I rarely read the afterword as I see no input to the meat and potatoes of the text. I did realize that the afterword did provide a fresh view to the book as a whole. It might even be effective to read the afterword /before/ starting to read the book itself as it can provide a fresh view to the book before trying to tackle the reading.

About the exercise

After having done the exercise, I feel it was a good practice. I might use it again but with a twist – I will write up questions about the first page/subchapter to a piece of paper and try to answer those questions as I read on. In that way not only will I illicit information, I will begin to understand the meaning of things.

The exercise was good fun as the text was close to my liking – it wasn’t a textbook but a guide, the topic was near to me, and it was short enough – so the exercise was a bit easy. It was good for a first experiment to this kind of reading, but I need to make this process more efficient. I might even learn more about this type of “stub skimming” (if there is no word for this type of reading, let it be that).

So right now the process goes like this:

1. Choose a book to your liking (any book will do)
2. Read the table of content
3. Read the preface
4. Read the first page of every chapter
5. Read the afterword

On every step, make notes and questions about the content
If there are fundamental questions about the content, read on and find the answer.

Thoughts

This is just one way to read. This is more like giving an indication of what the book is about. This is not used to understand to book but to have some knowledge of what there might be if you’d read on. This is a good way to tackle a book on before you read it thoroughly, because you already have questions about the content. On the time you re will systematically read it through (or parts of it) might be significantly faster and make it easier to understand.

Now I need to continue reading the book about reading a book. I think I will use this technique to give me a head start.

Testgasm

This is a blogpost to describe what was going on in the Rapid Software Testing class held by James Bach. This is not meant to be a comprehensive analysis on what happened or the lessons I learned but few highlights and hindsight analysis on what happened.

The dice game

...this is just a few of what we had...

Those that have been attending the dice game know that the initial exercise is hard. When one gets the idea on how the algorithm goes it’s easier to play variations on the game by group of friends or colleagues (not to say these are necessarily different). I (thanks to Michael Bolton’s course), Henri Hannuniemi and Sami Lehtonen (thanks to Antti Niittyviita) had played the game earlier and so we were grouped together and given a different algorithm than the others. I was so thrilled by the idea that I could use all the skills that I had learned to crack that nut open.

The game started and we were given a bunch of dice: a handful of regular, different colour/size dice; few D20 dice; few D10 hand drawn dice; a die that was in a transparent die; some “poker dice”, etc. We had so many variables that we became a bit confused. I started a list of different things that we could be analyzing. The size, the colour, dots or numbers, amount of dice, type of dice, arranging of dice, “zero is not zero” etc. We then arranged them so that there were different D6 coloured dice stacked together in different ways, five groups of five. James came to the table, we said “2” as a guess for every single dice pile and James replied “0”.

We had the miscellaneous dice lying in the stack nearby and I thought I ask a reply on that also. I said it “2” also and James replied “1”. A one? We had a huge number of different types of dice lying there. What could it be?

We proceeded to arrange the dice in a fashion that there were 2 or 3 special dice and regular dice bundled up so that all special dice were in use. All but one group ended up in a “0”. The one that was the “1” had the die where there was a dice within a dice. A frantic math calculation ensued. We tried to sum up the two dices together, but the result was always a “1”. So I took the die in hand and turned it so that James could see only “1” and a “2”, and we thought it was a “1”. The answer was “2”. Frustration!

Testgasm

We then analyzed the steps we took to get to that “2”. We had a couple of theories, but when the die was on the table, it was always a “1”. Heureka! We took five dice off the table and said “5”. And a five it was!

The thought process took 3 persons 20-25 minutes. The sparring between the team enabled us to try different even crazy-sounding ideas to good extent without exhausting our innovation. We simplified the data and made it more complex. We tried to look for changes in the output by varying the input. We used the different models of problem solving from our lives to figure out the pattern. We also solved a pattern composing of vocal input pattern (I save that for one of my exercise in the company) and one with a difficult mathematical pattern composing of differently grouped dice.

"Give me your hardest problem!"

Like James said, we were doing testing Kung-fu! Solving problems like matrial artists! Bonk! Blam! Thwaak! Zlott! Swoorsh! Phatam! Ka-pow! Like Batman fighting crime, we fought problems! (I’m getting a bit excited here, if you can’t read it from between the lines.) We all felt like we were invincible and we had the tools to crack every problem in the world! “Give me your hardest problem and I will solve it for you!”

The high after testgasm

After we got all the patterns solved we were all ecstatic about what we had done. We were in a problem-solving high! The word to describe the feeling would be “testgasm”. And truth be told, after a serious testing session where you find something awesome, something really important; you will get a testgasm. I’m not sure if other that testers understand the rush after a successful testing session; it is hard to explain. The joy of completing a hard task and feeling joyous about it in ways you never thought you could be. That feeling is something that testers all around are looking for when they test. Your face may light up and you yell “Yes! I got it!” and other people in the cubicle stare at you like you’re deranged.

Focus/Defocus

I had heard the concept of focusing/defocusing in Michael Bolton’s class before but I never truly understood the meaning behind it. ("If focusing is focusing, defocusing is the opposite of that") I tried to look for material online, but it was all vague and didn’t make an impact. The way that James described the method was simple: “When confused, focus; when frustrated, defocus.” Wow! When testing one sometimes loses momentum and struggles with a single piece of data for long time. Using input patterns that don’t find the problem may lead into frustration. I felt exactly that during an exercise about systematic testing.

James had us testing a piece of software that had a bug in it. We were asked to find the bug and then try to figure out why it fails. The input was a valid IP address. You know what I figured out? I have so strong built in mental models about meaningful number combinations that I was grinding my teeth and sweating to break that pattern. I had tried a pattern by testing the high and low numbers, duplicate numbers, you name it. Could I just be blind to the bug? Then James said “Look into your data. Can you see a pattern?” Yes I did. And lots of them! “Now try to find an input that is as different as possible from previous data.” And guess what IP address I used? “1.2.3.4” That’s right! I broke my pattern by trying “1.2.3.4”! What was I thinking?!?!

James was like “Dude! What the hell?” and I sat there frustrated and confused. Then I realized what he had meant. If I had drawn a line to represent my test data, it would have been like this:

I just needed to break the pattern and be more random and use data from between my clustered data pattern. And with first random IP address I put in, I found the bug. Later James explained why the program behaves like that, but I can’t remember the true root-cause of the bug, but I did learn a lot about focusing and defocusing.

Hung over (from learning)

It has now been two weeks since the course. After the class I was in a high that lasted through the weekend. I thought about giving a thorough analysis about the course but I see no need for that. I had the time to let the learning infuse to my spine and now I feel that the stuff I learned make even more difference. I did have an information hangover after the class and it was hard to get back to grips with non-testing work again. I was glad, however, that I could attend the Intensive course the next week so I wasn’t that bummed.

I recommend the class for everyone willing to improve their critical thinking, testing skills and/or argumentation skills. The exercises were good and to the point. The hot seat treatment James gave to some of us gave us experience to stand pressure and perform well when in a stressful situation. I also learned a lot about myself and the way I learn. It was also cool to hang out with great testers like Samuli Elomaa and Aleksis Tulonen.

I still have a long way to go and a lot of learning to do. (After James mentioned it) I began to think myself as a constant student, always learning. I promise James (and I will talk about this to Mr. Bolton) that I will try to compare the classes done by him and Michael. That way both of them could learn what could be done better or differently. That however will have to wait a few days (i.e. weeks) as I have a growing backlog of blog posts.

Flashback to Learning

I rarely have the concentration span to write complex and analyzing posts. If I believe that the subject has more to it than I can scribble in 90 minutes, it will more likely become a candidate for an article and will be more fine tuned than a regular blog post. Now I thought I would write an update to my blog describing the second day at the Rapid Testing Course, but then I thought writing another a diary-like entry was not exciting enough - and I want my scribbling to be at least a bit exciting.

So why do I have to make a blog post about something that I’m not willing to investigate? I don't! That's the beauty of it! But I want to make a quick post on something interesting. In order to do a blog post from something that requires little to no research it could be about an experience that is worth sharing. This post is not from today’s class but from past but the topic was brought up today in the class.

James Bach and Michael Bolton use the Socratic Method to help learn stuff in their courses. To simplify, they ask a bunch of questions from a tester to squeeze out an answer - not possibly the right answer but one good enough. I got my share of the method in the class but I’m not going to talk about that. I’m going to talk about an event that happened 4 months ago. (Enter fairy bells tinkle and a wobbly picture to demonstrate a flashback...)

I wrote a white paper about six months ago about heuristic testing implementation that was discussed in a Peer testing session late last year. The article was about to be published in a testing magazine and it was fine tuned for couple of months by the editor and a couple of voluntary proofreaders. I still wanted to have a comment from James Bach and Michael Bolton as I have a respect for their opinion in all testing related issues. And so I sent a reviewed draft to them and waited for approval.

What then happened, I couldn't expect. I had high hopes on the content of my article as it had been reviewed multiple times by bunch of different people. I thought I was in a verge of a breakthrough and praises would that flying my way. Instead, James’ reply was simple:

“I'm completely opposed to this.”

What followed was an exchange of emails between me and James about the problems with the article. It concluded in a suggestion to remove the essential elements that I based the article on. I thought James had got it all wrong and I requested him to discuss the difference in views that we seemingly were having. Skype was the medium of choice.

We ended up talking for 90 minutes. First we were talking about the article itself and he clarified some points he had made earlier. Then James challenged some of the essential elements of the article and I tried to defend them as best I could. All the while doing so I thought better descriptions for the principles I was using. I was on hot coals during the conversation: if I had chosen a bad wording for my answer, he would grab it and ask the real meaning behind the words I was using.

The article did not survive, but I did. Later I was a happy to have endured the barrage and decided to hone the article so that I could later publish with pride and joy. I never really understood how much the discussion had given me, though. I had learned some very valuable lessons. The critique on the article was secondary compared to the practice I got performing under pressure and all the while holding to my beliefs. As James said in the course today, there is none worse than tester who is “pathetically complying”. I could have said “OK. I won’t publish! Just don’t yell at me.” and I would have lost all my credibility. I learned how to defend something even though I might be wrong. Ilari Henrik Aegerter gave me a coaching session once where we practiced exactly that: how to be able to defend your point. And so I became better at explaining myself and defending my point under pressure.

What I would have done differently is that I never really tried to sympathize to James’ point of view as I so aggressively tried to get mine across. I could have got more out of the conversation if I had listened more carefully and sought for meanings and advice that he gave between the lines.

Having shared this experience also enforced the learning that I experienced, for I was able to verbalize some of the stuff that I had thought only briefly. Next steps for me are to revisit the article bearing the real purpose of the article in mind, and try to make it better (if not perfect). The article is resting and collecting dust currently, but I will tackle the beast soon enough. I believe that the Rapid Software Testing course will give the inspiration boost that I need to get it done (although I think extrinsic motivation plays no part in this, as I already WANT to finish the article). Too bad I didn't sign in for the Let's Test! conference with the paper. ;)

One down two to go!

And no, I will not link the King Diamond song here (even tho it’s cool). This blog post is about my experiences from the first (my second first day) at the Rapid Software Testing course by James Bach. I seem to be one of the lucky few who actually are able to have two goes at this amazing adventure to the deep dungeons of an exploratory testing mind.

Like I tweeted earlier, I fell into traps; the most obvious traps that can be set for a tester. I know I can do better so here’s a quick analysis on what happened and what could I have done better:

http://morningandotherstories.wordpress.com/2011/12/

Context: When I was confronted with a testing problem without a context, the first thing to do is to determine the context. Right? Here’s what I did: I thought I might sound too clever on asking and challenging, so I answered the question without determining the context. This lead into questioning my answer and making me realize that “we’re shooting live ammo here”. So I decided to sharpen up.

• I could have done what I (try to) do every time I’m facing a problem: Understand the context, the people involved, the background, the current status, the mission, etc; communicate with the client, stakeholders, testers, etc. That could have brought the whole problem into a new light. I fell into a clever-trap. 

Pattern and mental models: I had the advantage/disadvantage to see the formula for a pattern beforehand. The task was to determine what the pattern was. Those that have taken the RST course will remember this, right? I remember the point of this exercise being to break the assumption and to focus and defocus. Again, my previous knowledge bias struck in: I completely ignored testing the product and did some of the procedures I had seen done in previous classes. The pattern was the same, however, so basically I didn't lose anything…

• …EXCEPT FOR EXPERIENCE! I could have spent the time testing in ways I had not tested before. Now I had the time to do something different. Instead I tried to look good by using Excel (like James Whitaker, I think) and I made a simple script to produce test data. I didn't even bother to check the logs, as I had checked them numerous times before. Should they have changed the behavior even a bit, I wouldn't have been able to identify the pattern. I fell into a showman-trap.

http://members.multimania.co.uk/mrmal13/saints/amardas.htm

Documenting: The task was simple: Write tests and justify all of them. James specifically asked everybody whether they had made a justification for ALL the test they were going to run. An exploratory tester documenting test cases and justifying them all with specific, detailed texts? Can you picture that? I could hear myself saying within my head “Yes, James” without even questioning my so obvious bias. I tried to be a good boy and wrote them cases and then feel pride.

• The exercise was about justifying the testing but in a different way. My doing inexpensive testing and discovering something worth discovering, one can justify a more expensive testing. Like James said: “The cheap tests pave the way for the more expensive ones. They are the scouts that look for a good spot to place the artillery.” That comment alone is worth its own blog post. But instead of being inquisitive, exploratory tester, I became a test designer that is going to get fired. I fell to the guru-awe-trap.

So my first day was a success: more learning than I hoped for. I also thought that I had some form of advantage of knowing the stuff from previous, but it seems that I had misunderstood (or belittled) some basic principles of Rapid Testing. I felt some of the stuff explained more comprehensively, with sidetracks to bulk up the areas where I thought the content was thin (namely the modeling and project elements of Rapid Testing). I do wish James would have covered the modeling heuristics with more depth, but I will leave room for other areas for the next two days.

What I’m looking forward is the process of Rapid Testing itself. I somehow missed a whole lot on that subject when Michael Bolton was teaching the course. Also to hear more about the oracles would be nice. We scratched the surface today but we’re still to cover some of the mnemonics that I frequently use, like CIDTESTD, HICCUPPS, etc., but it was nice to see an update to SFDPOT with the Interface addition. I will be using that when I reinvigorate the workshop in near future.

The importance of the context

So as I’m heading home, I will share this one particularly funny event that happened in the beginning of the class. Here’s the story without context:

James Bach spit on Aleksis Tulonen.

If you add context, here’s what really happened: Some poor fellow was getting given the “Socrates treatment” by James and he had an act prepared for the response the guy was going to give. He asked a question and set a trap onto that question. He knew the answer would be something “horrifying”. So he took a sip of water after asking the question. When the poor fellow answered James theatrically spit the water from his mouth as if being so stupefied by the answer that he just couldn't bare it. Unfortunately the aim of the jet of water was a bit off. Aleksis, who happened to sit in the front row next to me, was a victim of the poor aim. Aleksis’ phone and computer got some droplets on them as collateral victim of this splendid act. Hopefully tomorrow’s newspaper doesn't have a title “A testing consultant got spit on by a guru”.

The magic medallion of testing - summary of a keynote by Rex Black

This is a summary about Rex Black’s keynote about Testing and Quality management at Nordic Testing Days 2012. This was the first keynote on the second day. People were still tired from the evening activities and some even skipped the keynote altogether. I was keen to see Rex’s track as I had some… opinions about ISTQB and the certification system as a whole. So, to be there and be able to challenge him was an opportunity I didn’t want to miss.

On Monday we had "argued" with Rex about metrics and coverage. Rex commented Sami Söderbloms statement about not measuring coverage by giving an example. The discussion went something like this: “With all things being equal, the first tester returns his test report with 90% coverage and another with 15% coverage. Which one would you trust more?” Rex commented. I raised my hand (the inner voice screaming "Me, me, me! Let me answer!". Sami then said that I could answer the question. I tried to question the true coverage by asking about what has been found. Rex replied by pleading to the all-things-being-equal statement. All things that were found were supposed to be equal (so as important bugs, issues, etc.) “But all things are never equal in software testing”, I said. Later I realized that if all things were equal, how can there be a difference in the coverage?

If we have all things equal, isn’t the coverage supposed to be equal too? If two clone testers are put to test a product in exactly the same way, don’t they both produce the same amount of coverage in any way you measure? Change anything and all things are not equal. So if in an all-things-being-equal-situation we have difference in coverage, our model to measure testing might be faulty, the person interpreting the results may be biased, etc. So how can one make an example from a situation that is faulty by default?

The keynote however began with Rex warning about using the c-word (coverage) and the m-word (metrics). I immediately knew this wasn’t going to be pretty. ;)

Rex gave a few cases where a manager was oblivious about the need for skilled testers. One manager had supposedly said something like this: “The requirements of a tester: heartbeat, respiration.” That struck home right then. I knew these managers and product owners exist! I had been in a project where the solution to testing was “ask people around if they have the time”. I then asked around and gathered as good a bunch of people I could find… and I tried to train them to be adequate testers. Some became good testers, some didn’t, but I had a team.

Feels like the requirements for a tester in fact are the aforementioned heartbeat and respiration. In some cases, though, also a certification might be squeezed. If we find a tester with a certification, one has to be a good tester! Right? Rex mentioned in the talk (which I already knew from previous experience with the ISTQB) that people who certify themselves with the foundation level certification do not need any skills from testing before the certification. It is “an entry-level certification”, Rex says. So how come people with no skills are certified as testing professionals? This leads to a situation where unskilled, untalented testers that look better to managers than people with REAL skills and attitude towards testing. I that what it is suppose to be doing?

The certification thing got me so winded up that I almost missed the rest of the keynote while trying to douse my building rage. (Just kidding! ;) ) Rex had some good thoughts about how to veer the Agile testing more towards intelligent testing – away from “Unit testing + Acceptance testing = Proper testing” an more towards testing throughout the life cycle. Automated testing is there only to support manual testing*. Usually Agile testing is a Waterfall within the Agile cycle where testing is done at the end of the cycle (if we have the time). Testing earlier, reviewing the deliverables in time and early enough, challenging designs, etc. – not just automating unit tests.

Rex also talked about the cost of bugs, but there was no ground breaking new information about that. I have read that from the ISTQB syllabus a while ago. And I still believe that counting bugs and measuring them against each other will lead into biased view that “all bugs are equal”. James Bach said it well once that “bugs should be thought as unicorns” How many unicorns fit into a cubicle? Two?) How many bugs are in a product? I know this is generalization from Rex’s part but I think it dangerous to make such claims as (like I mentioned earlier about the decision-making skills of a manager**) people WILL make the tools out of the metrics to compare PEOPLE against each other. "I found 4 bugs and he only found 3. I'm better, right?"

Rex also talked about the quality management. He mentioned that good testing doesn’t make good quality in its own. I agree. We need to be able to start making good quality from the top, starting with directors. We also need to be able to improve the processes, tools, skills and communication to make better quality. We can do it all the time while we work by challenging bad behavior and encouraging good behavior.

Anyway, I feel that the foundation level certification is not currently building the community towards good testing. It has become a magic medallion that is bought from the wandering gipsy saleswoman. And when the kings and queens see the medallion they say “This must be a skillful knight as he has the Badge of Knighthood”. And yes, the medallion is bought, not earned. The training courses for the foundation level solely aim to give answers to the exam questions. They do not encourage people to learn, but to memorize (the worst thing is that they might say that one doesn’t need this information in practice). Rex said that when his consultancy company holds these courses they focus on practical use of the things trained. I wish I could once attend one of his company’s courses to see how it is done.*** The whole certification issue might need a blogpost of it's own, but it will have to wait. Some people have already covered that for me.****

You might get a message that I didn’t like Rex or his keynote. I did in fact, and I encourage all of you to read Rex’s blog and the books he has written. I don’t say I agree with all he writes but they give you a lot to think about. Rex as a person is (to me) "a happy giant"*****. What I liked most about him is that he really listens to what you have to say. We had inspiring conversation with Rex after the keynote although I challenged him about the ISTQB. I hope to hear him talk again at some other event. And I was bummed that he couldn't join my workshop... :(


*) And to build confidence, some say. But, like Michael Bolton said in the Rapid Software Testing course: “Were not in the confidence building industry – we’re in the false confidence demolition industry!”
**) Says a guy who spent two years as a test manager and is currently a maintenance manager deputy.
***) Do you dare to invite me to a course, Rex? ;)
****) Jari Laakso's post http://jarilaakso.blogspot.fi/2012/02/test-is-dead-vs-istqb-kills-people.html
*****) He's like 7 feet tall.

Managing Heuristic Exploratory Testing Based on MindMap

The FreeMind mind map template that I use.

The MindMap allows me to create a structured testing quickly and efficiently so that the testing is accountable and efficient. The mind map functions as requirements document, test plan, test report, bug report and heuristics cheat sheet for my testing. The mind map functions simultaneously as the tool and the product of testing thus supporting the values of agile and lean development. I’ll explain how.

Gathering Information

As the picture above states that there are multiple heuristics to choose from and when starting to test the best way to start is to gather information. The “Project” heuristic is a good way to start developing a sense of what to test, how and in what order. Using the applicable heuristics one can form the basic knowledge about the test object and the surrounding project.

This phase may be as quick or as time consuming as necessary. The key is that you’ll establish the needed knowledge about the test object. This may include (but not restricting to) referring to requirements documentation or other documents, talking to product/project/solution owners/architects/managers, requesting consulting from developer/other tester or the like; anything you see helpful to your task at hand.

At this stage you’ll start also forming you testing plan. This may include equipments and resource scouting or gathering, scheduling and possibly limiting the scope of testing. By limiting the scope into testing that can be done in 90-120 minutes sessions the testing is more manageable and less tedious. Forming the testing into specific missions also makes your testing more motivating as you’ll achieve lesser goals on your way to larger goal.

At this point you have a mind map that looks something like this:

Project heuristics during Information gathering

You should have sought answers to the questions or decided they do not need answering. This forms the basics of your test planning. At this point you’ll also have a good grasp of what is important and should be tested first. It is possible that the priorities change during testing (and it should) as does the scope. This is however the framework in which the testing starts.

The gathered information is context-dependent and varies across domains. The mind map may also be tweaked to suit the domains special needs. Also the items/tokens may be chosen to best suit the purpose either increasing or decreasing the amount. What I use are Question (“Someone needs to tell me something regarding this topic”), Exclamation (“There is something in this topic that needs to be looked into in detail”), Idea (“Possible solution, test idea or a thought”), Blocker (“This topic might cause a block to testing now or at some point”), OK (“This topic is tested and considered not needing further testing”) and NOK (“This area requires more testing at some point”). The most important thing is that the choice of items supports the cause instead becoming the cause itself.

How to measure coverage?

As we have now established the basic knowledge behind the testing task we may start to sketch the coverage and depth of the testing. This is done by modeling the test object. Modeling is a simple way to map or… hmmmm… model the test object. This is done by creating a representation of the test area using the modeling heuristics. The models may be as detailed as needed or as vague as they can be, again what bests suits the context.

I myself start the modeling by mapping the structure of the test object. In case of GUI testing I try to sketch the basic layout of the GUI forms and deconstruct the components within the GUI. For example some text fields can be tested with same inputs (possibly). This also gives a good representation of the depth of the GUI (as if there are any forms /behind/ the previous). Menus are a good guide in deconstructing the GUI as they also may hint onto what functions the system has to offer.

The structure may consist of anything “physical” or fairly easily recognizable objects of the test object. The operations of a web service are easily modeled as are the fields within the operation. There are possibly even tools (spiders) that will form the structure model for you by minimal work. This is the first coverage model that I use and frankly the most simple. The coverage may even be derived into X/Y –coverage to sate the managers’ need for figures, but this is also to remind you of what has been done and what needs to be done.

Functional coverage is the list or the model of what functions the test object has to offer. This may also be somewhat easily derived from visual/physical things but also the hidden/hard-to-come-by functions should be stated. The function may however be used in a different way that it was intended so user/admin/automatic operations should be stated also either as their own or where they differ from the intended functionality.

The data coverage is important to some and should be deconstructed into a heuristic list. Easiest way is to find out what kind of data the structure requires and what is in the interface requirements or in database. Also analysis of what kind of data is it possible to insert into system and HOW! There may be data security issues in public interfaces or GUIs that need to be tested with different approach or depth.

Platform may affect all tests and if there are any platform dependencies they should be stated. Sometimes it is not seasonable to test everything in every platform so the criticality of the testing should be estimated on each platform. Sometimes test automation and changing the test approach may come handy if testing multiple platforms (i.e. mobile phone testing on different phone models).

Time-related testing is also a way to measure coverage of the test object. If there are time-related testing to do (performance testing, stress testing, concurrency testing, “24/7 testing”, etc.) these are also a way to measure coverage. Performance coverage may be important in a SOA world where critical components require consistent and good performance.

These form into a map of models of which we can inspect coverage and progress of testing.

The coverage break down using Models heurisrics

Heuristics support testing

The use of heuristics enables you to think of multiple ways to test something. One can apply many different techniques onto ones testing to dig up different kinds of information about the product. They can be test techniques (remember that exploratory testing is an approach not a technique), quick tests, tours, you name it. The point is that the heuristics mind map functions as a checklist (if that is the way you want to use it, for example coverage reasons) or as a heuristic in itself.

Whatever the approach to testing is the heuristics are there to help you in your thought process. Using the mind map to record your thoughts about some aspect of testing is advisable and easy, as you can connect techniques to models and parts thereof. For example using quick tests (“clicking frenzy”, “banging the keyboard”, etc.) to stroll through a GUI or flow testing to do some E2E testing, the mind map is a good tool to design testing on the go and to report any findings into the map.

The mind map can be used to report bugs using the video clips generated from test sessions in addition to possible description in the mind map. These help communication to developers, managers and clients, and writing the defect reports. The issues and questions that testing uncovers can be linked to oracle heuristics to make more case to the defect.

For example a basic stress test may indeed pass within the NFR description but the memory usage or the amount of handles may not even be described in the NFR. Of course it is up to the tester to choose the best way of describing his/hers findings. The rule of thumb is that there should be enough information to help the developers to fix the problem AND to make an appealing case to fix it! (No bug report is good enough if the bug doesn’t get fixed!)

Heuristic testing skills (for example Rapid Software Testing skills) are also important here as one can rely too much on the check list and forget to do the testing in their heads (where most testing is done). Heuristics should be applied (if applicable) not followed. There’s no harm trying to apply some heuristic in a context, but using the heuristic as a hammer and trying to smack everything is not advisable (it might even be stupid). They are there to help you reach your testing goal and provide tools to your brain.

If stuck, the heuristic mind map can give you a good boost into proper direction: just look at the heuristics (if you can’t remember them) and think of what to do next and how. Heuristics work also as guides in tweaking your plan. You may have gotten false information during your info-gathering or bugs prevent you from doing something. Heuristics can provide a solution to a changed situation when a new mission needs to be formed.

The testing heuristics applied

A product and a tool of testing

The agile principle describes unnecessary documents as bad (or that is how I have interpreted the manifesto). Michael Bolton talked about using “memos” in testing to help the brain to interpret all info pushed into it. Instead of being a document (a product which you produce) the memo is used as a tool to help testing. The mind map has a similar value to testing: it helps make notes about things you see, feel, and observe, but also enables you to use these notes as a report of what you have done (if someone asks).

The mind map serves as a tool to facilitate the testing but also as a product of testing. It serves as a platform for communication when describing what went wrong and how. It serves as a report of the testing to increase accountability. It serves as a session report of which the Test manager or other testers can fathom what you did, how and what important knowledge there is that they need to consider.

A finished mind map looks something like this:

A finished mind map with references to other heuristics and links to screen caps and video clips.

The questions should be answered, if there were any. The bugs, issues, ideas, etc. should be discussed and possibly closed. New testing sessions should be organized to cover areas that were not covered.

I used FreeMind 0.9.0 in my example and in my work daily. The example mind map was done to illustrate the possibilities of using mind maps. Normally my mind maps are a bit more restricted to certain type of testing.

The web is full of mind map software from browsed based (like mindmeister) into downloadable tools (like XMind). Just choose the one best suit you (or ask someone for a recommendation) and create a mind map that suits your style of testing. And get testing!

If you have had good experiences using mind maps, please share them. :)