Intelligent practices of software testing

Intelligent practices of software testing

(This was first published at dev.solita.fi)

Note! I am not using the term “best practice” here. Intelligent practice is something that helps people develop their practices into more suitable for their need. We actually should stop using “best practice” term. Instead we should use whatever helps us solve the problems at hand. The problem solving should come from people and their skills. Influences should be taken from everywhere! Practices should be molded to facilitate problem solving in YOUR context.

Very well. I came up with the following guidelines to help create a better testing environment for me and my colleagues. I am not trying to invalidate your practices that have been found to be best fitting for you. You should keep the practices you deem appropriate for your context. I encourage to revise your practices based on my own findings. It is up to you to change what ever you see reasonable, if anything.

Fight ambiguity with openness and communication

A few examples on ambiguity:

• The (written) requirements are vague and ambiguous at best. Usually they are out of context and/or vacuum packed. I remember ISTQB mentioning that “It is testers’ right to deny ambiguous requirements”. I believe it is the testers’ obligation is to address this as soon as it is found. And by addressing I mean talk with people. “I cannot figure out how the system should work. I found out that it works like this. Is this OK to you? If not, can you help me determine how it should behave?”
• The test cases are not concise. If we are to perform good enough testing we need to do testing. Writing test cases might be testing – executing test steps from a test case might be testing. This all depends on the skills of the tester. If the material we base our testing is poor, we need to be open about it. If we feel that we cannot do good enough testing, something needs to change. That change comes from talking about the issues, being open about the difficulties. 

If something is not clear, make it clearer. Find the information, people, resources that helps you make it clear. Learn from difficulties and be open when you face them. Hiding problems rarely leads to anything positive.

Fight ignorance with eagerness

Ignorance comes in many forms. One can claim she doesn’t know enough about the product to test it. Ignorance might paralyze us. Tasks can be daunting and we might procrastinate because we do not know some specific thing about the subject.

Enter eagerness! Try Proof of Concept type of actions. Try sandboxing the test area. Try having fun. Try getting a group of people who might already know about the domain, the product or the like, and test with them. Try walkthroughs. Try anything and everything to solve the problem. Be eager to to solve issues. If you can't solve them, be eager to get issues solved. Ignoring issues rarely leads to anything positive.

Fight über control with good-enough documentation and reliance to skills

Sometimes the aforementioned (ambiguity and ignorance) might lead into not trusting our ability to do proper testing. That might lead into a form of control that requires a vast amount of documentation and reports. There is a three-fold solution to this, I believe: rightly timed planning, unburdening reporting and reliance to people ability to the best possible job. When it comes to testing, people usually ask for test plans, test task descriptions (of various levels of detail) and test reports.

When doing good testing, it requires some planning to remove inefficiencies. That planning should be done in advance, albeit on a higher level. We must be able to plan our testing while we test, because we learn about the product, the project and everything surrounding us while we test. By timing our planning in sync with our testing, we can react to changes and discoveries more efficiently. The planning should happen as close as possible to the actual execution of the plan. Documents and plans deprecate quickly when new things are added. For example, Rapid Software Testing encourages planning and designing tests during testing. I have found out that a brief planning session before testing session is usually in order to be able to tackle the most important topic during that testing session.

We are chosen to perform the tasks because people expect we can do the task. There is little reason to stupefy the intelligent people that do things for us. Writing test cases, that basically make our brains redundant, is stupefying. In fact, it is stupid to write something that creates stupidity. “Stupid is as stupid does.” In order to be intelligent and harness out intelligent to our testing, it is wiser to write inspiring testing documentation. Missions, threads, ideas, broad descriptions, etc. encourage the use of our brains. I’d say “skill before process” and encourage to constant learning and teaching. Autonomy increases motivation see this post to do the best possible job. Critical thinking enables you to challenge your biases. Lateral thinking helps uncover possible issues. Thinking is the key, not the process or tools.

We also need good reporting off of our testing. Reports may be bug reports, testing transcripts, etc. An important thing to remember here is to focus on relevance and sufficiency of our reporting. The report should be a tool to improve and to facilitate discussion within our testing, development and/or project management. Thus I feel that reporting should be done in ways that support our cause, not because we want reports to begin with. Tools may vary from video recordings and screen captures to notebook scribbles. Anything is fair game as long as it supports the cause.

Fight measuring and quantification with reliance on feelings, observations and communication

Numbers are cryptic. Numbers can be interpreted in as many ways as there are people interpreting. Let’s say we have 100 test cases from which we have run 90. A clueless test manager might say: ”Only 10 more and we’re done”. An intelligent tester might say: “It seems we have 10 scripts to look at, but what else do we need to take into account?” Quantifying test cases (or bugs for that matter) is ridiculous. It is like counting unicorns! How many unicorns can fit into a cubicle?

A better approach to test case counting might be counting the time spent testing. Time is uniform in magnitude. Hour is as long in Finland as it is in New Delhi. What is the size of a test case? Few can tell. Measure what is worth measuring and what is objective.


If this is measuring our progress, how do we measure coverage?

We might want to ask “why we measure coverage” and then choose the metrics based on that. One good coverage indicator is a list of items in the product under test that we have touched. If an area is touched multiple times, we could be fairly certain that it is covered well enough, and we know the areas that are untouched. The human feelings come to play here. If a tester or a developer feels that some area is of poor quality, then we should play that feeling and test that area some more. A test case doesn’t tell how confident the tester is - talking about findings with the stakeholders might demolish the false confidence and lay ground for confidence based on actual findings and behavior.

Whatever you choose as your method of measuring, talk about it. The numbers are devious and one should ALWAYS talk about the coverage and confidence instead of using numbers to /prove/ why something is good. I support creating a measurement strategy that is little or no burden to the actual tester, and then use those findings to steer testing to the right path instead of proving something. Be critical about numbers because they can fool us easily. Be intelligent and critical on what and how to measure.

Summary

To summarize, my intelligent practices for software testing might include the following:

• Fight ambiguity with openness
• Fight ignorance with eagerness
• Fight über control with good-enough documentation and reliance to skills
• Fight measuring and quantification with reliance on feelings, observations

There might be more and I shall keep looking. The most important thing about these practices (or statements or whatever one might call them) is to communicate with people. Discuss your expectations and goals in testing. Discuss procedures and skills, how they can be improved. Whatever challenges you face, communication is the first step to solving the problem. An umbrella practice might go something like this:

• Fight <choose challenges> with <choose solutions> and communication

If you feel I need to clarify some of the practices, give more examples or add a fundamental practice to my list, please drop me a line. I’m more than eager to discuss. ;)

Comparing love to eight - A response to a blogpost

This is a response to a writing by Natalya Bosatskaya at Intexsoft.com’s blog. The blog post was named Scripted testing vs. Exploratory testing. This post tackles every paragraph individually and tries to raise some thoughts to people thinking the same way as Natalya in this post. She did a great job by trying to explain her points of view, but I felt some of them were off mark. I felt the need to write a longer response to her as the limited space of the comment field did not allow me to get my thoughts out correctly.

In this post, I will attempt to compare Scripted and Exploratory styles of software testing. On the third hand, scripted testing is seemed as a strict and serious process and exploratory one is seemed free and easy. But each test style has own swings and roundabouts. Let's look at them from different points and try defining appropriate conditions to use first one and second one.

It is a common mistake to think exploratory testing as “free or easy”. Exploratory testing (when done correctly) is a cycle of learning, testing and test designing. It is structured but can be as free as it needs to be. It doesn’t limit thinking but provokes it. Exploratory testing focuses on the cognitive process of testing, the lateral thinking and critical analysis of the product in testing. By referring as free, you might mean that it is guided by other things than documented test plans and scripts. Exploratory testing is guided by documents, checklists, feelings, heuristics, questioning, data, functions, missions – whichever is the most suitable way to steer testing in a context and in the situation at hand. Usually the exploratory testing focuses to test the most interesting/important/risk prone area of the product at any given time.

Scripted testing usually supposes two roles: test-designer and tester. Test-designer creates test scenarios beforehand: and then tester executes these test scenarios. Test-designer is a high skilled specialist and tester is a beginner at software testing.

It is true that scripted testing requires two roles. Why is that? Why not have the skills and the insight about the product in one person? That would make it cheaper, right? A beginner tester should learn about the product as much as possible so that he/she would not be a beginner. By learning about testing, about the context, about the product, about techniques, about tools, the tester gathers more knowledge that trying to reproduce scenarios written beforehand.

What happens when the scenario changes? Do we need the test designer to re-write the case? What if there is no time to run the test case that the designer wrote? Is that considered value?

Exploratory testing doesn't divide activities by roles and time. Tester thinks up test cases himself, executes them at once and according to the test results, he creates new test cases. It's obviously, that such tester must be an experienced specialist.

It is true that exploratory testing doesn’t have fixed roles. Exploratory testing does have roles. A person may possess certain amount of business knowledge and in a testing session he might take a role of transferring the knowledge to the other testers during the testing. He may have another role in a different session. There is no waste of time or effort by making written test cases beforehand, although they might use existing test cases as guide and information to learn more about the product AND the about the testing done using those scripts.

Exploratory testing is an approach, not a technique that you use to test a product. It is a mindset to explore the product using the skills that one already has and developing more skills during the testing by learning about the product thus being able to more accurately design future tests. A person that has no experience in exploratory testing needs to start doing testing the exploratory way in order to become an expert. No-one is expert in every context but they may form skills to be good enough in a context.

A test designer taken from the bushes is just as clueless as the beginning tester. But do you know how the test designer starts to design the tests? He explores the product, reads the documentation, questions the stakeholders, and looks for clues. He does exploratory testing but wastes the effort on writing the things down so that a “beginner” can then interpret what he has learned and try to replicate the thought behind the test case.

In case of scripted testing, your test team can consist of one high skilled test-designer (costly employee) and several beginners (low-paid workers). You will economize resources, new testers (even low skilled) can start working rapidly enough. Exploratory testing depends on qualification a lot. If tester has low qualification, he will use ad hoc testing in the guise of exploratory one.

Now you are saying that testing as an activity is a low paid job. Sounds like an artificial pay gap to me. A software designer should be a highly paid person who understands the product and the programmers should be low-paid workers. I'd like to think software testers as skilled group of engineers that in collaboration with the stakeholders (developers, project managers, customer) aim to deliver a quality product using techniques like questioning, testing, challenging, coaching, etc. I think the testers are as important part of a software project as a developer, although you might have a tester integrated into a team and have other roles also (but that's a whole other matter).

I do agree that a tester without a mindset and the structure might veer towards ad hoc. A skilled manager however understands the possibility of a new unskilled tester and guides the person towards structured, mission based exploratory testing. A person that has no knowledge about the product should be given the opportunity to learn about the product and to give ideas about how to test the product better.

You might say that a person without the skills to do exploratory testing is an exploratory tester. Given a chance to explore the product one might become more familiar with techniques, skills and tools required in testing the product efficiently. A person without testing skills forced to perform a task that is scripted eliminates the chance to learn and doesn’t give room to become more efficient.

Scripted testing gives you facilities of high planning and predictability. If we have ready test scenarios, we can estimate efforts on tests executing precisely enough. In case of exploratory style, planning is very difficult or impossible at all. There's no guaranty, too, that tester will execute all needed tests and not forget anything.

It is true that planning gives you predictability. That is why there are so many different ways to manage testing. Take Session-Based Test Management (SBTM); it is a documented way to plan testing in order to respond to the situation at hand AND take into account the future testing activities also. It is rarely so that things go as planned. Scripted testing is rarely able to respond to changes in the software. Let’s say a bug is found. We might need to take a little more time to execute some confirming testing around the problem, even do a another test or three. When the bug is fixed, it may require some additional testing. Might even happen that the fix uncovers other bugs. A set of carefully planned activities start falling like a dominos.

What happens when a test designer forgets something? What happens if the requirements analyst forgets something? What happens if the programmers forget something? Is the scripted testing able to respond to this situation? What happens if we run out of time and we have tests to run? Money that has gone into developing those tests that are not run goes to waste thus making scripted testing even more expensive (and you don't get to learn). Exploratory testing has no such waste.

Exploratory testing management techniques can tackle these things with predefined processes. When something is considered more important than something else, exploratory testing can change focus on the fly and focus on the newly important part of the product.

Scripted tester can give qualitative documentation. At least he can give test scenarios and reports about execution of this scenarios. Exploratory testers often have complications with reports. How can tester report what he has tested and how many tests still remain? How can manager show to customer what has been tested?

Test cases (just like bugs) are not objects of same size. It’s like counting 3 cars, 2 buses and a train – 6 things that move people. How many people are moved? How many things (that move) fit into a car park? Test cases are counted but how much is tested? One way to measure testing is the time spent on testing an area. An hour is an hour in every place in the world (OK, there may be some exceptions) so only qualitative measure should be time. An exploratory testing session takes 90 minutes so 2 persons doing testing for 2 sessions count as 6 hours of testing. How many test cases fits into 6 hours? 3? 300? You can’t tell, but I can tell that in 6 hours I can test an area (or areas) for six hours. Simple, isn't it?

An exploratory session might have a precise mission but allows distracting from the mission if it seems appropriate. Using the SBTM to guide testing you will always get a report of what was tested, that ideas came up during the testing, what bugs were found, what issues arose. A mission might be a user scenario or several of those. A mission might be to learn about “what does the menu bar do” by exploring the product, documentation and eliciting information from stakeholders. Documentation may be as simple as verbal description or as detailed as a screen-cap recording with Rapid reporter log files.

Using sessions as guide to what needs to be covered, we can adjust the plan as we go so that we add more sessions on a given area. At any moment we know what the current situation is with the information we currently hold. A manager should always interview the testers so that he/she can form a testing story which can be supported with gathered metrics (hours/functions, sessions/area, etc.).

So according to reasoning above, scripted testing comes to the fore. Let's find a few advantages of exploratory style.

In this part I disagree strongly. Let’s look at what kind of knowledge about exploratory testing puts scripted testing as the superior approach. However as my examples before were “to the point” and aimed to address the text only, some things about exploratory testing do take somewhat scripted approach while retaining the essence of exploration – learning, testing, and designing simultaneously though stressing different aspects at different times.

Exploratory testing is flexible and adaptive to changes unlike scripted one. If something changes in the application, test-designer must alter test scenarios, otherwise tester can't execute tests. Exploratory tester has test suits in own mind, so he can change them as fast as needed.

Even if we think the most scripted exploratory testing, it still has the advantage of exploring. Exploratory testers don’t need to keep everything memorized as we can use memos, checklists, heuristics, and post-it notes. The documentation fits the context and the need of the current testing situation. All unnecessary documentation should be shunned. Use the documentation as a tool for testing instead of the purpose or the product of testing.

Basic work of scripted tester is boring and monotonous. Exploratory tester has interesting and creative tasks. He does different kind of work, he is designer and executor, so his work is not routine.

Exploratory testing can be monotonous, but the an intelligent exploratory tester finds clues in the monotonous parts of testing to learn more, improve testing techniques and to design better test ideas. If a scripted test becomes monotonous, that could be a hint that it should be automated to give more time to actual testing. Exploratory testing utilizes lots of tools, automation scripts, basically everything that supports the testing and makes it faster. “Throw everything and the kitchen sink” analogy is good to describe the thought behind utilizing resources in exploratory testing.

And exploratory testing can become a routine task, but a good exploratory tester changes focus, de-focuses, makes assumptions and tries to see behind inference, look closer and look at the bigger picture. Exploratory tester turns the routine into his advantage!

In case of exploratory style, tests are created and executed almost simultaneously, so it lets starting tests execution and uncovering bugs earlier.

True. Why shouldn’t scripted testing start earlier? What’s stopping you? Is it the test design part that’s taking so long a time? You have the designer doing (what seems to me) exploration so that he can design the tests. By not putting all the time into writing the testing can begin earlier and can achieve much more learning in doing the testing. And why force the “beginner testers” to wait? Let them dig in also and learn as they go forward. The test designer can still make documentation as in missions for the testers, but retain the exploratory approach and put his skills into good use.

In the end, I will cast one more disadvantage of scripted testing. It's the pesticide paradox. If the same tests are repeated over and over again, eventually they will no longer find any new bugs.

That is why a test getting repeated several times should be considered to be automated and the effort can be put to exploration of the parts that the automation doesn’t cover. Automated tests can check the product and the parts that already work and leave the testing to the testers.

Repeating tests is like repeating the same question all over again. Does that bring value to the tesing (sic)? Does that bring value to the tesing (sic)? Does that bring value to the tesing (sic)? Does that bring value to the tesing (sic)? There might even be a bug in the test itself thus leaving some important areas untested or poorly tested. Automated testing requires also analyzing and exploring (they’re code after all).

In my opinion, it's not correct to consider that one test style is better than another. Each style can be suitable or not in dependance of goals, test process organization, customer requirements and other reasons. And what do you think?

You are correct again. One approach does not supersede others. They are all context dependant. However I see that in every context there is a need to improve. So why not try learning while testing? By forcing people to do testing and NOT learn is a style that is should not be considered suitable in any context.

We may do harm to the community by comparing two different concepts to each other. Like comparing a dog to a blue, or a love to eight (which by the way in a certain context is possible -> love=zero). I do however see that there are too many misconceptions about exploratory testing and that false knowledge must be corrected. Exploratory testing is just like driving a car: we respond to stimuli to adjust our thinking, bearing, or speed. We learn while we go into situations from the behavior of others, the car, and the environment and will determine the right approach as we get the grasp of what is going on. We do not stop the car at every situation and check the car driving manual to give us instructions how to handle the situation. Why do we think that testing requires us to rely on scripts that (in best case) are interpretation of someone else in some different time and context (or entirely obsolete)?

Best Practices (NOT!)

I've been a lazy boy with updating the English version of my blog. I'll whip myself into blogging frenzy and update the blog as often as possible. My first addition to my posting is my most frequent blogging on my Finnish blog.

For some reason I stumbled upon the JMB blog about maturity models and the critisism thereof. Within the posting there was a link about No Best Practices.

Our company has recently began its Quality Assurance project which is designed to raise the quality of all aspects of the company, from management and sales to code development and support. The thought struck me that "hey! We could improve the quality of testing as we increase the overall quality" and thus I started to think improvements through the company as a whole. The testing organization within the company should be revised and to make it more coherent and more effective. After a little brainstorming I ended up refining testing process, clarifying roles, increasing the testing skills and increasing the visibility of testing as a craft within the company.

Thus became the idea about testing architect and testing steering group. (Yeah, we aint that far in testing yet...) We already have a plethora of steering group and their main goal is for

sharing best practices, tools, frameworks and code.

Yay! Fun fun! Best practices! When all developers go by the best practices things can't go wrong, can they? Can they? Will they?

I myself strive towards continuous improvement in testing. Everything should be better than one done before. Besides I'm a bit of a context-driven test enthusiast and I like to hear what ye olde JMB has to say about stuff. (No offence, Michael Bolton!) Every single pratice, method and technique is attached into relevant context and time. With both of these being variable is it possible to say that a practice is best? Is there a practice that is better that the Best? If so can it really be said that a pracitice is best?

Could it be so that men (and women also) tend to veer towards practices that are already developed in some context and in their laziness or schedule pressure take these practices as graneted? Some people may use a certain method in his or her work and ignore more effective, more reasonable models and practices. Some people base their work on ".Net Best Practises" and nothing else and make that kinda code all their life.

Then there are people who innovate; break habits, patterns, prejudice; and challenge authorities! These "better practicists" and the force that drive every industry forward: they don't linger inte the artificial Best Practices but push through and take their work into another level. The others may imitate them and claim their practices as BEST. Just then the innovator carries on his or her work and strives for higher and higher.

Which one you are: Best practice person of Better practisist?