This is a response to a writing by Natalya Bosatskaya at Intexsoft.com’s blog. The blog post was named Scripted testing vs. Exploratory testing. This post tackles every paragraph individually and tries to raise some thoughts to people thinking the same way as Natalya in this post. She did a great job by trying to explain her points of view, but I felt some of them were off mark. I felt the need to write a longer response to her as the limited space of the comment field did not allow me to get my thoughts out correctly.
In this post, I will attempt to compare Scripted and Exploratory styles of software testing. On the third hand, scripted testing is seemed as a strict and serious process and exploratory one is seemed free and easy. But each test style has own swings and roundabouts. Let's look at them from different points and try defining appropriate conditions to use first one and second one.
It is a common mistake to think exploratory testing as “free or easy”. Exploratory testing (when done correctly) is a cycle of learning, testing and test designing. It is structured but can be as free as it needs to be. It doesn’t limit thinking but provokes it. Exploratory testing focuses on the cognitive process of testing, the lateral thinking and critical analysis of the product in testing. By referring as free, you might mean that it is guided by other things than documented test plans and scripts. Exploratory testing is guided by documents, checklists, feelings, heuristics, questioning, data, functions, missions – whichever is the most suitable way to steer testing in a context and in the situation at hand. Usually the exploratory testing focuses to test the most interesting/important/risk prone area of the product at any given time.
Scripted testing usually supposes two roles: test-designer and tester. Test-designer creates test scenarios beforehand: and then tester executes these test scenarios. Test-designer is a high skilled specialist and tester is a beginner at software testing.
It is true that scripted testing requires two roles. Why is that? Why not have the skills and the insight about the product in one person? That would make it cheaper, right? A beginner tester should learn about the product as much as possible so that he/she would not be a beginner. By learning about testing, about the context, about the product, about techniques, about tools, the tester gathers more knowledge that trying to reproduce scenarios written beforehand.
What happens when the scenario changes? Do we need the test designer to re-write the case? What if there is no time to run the test case that the designer wrote? Is that considered value?
Exploratory testing doesn't divide activities by roles and time. Tester thinks up test cases himself, executes them at once and according to the test results, he creates new test cases. It's obviously, that such tester must be an experienced specialist.
It is true that exploratory testing doesn’t have fixed roles. Exploratory testing does have roles. A person may possess certain amount of business knowledge and in a testing session he might take a role of transferring the knowledge to the other testers during the testing. He may have another role in a different session. There is no waste of time or effort by making written test cases beforehand, although they might use existing test cases as guide and information to learn more about the product AND the about the testing done using those scripts.
Exploratory testing is an approach, not a technique that you use to test a product. It is a mindset to explore the product using the skills that one already has and developing more skills during the testing by learning about the product thus being able to more accurately design future tests. A person that has no experience in exploratory testing needs to start doing testing the exploratory way in order to become an expert. No-one is expert in every context but they may form skills to be good enough in a context.
A test designer taken from the bushes is just as clueless as the beginning tester. But do you know how the test designer starts to design the tests? He explores the product, reads the documentation, questions the stakeholders, and looks for clues. He does exploratory testing but wastes the effort on writing the things down so that a “beginner” can then interpret what he has learned and try to replicate the thought behind the test case.
In case of scripted testing, your test team can consist of one high skilled test-designer (costly employee) and several beginners (low-paid workers). You will economize resources, new testers (even low skilled) can start working rapidly enough. Exploratory testing depends on qualification a lot. If tester has low qualification, he will use ad hoc testing in the guise of exploratory one.
Now you are saying that testing as an activity is a low paid job. Sounds like an artificial pay gap to me. A software designer should be a highly paid person who understands the product and the programmers should be low-paid workers. I'd like to think software testers as skilled group of engineers that in collaboration with the stakeholders (developers, project managers, customer) aim to deliver a quality product using techniques like questioning, testing, challenging, coaching, etc. I think the testers are as important part of a software project as a developer, although you might have a tester integrated into a team and have other roles also (but that's a whole other matter).
I do agree that a tester without a mindset and the structure might veer towards ad hoc. A skilled manager however understands the possibility of a new unskilled tester and guides the person towards structured, mission based exploratory testing. A person that has no knowledge about the product should be given the opportunity to learn about the product and to give ideas about how to test the product better.
You might say that a person without the skills to do exploratory testing is an exploratory tester. Given a chance to explore the product one might become more familiar with techniques, skills and tools required in testing the product efficiently. A person without testing skills forced to perform a task that is scripted eliminates the chance to learn and doesn’t give room to become more efficient.
Scripted testing gives you facilities of high planning and predictability. If we have ready test scenarios, we can estimate efforts on tests executing precisely enough. In case of exploratory style, planning is very difficult or impossible at all. There's no guaranty, too, that tester will execute all needed tests and not forget anything.
It is true that planning gives you predictability. That is why there are so many different ways to manage testing. Take Session-Based Test Management (SBTM); it is a documented way to plan testing in order to respond to the situation at hand AND take into account the future testing activities also. It is rarely so that things go as planned. Scripted testing is rarely able to respond to changes in the software. Let’s say a bug is found. We might need to take a little more time to execute some confirming testing around the problem, even do a another test or three. When the bug is fixed, it may require some additional testing. Might even happen that the fix uncovers other bugs. A set of carefully planned activities start falling like a dominos.
What happens when a test designer forgets something? What happens if the requirements analyst forgets something? What happens if the programmers forget something? Is the scripted testing able to respond to this situation? What happens if we run out of time and we have tests to run? Money that has gone into developing those tests that are not run goes to waste thus making scripted testing even more expensive (and you don't get to learn). Exploratory testing has no such waste.
Exploratory testing management techniques can tackle these things with predefined processes. When something is considered more important than something else, exploratory testing can change focus on the fly and focus on the newly important part of the product.
Scripted tester can give qualitative documentation. At least he can give test scenarios and reports about execution of this scenarios. Exploratory testers often have complications with reports. How can tester report what he has tested and how many tests still remain? How can manager show to customer what has been tested?
Test cases (just like bugs) are not objects of same size. It’s like counting 3 cars, 2 buses and a train – 6 things that move people. How many people are moved? How many things (that move) fit into a car park? Test cases are counted but how much is tested? One way to measure testing is the time spent on testing an area. An hour is an hour in every place in the world (OK, there may be some exceptions) so only qualitative measure should be time. An exploratory testing session takes 90 minutes so 2 persons doing testing for 2 sessions count as 6 hours of testing. How many test cases fits into 6 hours? 3? 300? You can’t tell, but I can tell that in 6 hours I can test an area (or areas) for six hours. Simple, isn't it?
An exploratory session might have a precise mission but allows distracting from the mission if it seems appropriate. Using the SBTM to guide testing you will always get a report of what was tested, that ideas came up during the testing, what bugs were found, what issues arose. A mission might be a user scenario or several of those. A mission might be to learn about “what does the menu bar do” by exploring the product, documentation and eliciting information from stakeholders. Documentation may be as simple as verbal description or as detailed as a screen-cap recording with Rapid reporter log files.
Using sessions as guide to what needs to be covered, we can adjust the plan as we go so that we add more sessions on a given area. At any moment we know what the current situation is with the information we currently hold. A manager should always interview the testers so that he/she can form a testing story which can be supported with gathered metrics (hours/functions, sessions/area, etc.).
So according to reasoning above, scripted testing comes to the fore. Let's find a few advantages of exploratory style.
In this part I disagree strongly. Let’s look at what kind of knowledge about exploratory testing puts scripted testing as the superior approach. However as my examples before were “to the point” and aimed to address the text only, some things about exploratory testing do take somewhat scripted approach while retaining the essence of exploration – learning, testing, and designing simultaneously though stressing different aspects at different times.
Exploratory testing is flexible and adaptive to changes unlike scripted one. If something changes in the application, test-designer must alter test scenarios, otherwise tester can't execute tests. Exploratory tester has test suits in own mind, so he can change them as fast as needed.
Even if we think the most scripted exploratory testing, it still has the advantage of exploring. Exploratory testers don’t need to keep everything memorized as we can use memos, checklists, heuristics, and post-it notes. The documentation fits the context and the need of the current testing situation. All unnecessary documentation should be shunned. Use the documentation as a tool for testing instead of the purpose or the product of testing.
Basic work of scripted tester is boring and monotonous. Exploratory tester has interesting and creative tasks. He does different kind of work, he is designer and executor, so his work is not routine.
Exploratory testing can be monotonous, but the an intelligent exploratory tester finds clues in the monotonous parts of testing to learn more, improve testing techniques and to design better test ideas. If a scripted test becomes monotonous, that could be a hint that it should be automated to give more time to actual testing. Exploratory testing utilizes lots of tools, automation scripts, basically everything that supports the testing and makes it faster. “Throw everything and the kitchen sink” analogy is good to describe the thought behind utilizing resources in exploratory testing.
And exploratory testing can become a routine task, but a good exploratory tester changes focus, de-focuses, makes assumptions and tries to see behind inference, look closer and look at the bigger picture. Exploratory tester turns the routine into his advantage!
In case of exploratory style, tests are created and executed almost simultaneously, so it lets starting tests execution and uncovering bugs earlier.
True. Why shouldn’t scripted testing start earlier? What’s stopping you? Is it the test design part that’s taking so long a time? You have the designer doing (what seems to me) exploration so that he can design the tests. By not putting all the time into writing the testing can begin earlier and can achieve much more learning in doing the testing. And why force the “beginner testers” to wait? Let them dig in also and learn as they go forward. The test designer can still make documentation as in missions for the testers, but retain the exploratory approach and put his skills into good use.
In the end, I will cast one more disadvantage of scripted testing. It's the pesticide paradox. If the same tests are repeated over and over again, eventually they will no longer find any new bugs.
That is why a test getting repeated several times should be considered to be automated and the effort can be put to exploration of the parts that the automation doesn’t cover. Automated tests can check the product and the parts that already work and leave the testing to the testers.
Repeating tests is like repeating the same question all over again. Does that bring value to the tesing (sic)? Does that bring value to the tesing (sic)? Does that bring value to the tesing (sic)? Does that bring value to the tesing (sic)? There might even be a bug in the test itself thus leaving some important areas untested or poorly tested. Automated testing requires also analyzing and exploring (they’re code after all).
In my opinion, it's not correct to consider that one test style is better than another. Each style can be suitable or not in dependance of goals, test process organization, customer requirements and other reasons. And what do you think?
You are correct again. One approach does not supersede others. They are all context dependant. However I see that in every context there is a need to improve. So why not try learning while testing? By forcing people to do testing and NOT learn is a style that is should not be considered suitable in any context.
We may do harm to the community by comparing two different concepts to each other. Like comparing a dog to a blue, or a love to eight (which by the way in a certain context is possible -> love=zero). I do however see that there are too many misconceptions about exploratory testing and that false knowledge must be corrected. Exploratory testing is just like driving a car: we respond to stimuli to adjust our thinking, bearing, or speed. We learn while we go into situations from the behavior of others, the car, and the environment and will determine the right approach as we get the grasp of what is going on. We do not stop the car at every situation and check the car driving manual to give us instructions how to handle the situation. Why do we think that testing requires us to rely on scripts that (in best case) are interpretation of someone else in some different time and context (or entirely obsolete)?
