Test Strategy in 10 minutes

This post covers both the workshop I did on test strategy at Solita as a part of the Testing Tuesday and the extempore workshop I arranged at the Nordic Testing Days in 2015. While both had the same agenda, they were vastly different. (Why this post is published now is because it had converted into a draft which I noticed only now.)

The one at Solita

The workshop was not supposed to be a slide show, like none of the workshops arranged at Testing Tuesday. Once again I drew stuff on the white board and let rip. I started by describing testing strategy quickly. Then we discussed about project elements, product elements and quality aspects. These were loosely based on the Rapid Software Testing models. The audience gave ideas on what kind of elements we need to take into account when choosing a test strategy. There is no comprehensive list on what we came up with, but we had a whole lot of ideas ranging from schedules to "pissed-offness" and expected user behavior.

After we covered most of the areas, I split the audience into three groups. They all were supposed to create a testing strategy in ten minutes. The product I chose was a web shop that sells knick-knacks and mails them to people. After ten minutes I had three completely different strategies.

The first one was focused solely on money. They mapped out every aspect that could hinder the income of revenue and prioritized them according to importance to the "owner" of the store.

The second one was a "software engineer approach". That group mapped out all the aspects that were important to different kind of engineers. There was an aspect of security testing, transaction testing, happy day testing of known processes and some complementary approaches. There was no prioritization but there was definite focus on tools and know practices.

The third one was focused on business processes. They mapped out as many user stories as they could and dissected them to steps. They then tried to figure out what kind of a prioritization would make sense to test the processes.

In 10 minutes we had three outstanding drafts of testing strategies. All of them had different approaches and they the strategy supported the focus they had chosen. They were not comprehensive, nor should they have been, but they were "good enough" to start testing the most important thing as soon as possible. We discussed the fact that when combined these three could actually complement each other. In a coffee break, we can create a draft of a strategy and even introduce that draft to a potential customer to explain our strategy to test their product.

All in all, the workshop was a success, since it spawned a new way to think testing strategy. I never thought I could choose a simple focus like "money" as the guideline of my testing. It does reveal the importance of talking to stakeholders to understand their values and needs. Without any templates or predetermined practice every team could conjure an awesome strategy which seemed even executable after they explained what they thought.

The one at Tallinn

Feeling confident on the outcomes of the workshop, I agreed with Helena Jeret-Mäe that I would do an extempore workshop during a coffee break of the Nordic Testing Days (see here). The workshop was supposed to be on the last day. I dragged a white board to the hallway near sofas and gathered people to do the exercise. I was able to gather a couple great minds of software testing on the sofa including Erik Brickarp and Santosh Tuppad. The workshop started by me explaining the idea and then I gave the same task to them: "Generate a test strategy in 10 minutes."

The problem was that the pro testers weren't too happy with the vagueness of the context. They wanted to know more about the product, more about the stakeholders, more about the project. In the end we didn't actually generate a context map for the product (plus some testing strategy elements).

The best outcome for me was (once again) to observe experts in their work and how the dynamics within a group actually work. Also the lack of structure was something they pointed out. Erik mentioned that we could have described four key aspects of the product and then figure out how to test those. Where the first three groups I had at my office all chose a focus, we had none. We could have spent a few minutes in the beginning to define where we would like to focus (even an arbitrary focus) and then create a strategy based on that. There were hints of focus surfacing when they interviewed me about the product, but none that was chosen as the key thread.

What next?

Having had two vastly different workshops on the same subject, I think it makes sense to arrange these even more. The original idea to this came from Fiona Charles at EuroSTAR 2013, but I think we approached it quite differently. I am planning to do these extempore workshops at every conference I attend, since it makes sense to give people a chance to try their skills at this. Every time I get a huge amount of pointers, ideas and lessons learned. I would think that it is the conversation that ensues rather than actually creating a strategy that counts.

The key lessons learned in both sessions were:
- The dynamics in the group determine a lot what kind of strategy is created
- Know your stakeholders!
- Choose a focus as a skeleton and then fatten it up
- Don't over-do it! Ten minutes might be enough to start testing the most important thing.
- Have some structure, but keep ideas flowing

When you see me at a conference, come ask when am I gonna pop out the white board. It might be the next coffee break. ;)

- Peksi

Mushroom-picking heuristic

Imagine this: You are in a forest and you are trying to find juicy mushrooms. You'd like to find some porcini, black chanterelle, maybe some yellow bearded milk-cap. Edible mushrooms nonetheless. Before you go, do you need something? Perhaps the following:

• the calendar which shows different mushroom appearance into your local forest (not too close to habitat because mushrooms ingest heavy metals)
• maybe some research on what the mushrooms look like that you are trying to find (also pictures thereof so you don't pick something poisonous that looks vaguely like edible mushrooms)
• some knowledge on how to pick mushrooms (pick them up intact and one piece either by twisting or pulling)
• maybe choose the weather when you go mushroom-picking (the mushrooms should be rather dry when being picked up)
• prepare the mushrooms as soon possible (remove sand, pine needles, moss, etc)
• use proper tools (a knife, a brush, a basket instead of a plastic bag)

These are things that you might want to consider when going out for mushrooms. Now, these are all preparatory things, bits of knowledge you might want to have before actually going to the forest. Obviously you can go without knowing these things, but you may end up with no mushrooms or worst case poisonous bastards like amanita or cortinars. The trip to the woods might have been productive nonetheless - you got fresh air and some exercise, maybe you had a good chat with a fellow mushroom-picker. Not all unprepared trips are totally worthless, but you need some preparations to achieve good results.

How does this translate to testing? You prepare yourself to the testing task by doing almost same things you do when you go mushroom-picking. Like so:

• You check the schedule for the most convenient time to do specific kind of testing. (In January you might want to ice-skating instead of mushroom-picking, which might still be fun.) If you're doing usability testing, you might want to choose a time when there is something someone can actually use. When doing penetration testing you might want to pick a time when there is something to penetrate. Cuz if you choose the timing badly you might not achieve the best results. Also checking the schedule gives you clues on how to time-box the testing.
• You might want to research the subject of testing. What should you be looking for, what things you expect to discover, what are the risks that are already known. Perhaps you want to learn more by exploring the product, by trying things, playing and clicking around, banging the keyboard with a shoe. You might have pictures of the GUI or architecture schematics, maybe a person to help you go about the product. Like mushroom-picking, you can stumble on interesting things, but to recognize the important, the critical, the alarming stuff you might need to do some research.
• You might need knowledge on how to do software testing. By clicking around without a purpose might not be good testing. A good tester has a skillset that she utilizes to perform good testing. It is also important to know the domain, how to perform testing in that particular product/domain/service.
• Choose the weather when you go testing... Urhm... Testability and configuration, choose the best starting conditions, datasets, timeslots, loads, etc. to achieve the best testing performance and to find interesting things. You might want to do testing when the backend is performing poorly or the network connections are bad. Maybe choose a dataset that is production like or maybe it should be a fuzzing test to generate weird data to the APIs.
• Prepare your mushrooms. Deliverables! You should take notes on your testing performance. This is to help you steer your testing, generate ideas that couldn't yet be executed, dot down risks and bugs you find. You can then explain to other people what you did, why you chose to do specific tests and checks, what did you find. If needed, prepare a useful report on the testing you did.
• Use proper tools. Testing is about using tools, obviously. Most important tool is your brain! Use it. Also use tools to help do things that are difficult or time-consuming, keep your concentration while testing. Use scripts when they are useful, record your screen, have quick note taking equipment. Use other people! Two brains are sometimes better than just one... A person can be a tool!

(This is not comprehensive list in any case. You might want to take other context variables into account to achieve the best result when you actually go testing.)

I'm now in the woods with my wellies on, my trusty 'shroom-basket, a J. Marttiini mushroom knife, and a backpack willed with sandwiches, a “Book of Mushroom and Black Magick”, coffee and some chocolate. Maybe a map, a compass, some survival gear if I get lost in the woods. I want to be prepared and tooled up. How the hell do I find those mushrooms?

This is where a Lévy Flight heuristic kicks in (a heuristic mentioned by James Bach at CAST 2014). It is an algorithm by which animals (and humans) go foraging. "When defined as a walk in a space of dimension greater than one, the steps made are in isotropic random directions." says Wikipedia. Essentially doing something in one spot until you move to other area to do something there. So, I stand on the road and head to the woods. I try to look for sweet spots in the woods, like decaying fallen trees, dry mossy areas under pines or furs. If I have dome my preparations correctly I look for these areas and I might know where to find them. So I start roaming in the woods and stumble on an area that has something interesting. A fallen tree! Yay! Maybe I'll find some black chanterelle there. So I look closely at the area and spend time there, perhaps picking some mushrooms or just scouting the area for clues where I might find some. After a while I head to a new location keeping in mind where I am in the forest.

So I move around the wooded area in a pattern that tries to achieve the best coverage of the important areas. The pattern might seem random, but I have a mission which I am trying to fulfill with the choices of direction to wander. When I am halfway on my walk, I might want to go towards the road so I don't end up too far when the sun goes down. I spend time on areas that are either rich in mushrooms or interesting places in themselves. Maybe I learn something while scouting for porcini, something about birds or types of moss I tread on. Maybe I note down areas that have some other mushroom that I am not intending to pick (you don't want to mix mushrooms in the same basket - I don't know why...). I take mental notes, maybe write stuff down, mark places on my map. All the while trying to achieve a goal I set for myself before I went foraging. On my way back I stumble on an abandoned building. Cool! I might take a look inside and maybe I find something interesting. Maybe an old newspaper or a book? I might spend time there even though I was set out to pick mushrooms. This is interesting new place and I want to know more of it. So I deviate from my initial mission and investigate the building. It's apparently someone's old home and its walls have sunken in to the ground. Maybe the architecture of the pre-WW2 era interests me, maybe the newspaper has some information from the "ye olde times". Maybe the book has a letter tucked in between the pages. I allow myself to deviate because this might be more important than mushroom-picking.

Back to the testing world. The woods turn into APIs, GUIs and code. The moss becomes the date on which I tread on. The mushrooms... They're not bugs, if that's what you thought. Mushrooms are information, relevant information. It can be about a behavior that is annoying the user, an error message in the wrong place, a risk that needs to be communicated to the stakeholders. There might be bugs, but there is so much more. The you go foraging in the software you can apply the Lévy Flight  heuristic either by accident or purposefully. It is called focusing and de-focusing. You focus on some area to find interesting things for some time. Then you move to other area. If the area you first stumble upon is hugely rich in things waiting to be discovered, you might spend most of your time there. Or you might just stop there briefly and look for more important things to discover.

You start with a mission and you head out to accomplish that mission. You take notes and notice things. You investigate things that look and feel important. You forage information on the product under test.

Here's a scenario that might give clues to choosing mission for your foraging.  On Monday you wonder if you should go picking mushrooms. It's a fine day, but instead of going head first into woods you barely know, you go investigate. You take your dog with you and go scouting the forest. You find a batch of blueberries and eat a few. Oh, they're so nice! The dog, Rover, eats some also. On Tuesday it rains. Bummer! So you decide to go to a shop and buy some equipment for the trip as soon as the rain stops. You decide to get a basket that has compartments for different mushrooms. You didn't even think of it before talking to the shop keeper who's apparently an expert on the matter of picking mushrooms. Great find! On Wednesday you are called to the office for an important meeting. A nice, sunny day wasted in meeting.  No time for foraging today. On Thursday you get your gear, your dog and head out to woods. You check the sweet spots you discovered on Monday and pick delicious mushrooms and even some blueberries. On Friday you make delicious mushroom stew with some potatoes and carrots. Then, to top it off, you make a blueberry pie. All the recipes for these you checked online but used your own twist to make them your signature dishes. A perfect way to start a weekend.

To sum it up, Mushroom-picking heuristic is a two-fold heuristic:
- First it is a preparation heuristic. It helps you create a mind model that allows you to plan for the up-and-coming testing session, testing phases, etc. To an acceptance testing session one might prepare differently than to a security testing phase. Nonetheless testing needs preparation and a good way to tackle the preparation is the Mushroom-picking heuristic. You should make the preparing heuristics to match your own context. Think of tools, background knowledge (oracles), time-frames and constraints, people attending the sessions, bug reporting procedures, facilities, etc.
- Second it is a testing management - a steering heuristic. It helps you move from one focus area to another. Focusing and de-focusing is one aspect. With note-taking you can keep track on the areas you have covered and to remind if there's need to return that area. Keep in mind the Stopping-heuristics so you won't get stuck too much in one area.

The Mushroom-picking heuristic is not comprehensive nor should it be. It is a model that might work or you may find it useless. Perhaps you might try it and give me feedback on how it worked and how I should improve it. It is a work in progress.

Have a tasty spring!

- Pastori