Testgasm

This is a blogpost to describe what was going on in the Rapid Software Testing class held by James Bach. This is not meant to be a comprehensive analysis on what happened or the lessons I learned but few highlights and hindsight analysis on what happened.

The dice game

...this is just a few of what we had...

Those that have been attending the dice game know that the initial exercise is hard. When one gets the idea on how the algorithm goes it’s easier to play variations on the game by group of friends or colleagues (not to say these are necessarily different). I (thanks to Michael Bolton’s course), Henri Hannuniemi and Sami Lehtonen (thanks to Antti Niittyviita) had played the game earlier and so we were grouped together and given a different algorithm than the others. I was so thrilled by the idea that I could use all the skills that I had learned to crack that nut open.

The game started and we were given a bunch of dice: a handful of regular, different colour/size dice; few D20 dice; few D10 hand drawn dice; a die that was in a transparent die; some “poker dice”, etc. We had so many variables that we became a bit confused. I started a list of different things that we could be analyzing. The size, the colour, dots or numbers, amount of dice, type of dice, arranging of dice, “zero is not zero” etc. We then arranged them so that there were different D6 coloured dice stacked together in different ways, five groups of five. James came to the table, we said “2” as a guess for every single dice pile and James replied “0”.

We had the miscellaneous dice lying in the stack nearby and I thought I ask a reply on that also. I said it “2” also and James replied “1”. A one? We had a huge number of different types of dice lying there. What could it be?

We proceeded to arrange the dice in a fashion that there were 2 or 3 special dice and regular dice bundled up so that all special dice were in use. All but one group ended up in a “0”. The one that was the “1” had the die where there was a dice within a dice. A frantic math calculation ensued. We tried to sum up the two dices together, but the result was always a “1”. So I took the die in hand and turned it so that James could see only “1” and a “2”, and we thought it was a “1”. The answer was “2”. Frustration!

Testgasm

We then analyzed the steps we took to get to that “2”. We had a couple of theories, but when the die was on the table, it was always a “1”. Heureka! We took five dice off the table and said “5”. And a five it was!

The thought process took 3 persons 20-25 minutes. The sparring between the team enabled us to try different even crazy-sounding ideas to good extent without exhausting our innovation. We simplified the data and made it more complex. We tried to look for changes in the output by varying the input. We used the different models of problem solving from our lives to figure out the pattern. We also solved a pattern composing of vocal input pattern (I save that for one of my exercise in the company) and one with a difficult mathematical pattern composing of differently grouped dice.

"Give me your hardest problem!"

Like James said, we were doing testing Kung-fu! Solving problems like matrial artists! Bonk! Blam! Thwaak! Zlott! Swoorsh! Phatam! Ka-pow! Like Batman fighting crime, we fought problems! (I’m getting a bit excited here, if you can’t read it from between the lines.) We all felt like we were invincible and we had the tools to crack every problem in the world! “Give me your hardest problem and I will solve it for you!”

The high after testgasm

After we got all the patterns solved we were all ecstatic about what we had done. We were in a problem-solving high! The word to describe the feeling would be “testgasm”. And truth be told, after a serious testing session where you find something awesome, something really important; you will get a testgasm. I’m not sure if other that testers understand the rush after a successful testing session; it is hard to explain. The joy of completing a hard task and feeling joyous about it in ways you never thought you could be. That feeling is something that testers all around are looking for when they test. Your face may light up and you yell “Yes! I got it!” and other people in the cubicle stare at you like you’re deranged.

Focus/Defocus

I had heard the concept of focusing/defocusing in Michael Bolton’s class before but I never truly understood the meaning behind it. ("If focusing is focusing, defocusing is the opposite of that") I tried to look for material online, but it was all vague and didn’t make an impact. The way that James described the method was simple: “When confused, focus; when frustrated, defocus.” Wow! When testing one sometimes loses momentum and struggles with a single piece of data for long time. Using input patterns that don’t find the problem may lead into frustration. I felt exactly that during an exercise about systematic testing.

James had us testing a piece of software that had a bug in it. We were asked to find the bug and then try to figure out why it fails. The input was a valid IP address. You know what I figured out? I have so strong built in mental models about meaningful number combinations that I was grinding my teeth and sweating to break that pattern. I had tried a pattern by testing the high and low numbers, duplicate numbers, you name it. Could I just be blind to the bug? Then James said “Look into your data. Can you see a pattern?” Yes I did. And lots of them! “Now try to find an input that is as different as possible from previous data.” And guess what IP address I used? “1.2.3.4” That’s right! I broke my pattern by trying “1.2.3.4”! What was I thinking?!?!

James was like “Dude! What the hell?” and I sat there frustrated and confused. Then I realized what he had meant. If I had drawn a line to represent my test data, it would have been like this:

I just needed to break the pattern and be more random and use data from between my clustered data pattern. And with first random IP address I put in, I found the bug. Later James explained why the program behaves like that, but I can’t remember the true root-cause of the bug, but I did learn a lot about focusing and defocusing.

Hung over (from learning)

It has now been two weeks since the course. After the class I was in a high that lasted through the weekend. I thought about giving a thorough analysis about the course but I see no need for that. I had the time to let the learning infuse to my spine and now I feel that the stuff I learned make even more difference. I did have an information hangover after the class and it was hard to get back to grips with non-testing work again. I was glad, however, that I could attend the Intensive course the next week so I wasn’t that bummed.

I recommend the class for everyone willing to improve their critical thinking, testing skills and/or argumentation skills. The exercises were good and to the point. The hot seat treatment James gave to some of us gave us experience to stand pressure and perform well when in a stressful situation. I also learned a lot about myself and the way I learn. It was also cool to hang out with great testers like Samuli Elomaa and Aleksis Tulonen.

I still have a long way to go and a lot of learning to do. (After James mentioned it) I began to think myself as a constant student, always learning. I promise James (and I will talk about this to Mr. Bolton) that I will try to compare the classes done by him and Michael. That way both of them could learn what could be done better or differently. That however will have to wait a few days (i.e. weeks) as I have a growing backlog of blog posts.

The branch that you’re sitting on is getting cut

The third track at Nordic Testing Days 2012 was a multiple case study or an experiment report from Kristjan Karmo from ASA Quality Services Oü. There were 5 cases that Kristjan introduced in his track, all real scenarios with some data changed to protect the privacy (also some figures exaggerated to make them more dramatic). They were all entertaining but each of them had a punch-line that summarized the case to a lesson learned. In the end all projects had one fundamental flaw that affected the results. Read through and see what it was.

The process is a tool - a fool with a tool is still a fool. Communication is the key!

The first case study one was a company called Wolf Inc., a company that wanted to change name and become Strawberry Inc. The task was supposed to simple: Find-and- replace and go to production. Simple, fast, infallible. No testing required. What happened was exactly the opposite. “Wolf” as a word changes in different situations: wolves, wolf’s, and even more if in a different language. Also all the logos, brochures, documents, templates, etc. were affected. All this was to be avoided by asking questions about the task, by listening to the people making the change. It was all about communication.

The second case was a “CRM on steroids” by a company called Fields & Co. There were lots of people involved, lots of locations, lots of documentation and interpretation thereof. The most catastrophic mistake was a misinterpretation of a date. The date was THOUGHT to be the “ready for testing” –date but it turned out to be the “ready for LIVE” –date. 1 hour session of critical exploratory testing on the product by all parties involved. Group effort and commitment to the end product saved the day this time.

The third case was a company called Forever Ltd. where testers were doing the best they could and producing bug reports. These reports ended up on the developer’s desk and were returned with a stamp “works on my machine”. The system consisted of multiple integrations of different products so the all the contractors said that the problem was on the other contractor’s court. The solution was to meet with all the parties and talk it through. People were negotiating about their defects and issues and then he proper stakeholder would claim the bug and fix it.

The fourth case was Pepper plc., a company that had a testing team of end-users. When the testers were introduced to the project, all testing done by the developers stopped. The problem was solved introducing a few experienced experts to the team and mandating the train testers to make more effective bug reports and testing was done using the exploratory approach. The lesson here would have been the experience and the structure the testers brought to the team.

The fifth case was a company called BadCom Corporation. The requirements for performance had been made early because the system was performance dependent. The product that came to testing was 200 times slower than the requirements stated. So the team did an effort to increase the performance and they managed to get it to be 10 times faster. That was still not enough, as the system still took lots of time to respond. An assumption was made that the 20 times slower would still be good enough. The reality was far worse at the performance rendered the system inoperable and useless.

So here’s what I think:

• In the Wolf-case the work group didn’t know what they were doing or they were blindly taking orders. No-one questioned the orders by looking at the id-tag hanging from their necks, etc. So questions that needed to be asked were not asked, thus creating a situation where a consultant firm was called to douse the fire.
• In the second case there were project management and consensus problems. Floor-level did not know enough about the scheduling thus making interpretations about them. What lacked was communication between stakeholders. The last minute exploratory testing show might have been a confidence boost, but feels like a show more than productive effort. Don’t know, but feels like it.
• Third case is a typical multi contractor problem. Everyone’s interested on part made by them and will not think of others because they’re not paid to think about them. The more bugs the other contractor has, the better we look. So there was no big picture or team in this case and everybody tried to cover their own asses. Again communication and challenging could have done the trick and much earlier.
• The fourth case was a basic bottleneck-situation, where the developers thought that they can now concentrate on their job and leave the testing to the test team. The problem was solved by amping up the test team, which was a good solution, but not what I’d had done. I would have talked to the developers and convinced them to do more testing and would have shown them the results if they did the testing.
• The fifth one was burned even though a consulting agency did their best to get things better. Here the issue was realized so late in the product life cycle that the fix was almost impossible. The performance would have been the top priority regarding testing if it was a performance ciritcal system.

So, all in all, all cases were doomed because of the lack of conversation and the critical thinking of a professional tester. By outsourcing testing you outsource thinking, and might lead into these situations all over again. So think before ending in a situation where the branch that you’re sitting on is getting cut. Use challenging, questioning, feedback, workshops, to raise communication in within the project.

Communication is the key to success.

In the face of failure - part 2

Sometimes we fail to achieve what we have set our minds to achieve. That is not always a bad thing as we might learn something about the process of failing. The failure itself may require us to change our perspective or approach to the task at hand. In the first part I covered what happens when a test fails. In the second part I look into people's failure in effort to do something. All along I try to offer insight and learning possibilities to the subjects.

When a person fails (or doesn’t fail)…

Tests can fail. They reveal information when they do so. Even a passed test may reveal new information. Same thing basically apply to human failures. Why did my effort fail? What are the critical points that lead into the failure? Is there a possibility that the failure was evident and there was nothing that could be done to prevent it? If not, how can I see the critical points in the future?

Are we the cause of failure?

There are hundreds of self flagellation books that motivate you not to fail. If we are to attain a critical mind and keep it, we need to realize that we are fallible. We make mistakes, ok? There's nothing you can do to rid yourself of all failures. By keeping in mind that you are allowed to fail, you may prioritize with more precision, plan important thing more carefully, and learn from events that may lead into the failure.

The worst enemy of a person (who fails or succeeds) is ignorance. The ignorance itself may be the cause of failure, but by ignoring the root cause for a failure prevents all learning. Ignorance leading to failure is the worst cause, but still a person can learn from ignoring as much as from anything else. A person may have a reason to ignore something due to scheduling, required effort/skill/resources, insufficient knowledge, or a memory lapse. This is a kind of benign ignorance, as the ignorance is used as a tool towards the goal. If a person chooses to neglect, belittle, condescend either other people directly or indirectly, it is malign ignorance. A person failing to deliver on time because of procrastination or lack of interest in the subject falls to the same malign category.

People can learn from ignorance, more easily from benign ignorance than from malign. You can always choose to take note on the thing you previously ignored and decrease the risk to fail. Even attitude of ignoring can be changed, but it could require huge effort to do so. It is encouraged that people rid themselves of ignorance to important matters. They may prioritize some things lower than others and thus they do not ignore them completely. Asking "am I ignoring important things?" could result in fewer failures. If a failure occurs, what was ignored and why?

We may fail also because of choosing the wrong tools, approach or methods. For example making a presentation about some scientific thingy and structuring it poorly, may result in failure. Choosing wrong viewpoints (or too few) can result in biased view and thus result in failure. We may even choose the right tool and fail because of lack of skills to use it. If I don't have the sufficient skills to do this, who has? Can I utilize that person’s skills in this task? Do I have to adjust my views to reach a better result or to suit the given task?

We may make a efficient contribution towards succeeding in the task but it may still fail. We may have overlooked some important information that could drastically change the outcome of our task, or we could have made the wrong interpretation. The material or the baseline itself on which we built our effort upon may be faulty. Have we enough information to successfully complete the task? How do we determine that? Is the opinion I have conclusive enough to lead into success?

"It's their fault!" ;)

All things leading into failure are not reliant upon the person doing the task. The task mey require effort from multiple people or organizations and all cannot be controlled. To succeed in a multi-person, multi-venue task we need to establish a knowledge which person is holding what information / is responsible of which deliverance / responsible form which sub-task. If keys to our success lie in another person's hands, we need to make sure those keys open the doors for us.

It could be that the venue where you were to hold the presentation is accidentally overbooked and your effort fails. They may have suffered from a Force Majeure or something completely irrelevant to the task at hand but preventing from succeeding in it. There are some cases where one can find learning opportunities regarding the Force Majeure incidents. Can I prevent the task from failing due to unexpected events? Does a failure due to those events cause failure to retry? Can I take precautionary actions to prevent those things affecting my task?

If the environment and the nature (a stampeding elephant horde is considered a force of nature. "Jumanji!") don't cause a failure you still have take into account the audience of your task. This may include the people taking part to a presentation, the stakeholders of a software project, etc. The person doing the task has little power to effect the audience. The stakeholders may have beliefs, reasons, attitudes, etc. that may hinder the successful execution of a task. Audience may choose to ignore your topic and perform a no-show (or have more important thing to do at the time). What can I do to ensure that the task fulfills the goals regarding the audience/stakeholders? Do I need to adjust my approach so that the stakeholders don't have to struggle to relate to me or my topic? Do I need to do research on the receiving end of my task?

By asking important questions about the task and the surroundings can drastically decrease the possibility of a failure. And even if you fail, you will have a solid base for learning from your observations and the information received from the analysis of a failure. Accepting that you are fallible increases the ability to think critically. By anchoring ourselves to the opinion that we can't fail, makes us vulnerable to being biased and blind to important details and information.

Don’t think failing as a bad thing...

As we know, we all fail at something. There's nothing wrong with that. Succeeding might result in fewer learning opportunities than failing in the same task. Still, we can use the same techniques we use to analyze failures to apply to analyzing success. Did I prioritize efficiently my time and tasks? Was there a pit that I fell into thus decreasing my chances of succeeding? How can I remove that deficiency in my future doings?

Attitude towards failure is one of the biggest obstacles towards success, as we think it makes us losers or bad people (at least some people do that). Every attempt is a learning opportunity, pass or fail. Inquisitive mind is the best tool to enable learning in failure as in success.

If you found this piece of blog post inspirational, please comment that it was inspirational. If not, then comment that it wasn't. What ever the response, tell me why. If we (failing or succeeding in our doings) do not get feedback on from our effort, we cannot tell whether it is a success or not. By saying so, I don't encourage people to praise without justification or mock unnecessarily. The task for commenting is to succeed in commenting. Would you rather fail or succeed in that?