Testgasm

This is a blogpost to describe what was going on in the Rapid Software Testing class held by James Bach. This is not meant to be a comprehensive analysis on what happened or the lessons I learned but few highlights and hindsight analysis on what happened.

The dice game

...this is just a few of what we had...

Those that have been attending the dice game know that the initial exercise is hard. When one gets the idea on how the algorithm goes it’s easier to play variations on the game by group of friends or colleagues (not to say these are necessarily different). I (thanks to Michael Bolton’s course), Henri Hannuniemi and Sami Lehtonen (thanks to Antti Niittyviita) had played the game earlier and so we were grouped together and given a different algorithm than the others. I was so thrilled by the idea that I could use all the skills that I had learned to crack that nut open.

The game started and we were given a bunch of dice: a handful of regular, different colour/size dice; few D20 dice; few D10 hand drawn dice; a die that was in a transparent die; some “poker dice”, etc. We had so many variables that we became a bit confused. I started a list of different things that we could be analyzing. The size, the colour, dots or numbers, amount of dice, type of dice, arranging of dice, “zero is not zero” etc. We then arranged them so that there were different D6 coloured dice stacked together in different ways, five groups of five. James came to the table, we said “2” as a guess for every single dice pile and James replied “0”.

We had the miscellaneous dice lying in the stack nearby and I thought I ask a reply on that also. I said it “2” also and James replied “1”. A one? We had a huge number of different types of dice lying there. What could it be?

We proceeded to arrange the dice in a fashion that there were 2 or 3 special dice and regular dice bundled up so that all special dice were in use. All but one group ended up in a “0”. The one that was the “1” had the die where there was a dice within a dice. A frantic math calculation ensued. We tried to sum up the two dices together, but the result was always a “1”. So I took the die in hand and turned it so that James could see only “1” and a “2”, and we thought it was a “1”. The answer was “2”. Frustration!

Testgasm

We then analyzed the steps we took to get to that “2”. We had a couple of theories, but when the die was on the table, it was always a “1”. Heureka! We took five dice off the table and said “5”. And a five it was!

The thought process took 3 persons 20-25 minutes. The sparring between the team enabled us to try different even crazy-sounding ideas to good extent without exhausting our innovation. We simplified the data and made it more complex. We tried to look for changes in the output by varying the input. We used the different models of problem solving from our lives to figure out the pattern. We also solved a pattern composing of vocal input pattern (I save that for one of my exercise in the company) and one with a difficult mathematical pattern composing of differently grouped dice.

"Give me your hardest problem!"

Like James said, we were doing testing Kung-fu! Solving problems like matrial artists! Bonk! Blam! Thwaak! Zlott! Swoorsh! Phatam! Ka-pow! Like Batman fighting crime, we fought problems! (I’m getting a bit excited here, if you can’t read it from between the lines.) We all felt like we were invincible and we had the tools to crack every problem in the world! “Give me your hardest problem and I will solve it for you!”

The high after testgasm

After we got all the patterns solved we were all ecstatic about what we had done. We were in a problem-solving high! The word to describe the feeling would be “testgasm”. And truth be told, after a serious testing session where you find something awesome, something really important; you will get a testgasm. I’m not sure if other that testers understand the rush after a successful testing session; it is hard to explain. The joy of completing a hard task and feeling joyous about it in ways you never thought you could be. That feeling is something that testers all around are looking for when they test. Your face may light up and you yell “Yes! I got it!” and other people in the cubicle stare at you like you’re deranged.

Focus/Defocus

I had heard the concept of focusing/defocusing in Michael Bolton’s class before but I never truly understood the meaning behind it. ("If focusing is focusing, defocusing is the opposite of that") I tried to look for material online, but it was all vague and didn’t make an impact. The way that James described the method was simple: “When confused, focus; when frustrated, defocus.” Wow! When testing one sometimes loses momentum and struggles with a single piece of data for long time. Using input patterns that don’t find the problem may lead into frustration. I felt exactly that during an exercise about systematic testing.

James had us testing a piece of software that had a bug in it. We were asked to find the bug and then try to figure out why it fails. The input was a valid IP address. You know what I figured out? I have so strong built in mental models about meaningful number combinations that I was grinding my teeth and sweating to break that pattern. I had tried a pattern by testing the high and low numbers, duplicate numbers, you name it. Could I just be blind to the bug? Then James said “Look into your data. Can you see a pattern?” Yes I did. And lots of them! “Now try to find an input that is as different as possible from previous data.” And guess what IP address I used? “1.2.3.4” That’s right! I broke my pattern by trying “1.2.3.4”! What was I thinking?!?!

James was like “Dude! What the hell?” and I sat there frustrated and confused. Then I realized what he had meant. If I had drawn a line to represent my test data, it would have been like this:

I just needed to break the pattern and be more random and use data from between my clustered data pattern. And with first random IP address I put in, I found the bug. Later James explained why the program behaves like that, but I can’t remember the true root-cause of the bug, but I did learn a lot about focusing and defocusing.

Hung over (from learning)

It has now been two weeks since the course. After the class I was in a high that lasted through the weekend. I thought about giving a thorough analysis about the course but I see no need for that. I had the time to let the learning infuse to my spine and now I feel that the stuff I learned make even more difference. I did have an information hangover after the class and it was hard to get back to grips with non-testing work again. I was glad, however, that I could attend the Intensive course the next week so I wasn’t that bummed.

I recommend the class for everyone willing to improve their critical thinking, testing skills and/or argumentation skills. The exercises were good and to the point. The hot seat treatment James gave to some of us gave us experience to stand pressure and perform well when in a stressful situation. I also learned a lot about myself and the way I learn. It was also cool to hang out with great testers like Samuli Elomaa and Aleksis Tulonen.

I still have a long way to go and a lot of learning to do. (After James mentioned it) I began to think myself as a constant student, always learning. I promise James (and I will talk about this to Mr. Bolton) that I will try to compare the classes done by him and Michael. That way both of them could learn what could be done better or differently. That however will have to wait a few days (i.e. weeks) as I have a growing backlog of blog posts.

Testers are rock stars!

Teemu Vesala wrote in his STC blog comparing testing to poetry. It began a thought process where I found myself comparing testing (noun) to classical music. "Testing is like composing classical music with all the nuances involved" was pretty much the message of my. This was the beginning of tweeting that went like this:

Teemu Vesala:
Poem writing and #testing has plenty of common. See my blog entry @ #stc http://ning.it/jF60aR

Pekka Marjamäki:
@teemuvesala #Testing is more like composing classical #music: even the tiniest sounds change the feel to it. And testing tells a story.

Teemu Vesala:
@pekkamarjamaki Acctually #testing is much like any art. No matter if it's poetry, music or painting. All have same properties.I love'em all

Testing is comparable to the art because they are basically built of same thing (both in their respective context). But which would not be comparable to art? What is art?

Wikipedia (ah! the an inexhaustible source of information!) describes art like this:

Art is the product or process of deliberately arranging items (often with symbolic significance) in a way that influences and affects one or more of the senses, emotions, and intellect.

Can testing be culture? Can there be testing without it being part of the project contextual (I made up the term. Wonder if there even is a term for a thing like that…) culture? Testing is THE element that offers information about the pretty much anything, because testing occurs on so many levels: from unconscious questioning and doubt into systematic examination of the whole system both dynamically and statically. So, yes, testing is a significant part of the culture (the context) in which the testing is done.

Testing is a set items (techniques, approaches, tools, strategies, etc.) that are deliberately arranged so that they form a basis to achieve a wanted result. As we implied before, the test can be performed adhoc, but as art, it does not always reach the goals that are sought. Testing is at least partially subjective, because it is the opinion and interpretation of the task at hand. By removing the subjectivity of art it is possible that ideals of beauty or a non-intelligent interpretations of things are forced to the spectator. The subjective nature of art is a force that makes art – well – art. The nature of the testing is subjective, which is definitely an asset to testing itself, because different people interpret different things in different ways. We can always say that "This thing is X. Period!", but it is not necessarily correct in context. The contexts makes subjectivity important.

We can not underestimate the importance of what feelings are to testing and testing to feelings. Both of these are factors that affect each other. In art, the composer, painter, choreographer, sculptor, you-name-it have feelings or emotions. These are driving forces that will create art. Of course there is art done without feelings, but is it art? Or is the lack of feelings a feeling itself? Is all that is art on some level? In addition, there is an art that does not affect the feelings (in a given context and subjectively). Art is trying to be bi-directional with the emotions. Testing tries that too: feelings (understanding the risks, prioritization, etc.) drive our testing forward and testing is guided by the feelings to help in our decision-making.

Art is expression, communication, statement and pleasure / annoyance. This is true for testing, word for word! Testing is expression: we express the current quality of the system by testing. Testing is communication: There is no other way to express and inform the quality to the stakeholders. Is there? Can quality be expressed by programming? It creates the quality which the testing expresses. Testing does not make quality. In addition, when it comes to pure communication, testing is a social event where communication between groups is important! How can we express the quality, if we do not communicate it to the stakeholders? And testing makes a statement: Michael Bolton said "Testing is defending the quality of the product." Testing comments the quality, testing defends the quality of the product. If testing doesn’t do that, who will?

What comes to pleasure, to me, every day of testing a great pleasure (my testers have undoubtedly noticed that). Every day that I can to promote the triumph of testing both organizationally and the genre produces pleasure. Every day that I learn more about the testing produces pleasure. Every time when I find a new bug system it produces pleasure (displeasure for the project manager ;) ).

To summarize: Testing is art. Testing will be appreciated as art. There are mathematicians, which state that mathematics are the greatest art (graphs based Fibonacci numbers, chaos theoretical curves and fractal art). Testers are therefore artists. We testers who respect ourselves and our skills can hold ourselves as the Rock stars of the IT world (or as Leonardo da Vincies). We bring the joys and the sorrows, we express feelings, even beauty. For me, the test is the biggest art.

Thoughts about ISTQB Advanced level certification course and exam

I got a permission to enter the ISTQB Advanced level Certificate - Test Manager course and to apply for the certificate. It is a course provided by Finnish company called FC Sovelto that is used to replace the former Intermediate and Practitioner level certificate by ISEB. It's been a year since my former certificate exam of foundation level. Since then I have had plenty of opportunities to assess the pros and cons of certification.
The course provides coaching for the certification exam. The exam can be taken without the course and I'm sure plenty of people who take the exam do not go to courses especially if they're paying the whole thing themselves.

So, what can this certificate give me? What are the gains in professional level? How does my company benefit from me getting a cert? How does my future get brighter if I get the cert? What benefits can I achieve by going to the prep course? There are many questions and the answers for an exploratory extrovert tester like me aren't always the ones I'm liking to hear.

How does a company benefit from someone to certificate themselves as a test manager? My first thought is "No how" 'cause it's only a certificate - a piece of paper with a watermark on it and a fancy signature. How could a company benefit from sending an employee to a course that might be paid by the company (and it cost a whopping lot!) and it may cause the company to check the salary of the certificated person? They're all expenses to the company! On the other hand is it worthy to encourage people to pay for their own certification? That way there is no direct expenses to the company.

So how does the certification show in companies that have them? In reality (or at least in my reality) the certification of an employee is a great benefit in a company as a whole. The certificate in itself is a great achievement. In competition situation the company can have a trump card and say "we have a fully certificated test team and test management" by which they can sell better quality (this means they must reach the set bar). This is also important to the image of the company because there is someone in the ranks who know the standards and principles of the industry (though artificial they may be). Some client can even have the certification as a requirement for the deal to be closed. I.e. Microsoft certificates are mandatory on some projects for some clients, so why can't a testing cert be (somewhere in future).

The other angle is pure craft. How can the skills and knowledge of an individual affect on a corporate level? When company acquires more knowledge and skill it increases the "skill capital" or the "skill pool". The know-how not present in company must be attained from other sources like contractors, or the job the skill was required for was carried through with present know-how with might have resulted in low quality or undesirable resuts. Certificate training support the acquisition of the skills and the certificate is a document to prove that the requirement for those skills are met. (Some may argue that the certificate is nothing but a proof of capabilities to learn litany of test vocabulary. They can have their opinion if they can make a good argument about it.) Does it provide skill it the course only aims to pass the exam? How can you be sure that the course provides the skill not a lithany? These kind of thing are worth to take into consideration when deciding whether to attain the course or not.

Third point of view is both corporate and personal. Like every testing event the course is a great opportunity to make contacts. The word about your company gets spread around and the person attaining the course gets to meet other test spirited people from other companies and domains. This can lead into cooperation and contracting that can prove valuable for both companies. But most of all the testers can exchange thoughts and view about testing and test related stuff.

In conclusion on the corporate side, there are three things in certification (and the course) that can benefit a company: the image, the craft and the connections.

There are bad things in certification as in all good things. The image of the company may begin to transform into a rigid and standard-obeying corporation in some professional scenes and may hinder the acquisition of these kind of clients. More over the cert may lead into a rigid process model that trim out all exploratory spirit and drive people to an inflexible frame where there are no room for innovation or personal thought. These may not be accurate in any way but they are assumptions what may happen is a land slide is triggered. The testing processes can be agile and light (I’m not going to explain what is "agile" or "light") even though they are based in standardized processes.

On a personal level certification is harder assess in pros and cons. The con might be the box-thinking and veering towards a specific mind set. Certification might cause the tester to take pre-chewed (standardized) procedures and techniques as his own and forget all other. On the other hand not being certificated might mean you have to reinvent the wheel. My opinion is that the ISTQB certification should be considered as in "learning" the certification rather than "owning" the certification. Can one maintain freshness in ones thoughts while obeying the standards? Can one be separated from the limitations and restrictions brought by the certificate (are there any?) and all the while know the testing terms and standards on a certification level? Is the "main stream" a bad thing? Is the "counter stream" a good thing?

So what benefits does a certification give you? What disadvantage may it bring? The forthright benefit on personal level is the increase in personal skill in case you take part on the course and you do not already have the skills. If you know "everything" before the course and the exam the raise in skill is not relevant basis to acquire the cert. Disadvantages may also include the expenses the course and cert may cause especially if one does not pass the exam the first time.

How does the certification benefit ones career? How does it strengthen/weaken the position in company? The certification brings certain stability to one's position in company. It indicates that a person knows certain things and he has a document to prove it. If it is the industry standard (in one point of view) certification then its value is much increased. I won't describe any alternative certifications in testing industry 'cause I know so little about them, but what I’ve heard there are some certifications that are not ISTQB-related. Nevertheless the certification strengthens the position in company. The weakening of ones position is relevant only in cases where the certified person does not meet the standards of the certification in every day work. This means that the bar is higher than for the uncertified person. Certification also benefits the career in long term as a personal marketing tool. It may be the ticket to places where uncertified tester can not go (although who would want to go there? *wink*). in job interview the certification is a certain way to prove you have the skills and knowledge necessary. I won't go in detail into the pros and cons of certification in job interview but lets just say that its a double-edged-razor - certified may get labelled as box-thinking tester while uncertified are thought to be more fresh thinking. And just like in corporate side the contacts are the bread and butter of testing events. This benefits both the current situation and the future in form of scouting new job opportunities etc.

So, how do I get the most out of the certificate course and from the certificate itself (given that I pass the exam)? I strive to create as many a connection as possible in the event and to bring up the name of my company in every possible turn. I try to challenge the testers and to get as much testing info as possible. I also try challenge my own thought patterns against the ISTQB model and the other way round so that I don't lose the freshness and the awareness achieved from my work in exploratory testing. The more i question the things I hear the better I learn new thing. I get new points of view to things and I can also learn new thing to support my knowledge in testing in general. Maybe I think some of these ISTQB thing are good for keeps and I take them with me and use them where best suited.

Preacher man Marjamäki

I was reading own blog entries and I started to truly think about my own writings. Those writings have clearly the idea and insights - some of the insights derived from others, but still. However, part of my text is “porridge” - it lacks the red wire. I am a novice blogger, so it is certainly not a great disadvantage here, but when I read more and more testing blogs of other, I see they write long and coherent entities, where as mine are short and fragmentary.

So I have found the opportunity to develop myself from my own errors, which I did not even know were there at the time of writing. Someone may disagree on whether the writing style in itself flawed, but perhaps "lacking" is the word that describes it best. Whet perhaps led to the creation of this writing was a text about challenging oneself by James Bach. So I found a questionable approach in my posts, and it cries for improvement.

So how do we improve the concept, which is not really wrong - or at least not harmful? How to motivate to repair, which in itself is not necessary to be repaired?

-

When the tester examines an application (i.e., exploratory tester) he finds bugs. He also finds deviations. Fault in this case would probably be writing off topic and / or misspellings within the text. Potential errors nonetheless.

Typographical errors in this case are the lowest of severity, and the repair improves (only) the cosmetics. They make it more compromise usability (readability) when the user (reader) has to constantly think about whether a typo is a typo or deliberately wrong (or even a different word). Repair is easy in these cases, so despite the low severity the correction should be made to find out the “bug” is noticed.

Factual errors are more severe errors. It may mislead the reader, which is not usually the purpose of the blog. In addition, it reduces the confidence enjoyed by the writer, so his texts published as “true” are not considered true but false. Reviewing context is crucial! Factual errors, severe errors! Correction in these cases results in a new version of the text or freezing the text. If a bug can be found in an application and it prevents the use of the application to full extent or to which it is designed, it will be deactivated and make the necessary corrections in order to function as desired (as expected) in a way.

Deviation in this case is a deviation from the actual or supposed truth. If something does not match the reader's prior expectations, he considers it of a deviation. A deviation in itself is not a fault, it has been just implemented different from expectations. If someone publishes the text, which is different from the mainstream, he must to be able to justify his choice to gain of user satisfaction (in which case it becomes a feature), or the deviation turns to be a fault (i.e., a defect in the text). The same applies to applications where a feature is the done in violation of the requirements (prejudices, assumptions, expectations).

Derived from that: Is there a previous blog post that is defective? Does my unintentionally preaching-like blogging result in faulty, incorrect or different writings? My bias is that they are both defective and deviant. Let’s see whether it is true...

My intention in the text "How do I interview a new tester?" was to write down ideas and to create guidelines for myself and other similarly interested, who wrestle with the same thing. The text is incomplete on many parts, due to a lack of perspective. References to existing texts and comparing them would have added depth to the text. For example, Software Testing Club, funny book, "The Ridiculously Simple Guide Test Building A Team" would have been good reference material ... Ashamed that I made probably most of the "best proven" things mentioned in the eBook and I probably asked exactly the wrong things. (As it turned out, I wrote the whole shebang again. But this refers to the original, Finnish version of the text.)

"Performance Testing Integration Project" was designed to provide a clear frame on performance testing – area where I hadn’t gone before. The post studied the technical point of view and may even provide a reference material for those interested. As a result, the post was quite successful individual, in my opinion. It was based on existing performance testing of the model. This could have been mentioned in the text, because I make it seem like I invented the whole testing model.

So that this blogging does not change into defensively, I want to keep it in the critical path. As I told earlier, these were the assumptions and outcomes, and comparing them. The outcomes were somewhat different from the assumptions, but in some cases (as in "Performance Testing Integration Project"), the deviations were very small. Although they were not well-founded, they were such that their correction is not priority number one. Au contraire, "Testing Jin and Jang!" fails to meet expectations by a long shot. As I say in the text "My thoughts began to roll like crazy at the time -", which was true at the time, the red wire in my head is not conveyed to the text as I would have liked it to. In retrospect, I could filter the preaching part out of the text, and focus on the essential and the cold facts. Would I be able to submit my own experiences or to refer to other’s experiences on the subject? Could that material be gathered and refined in to a good and sententious and incite other to think "like mad"? The text is lacking the touch with reality, which prevents the reader from taking the text seriously. This therefore calls for a re-write due to the defects and deviations of the post.

-

So, how to motivate the repair, if the repair itself is not necessary? Self-improvement! If you have the opportunity to develop yourselves, it should most certainly be done. You never know where reviewing your own shortcomings may lead.

In order that this blogging does not turns into preaching on behalf of self-improvement, I would like to stress that all faults, errors and deviations should be identified and investigated. The correction of these may happen spontaneously by studying and searching for them. This could be like examination of the desert: the examination of the sand reveals an anomaly in the smooth surface od the desert (the tip of the pyramid), examination that the finding (digging) is revealed to be mightiest tomb monument in the World containing divine treasure and riches beyond imagination.

How do I interview a new tester?

I'm bound to get my baptism in fire in the field which I have no previous experience. I'm going to interview two would-be testers ja pick the most qualified (or suggest to take both if they are both very good, no point cast pearls before swine). He/she/both (yes there is a male and a female applicant coming) is/are applying for an intern from university re-educating program where people are trained to be software testing experts. (Yeah, no one can be an expert without proper practical knowledge. But to become expert straight from the school bench! Huh?) This is a great opportunity for me to try my wings as a test manager and as an interviewer. Because I'm the one and only tester in our company (or fully involved in testing) I get to have a say in the selection.

Now I should find the right questions to dig out the info about occupants' testing skills. Because the information is scarce and hard to find (at the time of writing I searched stuff in Finnish and everybody seems to keep these things to themselves), so I decided to write mine down for later use. After I had searched far and wide for the perfect reference material I started writing... only to later discover some really useful material. These were in English, so they support the re-writing more that the original post.

I managed to find a funny eBook about building a testing team by STC. The book is hilarious because it portrays the interview process as a scripted, unintelligent process where the true merits of testers are irrelevant. However it contains a fragment of truth, which I incorporated in my interview material.

Also I found a document written by some guy called Kaner *wink* and I briefly had look at his thoughts about recruiting. He had some great thought about forming the right kind of questions like:

"What would you do with a product that came to you without specifications?'

Instead I ask,

"Have you ever worked on a product that came to you without specifications? Tell me about the challenges this raised and how you handled them. (And then, as a follow-up question,…) What do you think you did particularly well in that situation? (And then…) What did you learn that will help you handle this better in the future?"

The document contains PRETTY good information concerning large scale recruiting. This however was about recruiting a test intern for two and a half months. So I took a quick look at the document and buried it deep in my mind, and it popped out while I was rewriting this post! (Man! was I dumb not to read it fully through! Next time, baby... Next time. We got a good intern 'though.)

Off to the interview!

First of all the applicant should tell about his/her experience in testing scene and IT-world. This should include at least:
- project models (SCRUM, Waterfall, etc.) bonus is always to include one one your company uses most of the time
- testing levels (unit testing, integration testing, system testing, etc.)
- techniques (manual, scripted, automation, etc.)
- methods (exploratory, experience based, bug catching, test case based, ets.)

If the applicant does not include all these in the story, the interviewer should feel free to ask about those. Depending of the possible future role, one could also enquire the following:
- test tools (what has the applicant used, how and to what purpose)
- test planning and designing experience (test cases, test plans, test level plans, automation architectures, etc.)
- experience in code writing and project managing (PERL, LAMP, .Net, Java, etc.)
- test managing (reporting, managing team, customer support, etc.)

If the applicant has education, taken classes or what ever that may support the testing, it could be enquired if it has not come up. In addition possible certificates (although overlooked by some ranking testers out there) in testing genre and IT in general can give indications of the persons capabilities if the needed amount of information has not yet presented itself. In addition if the applicant has some examples to show from test cases he has designed or scripts written they might prove useful.

Even though the applicant has the merits to support his selection he must be valuated as in fitting in the team. If you have a group of tester at their twenties, a world-seen tester may change the dynamics in the team and break the successfully running engine. Whether looking for team with diversity or as must similar personalities and skills, the key is to make sure the "new guy" fits in. This raises the verbal and written skills of the applicant.

Whether the applicant is suitable or not the most important thing there is about a would-be tester is that he/she must have passion for the craft! If you have an applicant with considerable skills and merits but no passion what-so-ever he'll only hinders the test team and makes a good thing plummet to ruins.