Making a requirements pie

This is a summary of the track by Ainars Galvans at Nordic Testing Days 2012 and how I viewed the subject. The topic was “Care for quality, not for bugs! Do you know the difference?“.

I met Ainars the day before the conference at the hotel and we discussed about bugs and quality. We are on the same line with the overall quality but I disagree with some points he made. So here is what happened and how I saw it and some thought about challenging the claims.

“The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.” –Cem Kaner

See the difference?

True and true. Ainars had great thoughts about increasing the value of the bugs. By focusing on the bugs that had a story that compelled to the developer (and the triage of people deciding which to fix), you will get more out of the bugs that you find. When I think of myself as a tester, I tend to get as much information about the product and thus uncover as much bugs as I can. I know that by doing so I learn about the product, the risks, the critical areas, the things that people making decisions value the most. When Ainars said “don’t report the bugs that aren’t getting fixed”, I was shocked. Am I doing it the wrong way? I may have a huge amount of rejected bugs, but it’s my way of learning. What if I uncover some minor looking manifestation of a bug that turns out to be a showstopper? I don’t know if I don’t ask and communicate.

Usually when you start communicating with the developer about a bug that seems like a show stopper, he will immediately see that it should be fixed immediately. If he can pinpoint the problem, you won’t need to make a bug report if the problem is sufficiently communicated onwards. So this is where I disagree with Ainars; I will make bug reports but I get the important ones fixed.

Ainars had great thoughts about prioritizing testing. He said that he’s not going to spend his time on testing something he KNOWS is going to pass, so he tests something he doesn’t know that will pass. He saves time doing so and can focus on the important areas of testing. So you should always do the most important testing at the every moment.

How do you know what is important? Ask! I ask if there is some area that is valued by the product owner and I focus on that. Then I ask what the next valuable thing is and focus on that. By time I get some testing done I can start formulate my own view about the importance of different things. So Ainars had a good thing going on there, as he knew the product so good he was able to guide his own testing prioritization using his knowledge of the product. The knowledge acquired by learning and testing the product.

Picking up requirements.

Ainars also introduced a cool way to do exploratory testing. He called it “Requirements driven exploratory testing”. They were almost like test cases but more like missions or topics, which were then tested. They were measured using traffic lights: Green (good enough testing AND good enough quality),Yellow (not started) and Red (not good enough testing OR not good enough quality). I challenged the concept as the testing team was focusing on only documented requirements, and I still do! There is a need to test also things that the client takes for granted and things that are discussed but not documented. I also challenged the fact that requirements are usually ambiguous, and that “requirements are not items to be gathered”. They're not like apples that you can get into your basket and bake a testing pie out of them. Ainars could clarify that if he can. From the presentation I did not get an answer to that.

As a summary, I got a lot out of the track as it provoked me to think about how I do my testing, but also I got a chance to challenge the speaker. I feel that Ainars has a lot to say about this and I will be checking his blog as often as I can. I hope he has a way to address my concerns about the requirements and “not-important” bugs.

How do I interview a new tester?

I'm bound to get my baptism in fire in the field which I have no previous experience. I'm going to interview two would-be testers ja pick the most qualified (or suggest to take both if they are both very good, no point cast pearls before swine). He/she/both (yes there is a male and a female applicant coming) is/are applying for an intern from university re-educating program where people are trained to be software testing experts. (Yeah, no one can be an expert without proper practical knowledge. But to become expert straight from the school bench! Huh?) This is a great opportunity for me to try my wings as a test manager and as an interviewer. Because I'm the one and only tester in our company (or fully involved in testing) I get to have a say in the selection.

Now I should find the right questions to dig out the info about occupants' testing skills. Because the information is scarce and hard to find (at the time of writing I searched stuff in Finnish and everybody seems to keep these things to themselves), so I decided to write mine down for later use. After I had searched far and wide for the perfect reference material I started writing... only to later discover some really useful material. These were in English, so they support the re-writing more that the original post.

I managed to find a funny eBook about building a testing team by STC. The book is hilarious because it portrays the interview process as a scripted, unintelligent process where the true merits of testers are irrelevant. However it contains a fragment of truth, which I incorporated in my interview material.

Also I found a document written by some guy called Kaner *wink* and I briefly had look at his thoughts about recruiting. He had some great thought about forming the right kind of questions like:

"What would you do with a product that came to you without specifications?'

Instead I ask,

"Have you ever worked on a product that came to you without specifications? Tell me about the challenges this raised and how you handled them. (And then, as a follow-up question,…) What do you think you did particularly well in that situation? (And then…) What did you learn that will help you handle this better in the future?"

The document contains PRETTY good information concerning large scale recruiting. This however was about recruiting a test intern for two and a half months. So I took a quick look at the document and buried it deep in my mind, and it popped out while I was rewriting this post! (Man! was I dumb not to read it fully through! Next time, baby... Next time. We got a good intern 'though.)

Off to the interview!

First of all the applicant should tell about his/her experience in testing scene and IT-world. This should include at least:
- project models (SCRUM, Waterfall, etc.) bonus is always to include one one your company uses most of the time
- testing levels (unit testing, integration testing, system testing, etc.)
- techniques (manual, scripted, automation, etc.)
- methods (exploratory, experience based, bug catching, test case based, ets.)

If the applicant does not include all these in the story, the interviewer should feel free to ask about those. Depending of the possible future role, one could also enquire the following:
- test tools (what has the applicant used, how and to what purpose)
- test planning and designing experience (test cases, test plans, test level plans, automation architectures, etc.)
- experience in code writing and project managing (PERL, LAMP, .Net, Java, etc.)
- test managing (reporting, managing team, customer support, etc.)

If the applicant has education, taken classes or what ever that may support the testing, it could be enquired if it has not come up. In addition possible certificates (although overlooked by some ranking testers out there) in testing genre and IT in general can give indications of the persons capabilities if the needed amount of information has not yet presented itself. In addition if the applicant has some examples to show from test cases he has designed or scripts written they might prove useful.

Even though the applicant has the merits to support his selection he must be valuated as in fitting in the team. If you have a group of tester at their twenties, a world-seen tester may change the dynamics in the team and break the successfully running engine. Whether looking for team with diversity or as must similar personalities and skills, the key is to make sure the "new guy" fits in. This raises the verbal and written skills of the applicant.

Whether the applicant is suitable or not the most important thing there is about a would-be tester is that he/she must have passion for the craft! If you have an applicant with considerable skills and merits but no passion what-so-ever he'll only hinders the test team and makes a good thing plummet to ruins.