The magic medallion of testing - summary of a keynote by Rex Black

This is a summary about Rex Black’s keynote about Testing and Quality management at Nordic Testing Days 2012. This was the first keynote on the second day. People were still tired from the evening activities and some even skipped the keynote altogether. I was keen to see Rex’s track as I had some… opinions about ISTQB and the certification system as a whole. So, to be there and be able to challenge him was an opportunity I didn’t want to miss.

On Monday we had "argued" with Rex about metrics and coverage. Rex commented Sami Söderbloms statement about not measuring coverage by giving an example. The discussion went something like this: “With all things being equal, the first tester returns his test report with 90% coverage and another with 15% coverage. Which one would you trust more?” Rex commented. I raised my hand (the inner voice screaming "Me, me, me! Let me answer!". Sami then said that I could answer the question. I tried to question the true coverage by asking about what has been found. Rex replied by pleading to the all-things-being-equal statement. All things that were found were supposed to be equal (so as important bugs, issues, etc.) “But all things are never equal in software testing”, I said. Later I realized that if all things were equal, how can there be a difference in the coverage?

If we have all things equal, isn’t the coverage supposed to be equal too? If two clone testers are put to test a product in exactly the same way, don’t they both produce the same amount of coverage in any way you measure? Change anything and all things are not equal. So if in an all-things-being-equal-situation we have difference in coverage, our model to measure testing might be faulty, the person interpreting the results may be biased, etc. So how can one make an example from a situation that is faulty by default?

The keynote however began with Rex warning about using the c-word (coverage) and the m-word (metrics). I immediately knew this wasn’t going to be pretty. ;)

Rex gave a few cases where a manager was oblivious about the need for skilled testers. One manager had supposedly said something like this: “The requirements of a tester: heartbeat, respiration.” That struck home right then. I knew these managers and product owners exist! I had been in a project where the solution to testing was “ask people around if they have the time”. I then asked around and gathered as good a bunch of people I could find… and I tried to train them to be adequate testers. Some became good testers, some didn’t, but I had a team.

Feels like the requirements for a tester in fact are the aforementioned heartbeat and respiration. In some cases, though, also a certification might be squeezed. If we find a tester with a certification, one has to be a good tester! Right? Rex mentioned in the talk (which I already knew from previous experience with the ISTQB) that people who certify themselves with the foundation level certification do not need any skills from testing before the certification. It is “an entry-level certification”, Rex says. So how come people with no skills are certified as testing professionals? This leads to a situation where unskilled, untalented testers that look better to managers than people with REAL skills and attitude towards testing. I that what it is suppose to be doing?

The certification thing got me so winded up that I almost missed the rest of the keynote while trying to douse my building rage. (Just kidding! ;) ) Rex had some good thoughts about how to veer the Agile testing more towards intelligent testing – away from “Unit testing + Acceptance testing = Proper testing” an more towards testing throughout the life cycle. Automated testing is there only to support manual testing*. Usually Agile testing is a Waterfall within the Agile cycle where testing is done at the end of the cycle (if we have the time). Testing earlier, reviewing the deliverables in time and early enough, challenging designs, etc. – not just automating unit tests.

Rex also talked about the cost of bugs, but there was no ground breaking new information about that. I have read that from the ISTQB syllabus a while ago. And I still believe that counting bugs and measuring them against each other will lead into biased view that “all bugs are equal”. James Bach said it well once that “bugs should be thought as unicorns” How many unicorns fit into a cubicle? Two?) How many bugs are in a product? I know this is generalization from Rex’s part but I think it dangerous to make such claims as (like I mentioned earlier about the decision-making skills of a manager**) people WILL make the tools out of the metrics to compare PEOPLE against each other. "I found 4 bugs and he only found 3. I'm better, right?"

Rex also talked about the quality management. He mentioned that good testing doesn’t make good quality in its own. I agree. We need to be able to start making good quality from the top, starting with directors. We also need to be able to improve the processes, tools, skills and communication to make better quality. We can do it all the time while we work by challenging bad behavior and encouraging good behavior.

Anyway, I feel that the foundation level certification is not currently building the community towards good testing. It has become a magic medallion that is bought from the wandering gipsy saleswoman. And when the kings and queens see the medallion they say “This must be a skillful knight as he has the Badge of Knighthood”. And yes, the medallion is bought, not earned. The training courses for the foundation level solely aim to give answers to the exam questions. They do not encourage people to learn, but to memorize (the worst thing is that they might say that one doesn’t need this information in practice). Rex said that when his consultancy company holds these courses they focus on practical use of the things trained. I wish I could once attend one of his company’s courses to see how it is done.*** The whole certification issue might need a blogpost of it's own, but it will have to wait. Some people have already covered that for me.****

You might get a message that I didn’t like Rex or his keynote. I did in fact, and I encourage all of you to read Rex’s blog and the books he has written. I don’t say I agree with all he writes but they give you a lot to think about. Rex as a person is (to me) "a happy giant"*****. What I liked most about him is that he really listens to what you have to say. We had inspiring conversation with Rex after the keynote although I challenged him about the ISTQB. I hope to hear him talk again at some other event. And I was bummed that he couldn't join my workshop... :(


*) And to build confidence, some say. But, like Michael Bolton said in the Rapid Software Testing course: “Were not in the confidence building industry – we’re in the false confidence demolition industry!”
**) Says a guy who spent two years as a test manager and is currently a maintenance manager deputy.
***) Do you dare to invite me to a course, Rex? ;)
****) Jari Laakso's post http://jarilaakso.blogspot.fi/2012/02/test-is-dead-vs-istqb-kills-people.html
*****) He's like 7 feet tall.

High Six!

"What happens in Tallinn stays in Tallinn." 

That is not the case here as I will share some of the cool stuff that happened after the official Nordic Testing Days conference program. I had the grand master-plan to go into to the room and do some blogging, but obviously I chose to confer with my fellow testers.

The special event was at the 7th floor of the beautiful Meriton Grand Conference & Spa Hotel from where you could see the walls of the old town of Tallinn. The scenery was beautiful, the snack food was great, and plethora of beverages to suit the tastebuds of every mouth. More so, the place was loaded with thinking, breathing testers (which at one point demanded to go to the balcony to look at the great scenery). The opportunities to confer were limitless.

The conference organizers had invented this “speed dating” conferring game where a person had the opportunity to get to know another in 3 minutes. After 3 minutes the partner changed. Or that was how it was supposed to go. The people were so interesting so after 3 minutes the talking continued and the organizers had to go push people to switch partners. I met awesome people from Skype, Swedbank, Asa Quality services, and many other companies. The good thing was that they all had business cards. The bad thing was that I didn’t. Mental note: “Ask HR to get me some. In fact I DO meet people outside F-secure.”.

I had good conversations with Sami Söderblom, Jari Mäkeläinen and Rex Black about testing, economics and Jujitsu. Also Ainars Galvans, Helena Jeret-Mäe, Tobbe Ryber, Raimond Sinivee, and the guys from Playtech! I really appreciate all the insights that I got from you about test automation and Model-based testing!

The hotel staff then ushered us out of the building and we went to a German restaurant to have some refreshment. The Estonian-Finnish-Swedish group ended up talking about how to improve one’s career. All in all the discussions after official program was just as good as the tracks and workshops. It was nice to share thoughts with other like-minded. And it was cool to see the pictures from the evening activities! :D

A little more than a week after the conference, I’m still feeling uplifted by the good people of Nordic Testing Days at Tallinn, Estonia. I was in the train a few days ago and I saw something that once again brought all the memories back to my head: High-Six!

What happens when you put Estonian, Swedish and Finnish people in a German restaurant? -High 6!

Be the best known if not the best!

This blog post is a summary and might not justify the zeal, the enthusiasm, and the humor the original keynote had. Again, this is my view about the track. There might be something that I have misunderstood, so challenge me. The first day's last keynote at Nordic Testing Days was done by Torbjörn “Tobbe” Ryber, a Swede with great sense of humor* and huge talent in speaking about testing**. I was never heard of this guy before although I had a book written by him in my bag!

Torbjörn "Tobbe" Ryber

I had expectations*** that we were going to receive a rant about getting oneself certified and perfecting the art of writing test cases, or something of that general direction. Little did I know that this was going to be one of those life changing experiences. Before the keynote I had some views about my own testing career and the direction I was trying to go, but the insights Tobbe would give. The track was called "How to become a really great tester"

The presentation started with simple introduction to what kind of career Tobbe had earlier, and ended into this magnificently inspiring talk about building a software testing career. From year 1995 he had been reading testing books and doing testing to some extent. If I remember correctly he was appalled by the bad quality of something their company was developing and he was thrown to test it. He had different experiences and events that affected the career choices he had made, but he tried to get as much information about testing as possible. Around year 2000 Tobbe decided that he was going to be the best known tester if not the best. Also there were no suitable classes for him, so he started to teach testing.

So, this guy decides that he wants to be a tester, and to be a good one. Can we just decide that we want to be something and then get it? Obviously we can. This reflects to my own career as a tester. If Tobbe started his career in 1995, mine began at 2007 by getting thrown into the deep end of software testing. I was assigned to do test planning, design and execution to a mail/parcel logistics software project. Having no experience of testing theory or practices, I began to search every single Finnish testing blog, book, and/or expert from which I could illicit information. I said to my boss that “I want to be the best tester in this company”, which was fairly easy as I was the only one.**** Nonetheless I had similar start on my career as Tobbe had; we made a decision and we started acting on it.

Back to the business! Tobbe describes his career in three acts: the second one beginning around 2000 and trying to float in the IT-crisis. He fought the situation by getting certified, by joining the Swedish Association of Software Testing, and by contributing to the community by teaching testing and applying his accumulated knowledge to his work. This all seems like an easy job, but I can assure you that while trying to do the best job at the office and simultaneously trying to gather as much knowledge and skills as possible is not a walk in the park. He started to build his fame by writing about testing, joining peer workshops and all kinds of activities. Eventually he published a book about Software test design and thus building even harder foundations to his career.

So if you want to be well-known in the community, you might consider the following:

• Have something to say*****
• Go and talk to other testers within your community
• Hold  workshops, trainings, tracks, experience reports******
• Write about what you do and what you would like to do

I started my career first by reading blogs and articles. Then I went where other testers were and started conferring with them. I got my first testing connection by sending an email to Maaret Pyhäjärvi and asking does can I have some of her training slide sets. Then I started going to events and trainings to get more connections and eventually started holding workshops on my own. I also started writing a blog about testing, first I wanted to challenge the community to be more loud about itself and then about what I did and what I wanted to do. So basically I started to spread my network, and gathering experiences and connection.

Tobbe had achieved his status in the community by doing all kinds of activities that supported his career, skills, job opportunities and familiarity to others. He chose to do things that he liked and not to do things he didn't like (thereby making testing more FUN). He wanted to enjoy testing. During the keynote he never said “certify yourself, “do this” or “read this book”, he stressed the freedom of choice for people to develop their own career to the way they choose. One thing he stressed, though. Everyone should have a plan how to develop oneself. The route chosen is one’s own business, but without a plan the route is hard to be travelled. The plan may get changed during the travel – even the destination might change – but that’s a part of growing and developing. Nobody knows everything, but one could consider planning it ahead.

When	What	How	Why
2012	Have a public presentation	Search for conferences near Helsinki and propose a track or a workshop.	To gain confidence in performing and to gain familiarity
2012	Publish an article	Search for a suitable publisher and offer an original article about something testing related	To gain familiarity and to establish a base of references for myself or others
2013	Take the BBST exam	Talk to people who have done it and find out how, when, where it can be done, and WHY	...

I made my 5-year plan a year ago. I have almost achieved the goal in the first year, but I still have a long way to go. And I have also made a 10-year plan to complement the first. I have had a lot of sidetracks on my plan, but the overall direction has been the same all the way. I think Tobbe had had his share of sidetracks and obstacles during his career, but he is the best known Swede testing expert to me.

Just like Tobbe asked after his keynote, I will ask you: “What is YOUR Plan?” How are you going to become a great tester?

The first step is the easiest: Want it.

PS. My plan looks like this:

*) A Finn saying a Swede has a sense of humor means that he has an enormous sense of humor to the rest of you.
**) I haven’t seen him test so I cannot say he has a talent in testing. ;) I assume that he has some talent, but again this Finn-Swede-competition-thing might mean that he also has a huge talent in actually doing the testing.
***) Yes, I’m still a Finn. ;)
****) Later I rephrased it to be "I want to be the best in the company in testing" and later "I want to be the best known tester in the Finland".
*****) If you don’t have something to say you could start by challenging someone who has something to say. A good place is to challenge those whose blogs, papers, book you read. Even though the challenge might be Devil’s advocate –kind of challenge, it usually generates conversation and thus give you something to say (and you might learn something new).
******) You could start at your own office to get familiar with speaking out loud and to face challenging. You can then join peer conferences and share what you have already done to your colleagues. Eventually you can proceed to talk in conferences and events, if you feel like it.

Bringing flavor to the functional testing

"Yo! Let's bring som flava to tha functional testin'!"

I had made a plan to attend at least one workshop on the first day. So instead of going to the Beta-testing workshop I decided to build my skills in the NFT-side. To me, non-functional testing is performance testing, security testing, conformance testing, etc. Things that measure the quality criteria of the product (how fast it is, how secure is it, etc.). As I am a mostly a black box tester, I usually try to focus on the behavior of the functionalities and what the product does. This doesn’t exclude the other approaches, but that’s just my main focus.

The workshop was Non-functional testing on mobile devices by Nikolai Pavlov. I was thrilled that I would be able to test mobile devices with an expert, so I was prepared to be blown away. Sadly, my expectations were not fully fulfilled. I will describe the session and what I thought about it.

First of all, I really liked the approach that compared functional and non-functional testing, the What and the How. I have been doing functional testing and some non-functional testing, but Nikolai was able to bring out the essential in both testing approaches. I did however disagree with the test-case based approach to the functional testing, but I did not challenge him for that. I was there to focus on the NFT.

Nikolai used simple but effective examples to describe the non functional testing as a complementary approach to the functional testing. “User MUST be able to sign in (functional), and Sign-in time should be equal or less than 5 seconds (non-functional)”. So basically the NFT can bring more flavor to the functional testing and thus make it more efficient and deeper.

One aspect of non-functional testing is to keep things simple. There is a vast jungle of non-functional requirements out there, so we should focus on the ones that bring most value to the product and to the users. At Skype (Nikolai Pavlov is a Mobile QE Lead at Skype) their main focus was on 5 things: application size, start-Up time, responsiveness, memory footprint, and battery life. Some of those things had never even occurred to me as a non-functional requirement, so I was really looking forward to the hands-on-part of the workshop.

Nikolai then presented ways to test those different aspects of the product. They had really cool tools and methods to test and measure the NFR. I was really impressed by the battery-life testing tool. Also the aspect of integrating start-up time measuring to you automate tests was a really cool idea. However after the theory part was finished, the workshop ended. “No hands-on?” the inner voice in me screamed.

The workshop would have been more efficient if Nikolai would have cut out some of the theory and would have given something to tests during/after his theory part so we attendant would have brought more substance to the workshop. I’m still thinking how I could test my Android-phone the “non-functional way”, but I have no place to start as we had no hands-on testing in the workshop. I wish Nikolai will do some adjustments to the presentation and offer some hands-on testing in the next session. But all things put together, the track was inspiring and got me thinking different ways to implement to my daily work. Maybe I could focus my effort on the memory consumption, threads and stuff… What about the conformance? Can I use fuzzing to make the apps crash? ...

The Adventures of Exploratory Monkey

This is a summary of Sami Söderblom’s track about exploratory testing and finding the inner adventurer. He presented this at the Nordic Testing Days 2012 in Tallinn. I found this track one of the most interesting ones as this cuts to the core of testing by stripping the excess and unnecessary activities away and focuses on the essence of truly good software testing.

Before I start ripping the track apart (well actually quite the opposite), I want to mention that this is somewhat in line with James Whitaker’s views* about exploring the software. After watching a video about large scale exploratory testing** I saw great many similarities in Sami’s track. I wonder how much of an influence has James Whitaker been to this track…

Sami had some great stuff going on in the track about exploratory testing. Great to me, because he tackled some of the issues that I’m struggling with when I try to explain exploratory testing to people new to the concept. The loop of Design, Execution, Observe, Learn and Adapting was the kind of thing people should be familiar with. By removing some action from the testing, we will result in a cumbersome approach that hinders the effective testing that we are trying to achieve with exploring. Also it was great thing for others so that they can get a firm grasp of what exploratory testing is in its most fundamental level.  Sami had great examples using the Battleship metaphor, the Great explorers of different domains, and the way of surviving with as little documentation as possible.

We have all played battleship game, right? B4 and c8, hit, destroyed, and so on, you know the drill. Some people think testing like the Battleship game and rely that if we cover all the grids by test cases we will find bugs. But the bugs don’t play by the rules! There are ships outside the grid! There are ducks instead of ships! There are mines within the grid! Why should we play by the rules? Why not do everything it takes to find all the ships? Use hammers instead of “C4” (or even real C4). Use a vacuum cleaner to suck in all the ships. Use an umbrella over the mine so you won’t break the environment by accident (and thus hinder your testing effort). Use whatever tools and techniques you have to get the job done! Cheating is allowed in software testing!

Sami introduced a few cool people from real world and from fictional world as his favorite explorers. Dr. House is an explorer, there’s no doubt about that. He tries all things he knows and utilizes all possible resources, techniques, tools, medicine, etc. to get the problem solved. He might try things that don’t necessarily cure the illness but that uncover more information about the problem. Sami also told about Sherlock Holmes, Gordon Ramsay and Jeremy Clarkson, and made cool examples how they solve problems or challenging tasks. This made me want to read more about this Sherlock Holmes, and I already started recording the House tv-series from the 1st season.

He talked also about reporting bugs. This is quite contradicting to the track made by Ainars Galvans, as Sami sees that everything must be reported as they might be a manifestation of something more severe. The root-cause is what matters! So if we don't know the root-cause, we might miss a terrible bug by thinking it's trivial.

All in all Sami managed to bring new information about the world of exploratory testing to the audience and about the building blocks of quality. I’m happy that he showed the Venn-diagram that had "testing" and "verification" in separate circles AND emphasizing that both of them need the other to be at their most effective.

I recommend reading Sami’s blog about exploratory testing. He has great thoughts about testing and quality. He will eventually write report of the conference, so follow the blog and hear his thoughts about his own performance and of others.

"All testing is exploratory!" -Sami Söderblom

*) “Exploratory Software Testing: Tips, Tricks, Tours, and Techniques to Guide Test Design” by James A. Whitaker 
**) YouTube video about large scale exploratory testing by James Whitaker

The branch that you’re sitting on is getting cut

The third track at Nordic Testing Days 2012 was a multiple case study or an experiment report from Kristjan Karmo from ASA Quality Services Oü. There were 5 cases that Kristjan introduced in his track, all real scenarios with some data changed to protect the privacy (also some figures exaggerated to make them more dramatic). They were all entertaining but each of them had a punch-line that summarized the case to a lesson learned. In the end all projects had one fundamental flaw that affected the results. Read through and see what it was.

The process is a tool - a fool with a tool is still a fool. Communication is the key!

The first case study one was a company called Wolf Inc., a company that wanted to change name and become Strawberry Inc. The task was supposed to simple: Find-and- replace and go to production. Simple, fast, infallible. No testing required. What happened was exactly the opposite. “Wolf” as a word changes in different situations: wolves, wolf’s, and even more if in a different language. Also all the logos, brochures, documents, templates, etc. were affected. All this was to be avoided by asking questions about the task, by listening to the people making the change. It was all about communication.

The second case was a “CRM on steroids” by a company called Fields & Co. There were lots of people involved, lots of locations, lots of documentation and interpretation thereof. The most catastrophic mistake was a misinterpretation of a date. The date was THOUGHT to be the “ready for testing” –date but it turned out to be the “ready for LIVE” –date. 1 hour session of critical exploratory testing on the product by all parties involved. Group effort and commitment to the end product saved the day this time.

The third case was a company called Forever Ltd. where testers were doing the best they could and producing bug reports. These reports ended up on the developer’s desk and were returned with a stamp “works on my machine”. The system consisted of multiple integrations of different products so the all the contractors said that the problem was on the other contractor’s court. The solution was to meet with all the parties and talk it through. People were negotiating about their defects and issues and then he proper stakeholder would claim the bug and fix it.

The fourth case was Pepper plc., a company that had a testing team of end-users. When the testers were introduced to the project, all testing done by the developers stopped. The problem was solved introducing a few experienced experts to the team and mandating the train testers to make more effective bug reports and testing was done using the exploratory approach. The lesson here would have been the experience and the structure the testers brought to the team.

The fifth case was a company called BadCom Corporation. The requirements for performance had been made early because the system was performance dependent. The product that came to testing was 200 times slower than the requirements stated. So the team did an effort to increase the performance and they managed to get it to be 10 times faster. That was still not enough, as the system still took lots of time to respond. An assumption was made that the 20 times slower would still be good enough. The reality was far worse at the performance rendered the system inoperable and useless.

So here’s what I think:

• In the Wolf-case the work group didn’t know what they were doing or they were blindly taking orders. No-one questioned the orders by looking at the id-tag hanging from their necks, etc. So questions that needed to be asked were not asked, thus creating a situation where a consultant firm was called to douse the fire.
• In the second case there were project management and consensus problems. Floor-level did not know enough about the scheduling thus making interpretations about them. What lacked was communication between stakeholders. The last minute exploratory testing show might have been a confidence boost, but feels like a show more than productive effort. Don’t know, but feels like it.
• Third case is a typical multi contractor problem. Everyone’s interested on part made by them and will not think of others because they’re not paid to think about them. The more bugs the other contractor has, the better we look. So there was no big picture or team in this case and everybody tried to cover their own asses. Again communication and challenging could have done the trick and much earlier.
• The fourth case was a basic bottleneck-situation, where the developers thought that they can now concentrate on their job and leave the testing to the test team. The problem was solved by amping up the test team, which was a good solution, but not what I’d had done. I would have talked to the developers and convinced them to do more testing and would have shown them the results if they did the testing.
• The fifth one was burned even though a consulting agency did their best to get things better. Here the issue was realized so late in the product life cycle that the fix was almost impossible. The performance would have been the top priority regarding testing if it was a performance ciritcal system.

So, all in all, all cases were doomed because of the lack of conversation and the critical thinking of a professional tester. By outsourcing testing you outsource thinking, and might lead into these situations all over again. So think before ending in a situation where the branch that you’re sitting on is getting cut. Use challenging, questioning, feedback, workshops, to raise communication in within the project.

Communication is the key to success.

Making a requirements pie

This is a summary of the track by Ainars Galvans at Nordic Testing Days 2012 and how I viewed the subject. The topic was “Care for quality, not for bugs! Do you know the difference?“.

I met Ainars the day before the conference at the hotel and we discussed about bugs and quality. We are on the same line with the overall quality but I disagree with some points he made. So here is what happened and how I saw it and some thought about challenging the claims.

“The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.” –Cem Kaner

See the difference?

True and true. Ainars had great thoughts about increasing the value of the bugs. By focusing on the bugs that had a story that compelled to the developer (and the triage of people deciding which to fix), you will get more out of the bugs that you find. When I think of myself as a tester, I tend to get as much information about the product and thus uncover as much bugs as I can. I know that by doing so I learn about the product, the risks, the critical areas, the things that people making decisions value the most. When Ainars said “don’t report the bugs that aren’t getting fixed”, I was shocked. Am I doing it the wrong way? I may have a huge amount of rejected bugs, but it’s my way of learning. What if I uncover some minor looking manifestation of a bug that turns out to be a showstopper? I don’t know if I don’t ask and communicate.

Usually when you start communicating with the developer about a bug that seems like a show stopper, he will immediately see that it should be fixed immediately. If he can pinpoint the problem, you won’t need to make a bug report if the problem is sufficiently communicated onwards. So this is where I disagree with Ainars; I will make bug reports but I get the important ones fixed.

Ainars had great thoughts about prioritizing testing. He said that he’s not going to spend his time on testing something he KNOWS is going to pass, so he tests something he doesn’t know that will pass. He saves time doing so and can focus on the important areas of testing. So you should always do the most important testing at the every moment.

How do you know what is important? Ask! I ask if there is some area that is valued by the product owner and I focus on that. Then I ask what the next valuable thing is and focus on that. By time I get some testing done I can start formulate my own view about the importance of different things. So Ainars had a good thing going on there, as he knew the product so good he was able to guide his own testing prioritization using his knowledge of the product. The knowledge acquired by learning and testing the product.

Picking up requirements.

Ainars also introduced a cool way to do exploratory testing. He called it “Requirements driven exploratory testing”. They were almost like test cases but more like missions or topics, which were then tested. They were measured using traffic lights: Green (good enough testing AND good enough quality),Yellow (not started) and Red (not good enough testing OR not good enough quality). I challenged the concept as the testing team was focusing on only documented requirements, and I still do! There is a need to test also things that the client takes for granted and things that are discussed but not documented. I also challenged the fact that requirements are usually ambiguous, and that “requirements are not items to be gathered”. They're not like apples that you can get into your basket and bake a testing pie out of them. Ainars could clarify that if he can. From the presentation I did not get an answer to that.

As a summary, I got a lot out of the track as it provoked me to think about how I do my testing, but also I got a chance to challenge the speaker. I feel that Ainars has a lot to say about this and I will be checking his blog as often as I can. I hope he has a way to address my concerns about the requirements and “not-important” bugs.