The branch that you’re sitting on is getting cut

The third track at Nordic Testing Days 2012 was a multiple case study or an experiment report from Kristjan Karmo from ASA Quality Services Oü. There were 5 cases that Kristjan introduced in his track, all real scenarios with some data changed to protect the privacy (also some figures exaggerated to make them more dramatic). They were all entertaining but each of them had a punch-line that summarized the case to a lesson learned. In the end all projects had one fundamental flaw that affected the results. Read through and see what it was.

The process is a tool - a fool with a tool is still a fool. Communication is the key!

The first case study one was a company called Wolf Inc., a company that wanted to change name and become Strawberry Inc. The task was supposed to simple: Find-and- replace and go to production. Simple, fast, infallible. No testing required. What happened was exactly the opposite. “Wolf” as a word changes in different situations: wolves, wolf’s, and even more if in a different language. Also all the logos, brochures, documents, templates, etc. were affected. All this was to be avoided by asking questions about the task, by listening to the people making the change. It was all about communication.

The second case was a “CRM on steroids” by a company called Fields & Co. There were lots of people involved, lots of locations, lots of documentation and interpretation thereof. The most catastrophic mistake was a misinterpretation of a date. The date was THOUGHT to be the “ready for testing” –date but it turned out to be the “ready for LIVE” –date. 1 hour session of critical exploratory testing on the product by all parties involved. Group effort and commitment to the end product saved the day this time.

The third case was a company called Forever Ltd. where testers were doing the best they could and producing bug reports. These reports ended up on the developer’s desk and were returned with a stamp “works on my machine”. The system consisted of multiple integrations of different products so the all the contractors said that the problem was on the other contractor’s court. The solution was to meet with all the parties and talk it through. People were negotiating about their defects and issues and then he proper stakeholder would claim the bug and fix it.

The fourth case was Pepper plc., a company that had a testing team of end-users. When the testers were introduced to the project, all testing done by the developers stopped. The problem was solved introducing a few experienced experts to the team and mandating the train testers to make more effective bug reports and testing was done using the exploratory approach. The lesson here would have been the experience and the structure the testers brought to the team.

The fifth case was a company called BadCom Corporation. The requirements for performance had been made early because the system was performance dependent. The product that came to testing was 200 times slower than the requirements stated. So the team did an effort to increase the performance and they managed to get it to be 10 times faster. That was still not enough, as the system still took lots of time to respond. An assumption was made that the 20 times slower would still be good enough. The reality was far worse at the performance rendered the system inoperable and useless.

So here’s what I think:

• In the Wolf-case the work group didn’t know what they were doing or they were blindly taking orders. No-one questioned the orders by looking at the id-tag hanging from their necks, etc. So questions that needed to be asked were not asked, thus creating a situation where a consultant firm was called to douse the fire.
• In the second case there were project management and consensus problems. Floor-level did not know enough about the scheduling thus making interpretations about them. What lacked was communication between stakeholders. The last minute exploratory testing show might have been a confidence boost, but feels like a show more than productive effort. Don’t know, but feels like it.
• Third case is a typical multi contractor problem. Everyone’s interested on part made by them and will not think of others because they’re not paid to think about them. The more bugs the other contractor has, the better we look. So there was no big picture or team in this case and everybody tried to cover their own asses. Again communication and challenging could have done the trick and much earlier.
• The fourth case was a basic bottleneck-situation, where the developers thought that they can now concentrate on their job and leave the testing to the test team. The problem was solved by amping up the test team, which was a good solution, but not what I’d had done. I would have talked to the developers and convinced them to do more testing and would have shown them the results if they did the testing.
• The fifth one was burned even though a consulting agency did their best to get things better. Here the issue was realized so late in the product life cycle that the fix was almost impossible. The performance would have been the top priority regarding testing if it was a performance ciritcal system.

So, all in all, all cases were doomed because of the lack of conversation and the critical thinking of a professional tester. By outsourcing testing you outsource thinking, and might lead into these situations all over again. So think before ending in a situation where the branch that you’re sitting on is getting cut. Use challenging, questioning, feedback, workshops, to raise communication in within the project.

Communication is the key to success.

Making a requirements pie

This is a summary of the track by Ainars Galvans at Nordic Testing Days 2012 and how I viewed the subject. The topic was “Care for quality, not for bugs! Do you know the difference?“.

I met Ainars the day before the conference at the hotel and we discussed about bugs and quality. We are on the same line with the overall quality but I disagree with some points he made. So here is what happened and how I saw it and some thought about challenging the claims.

“The best tester isn’t the one who finds the most bugs or embarrasses the most programmers. The best tester is the one who gets the most bugs fixed.” –Cem Kaner

See the difference?

True and true. Ainars had great thoughts about increasing the value of the bugs. By focusing on the bugs that had a story that compelled to the developer (and the triage of people deciding which to fix), you will get more out of the bugs that you find. When I think of myself as a tester, I tend to get as much information about the product and thus uncover as much bugs as I can. I know that by doing so I learn about the product, the risks, the critical areas, the things that people making decisions value the most. When Ainars said “don’t report the bugs that aren’t getting fixed”, I was shocked. Am I doing it the wrong way? I may have a huge amount of rejected bugs, but it’s my way of learning. What if I uncover some minor looking manifestation of a bug that turns out to be a showstopper? I don’t know if I don’t ask and communicate.

Usually when you start communicating with the developer about a bug that seems like a show stopper, he will immediately see that it should be fixed immediately. If he can pinpoint the problem, you won’t need to make a bug report if the problem is sufficiently communicated onwards. So this is where I disagree with Ainars; I will make bug reports but I get the important ones fixed.

Ainars had great thoughts about prioritizing testing. He said that he’s not going to spend his time on testing something he KNOWS is going to pass, so he tests something he doesn’t know that will pass. He saves time doing so and can focus on the important areas of testing. So you should always do the most important testing at the every moment.

How do you know what is important? Ask! I ask if there is some area that is valued by the product owner and I focus on that. Then I ask what the next valuable thing is and focus on that. By time I get some testing done I can start formulate my own view about the importance of different things. So Ainars had a good thing going on there, as he knew the product so good he was able to guide his own testing prioritization using his knowledge of the product. The knowledge acquired by learning and testing the product.

Picking up requirements.

Ainars also introduced a cool way to do exploratory testing. He called it “Requirements driven exploratory testing”. They were almost like test cases but more like missions or topics, which were then tested. They were measured using traffic lights: Green (good enough testing AND good enough quality),Yellow (not started) and Red (not good enough testing OR not good enough quality). I challenged the concept as the testing team was focusing on only documented requirements, and I still do! There is a need to test also things that the client takes for granted and things that are discussed but not documented. I also challenged the fact that requirements are usually ambiguous, and that “requirements are not items to be gathered”. They're not like apples that you can get into your basket and bake a testing pie out of them. Ainars could clarify that if he can. From the presentation I did not get an answer to that.

As a summary, I got a lot out of the track as it provoked me to think about how I do my testing, but also I got a chance to challenge the speaker. I feel that Ainars has a lot to say about this and I will be checking his blog as often as I can. I hope he has a way to address my concerns about the requirements and “not-important” bugs.

How to cope with few end-user-testers in a testing project?

This is a summary about the 2nd track of the first day at Nordic Testing Days 2012, how I interpreted the process presented and what learning from that track. The track focused on building skills for the Business end-users to be able to test the system under test properly and as quickly as possible.

The track was performed by Maili Markvardt and was called “Testing: saved by the end-users”. I wasn’t expecting much out of it (I wonder why? Was I biased somehow?), but what eventually went down was very inspiring. Personally I have been in situations where the testing budget is so small that all guns must be brought to bearing. I thought this was one of those rants that “we didn’t have enough time” or “the client was behaving like child” kind of situations. Luckily it was not.



She started the track by presenting the situation: a total makeover of all information systems in a company, 7 large systems with ~90 interfaces, integrated in a “big bang” fashion. Immediately she got my attention, I’d been in a situation like that in my previous company (although not that bad, but similar on many levels). The system was done by many contractors and had about ~250 people involved. No professional testers and 1 test manager, Maili.

The problem was being solved by giving her a bunch of business people – people with no testing skills apart from some IT-skills acquired from previous life. So there was the lack of skills that needed to be taken care of.

How she handled the situation? Well, she trained the testers. The truth is that you can’t train people to become testers without them having enough time and motivation to be trained, or it can be very difficult. What she did was tackle the source of the problem: get a written mandate from the manager of these business people to be available as tester the promised amount of time. And holding them to that promise was (if I interpreted it correctly) an uphill battle. The business people obviously had their own work to take care of.

The would-be testers had little to none experience in testing. What Maili trained to them was the basic testing skills; mindset, test analysis and bug reporting. The approach was test case driven and governed by the test manager. Some challenges started to immerge at early points of testing; the business process was still unclear and under constant change, documentation was too complex, test cases too generalized, and people lacked the base knowledge to execute the test cases.

This is where I started thinking the possibilities of doing the testing the exploratory way. Could they have utilized the Rapid software testing approach to maximize learning and be more responsive to change? As the testers face these kinds of problems, strict and over-documented processes tend to fail due to lack of flexibility. I immediately reflected this to my own work and saw that we had the advantage to use RST to our advantage and benefit from the learning. Sure we also had problems, but the way we communicated with the stakeholders enabled us to make our testing more “to the point” and we uncovered lots of information that the stakeholders didn’t know they needed.

The ugly world of constant change took its toll on testing. Lots of bugs were found and not all of them were reporting real issues. The bug tracking process required the test manager to be a filter that would channel the bugs to correct stakeholders. Due to ambiguous documentation and knowledge about the interfaces (the testing for some of them was postponed to very late stages of testing), testing was difficult and the systems suffered from poor quality, and thus led into postponing the “Go LIVE” three times. The testing might have been the best quality available in that situation, but the amount of work forced the prioritization to ignore some areas until the very last days of testing.

All this put aside, Maili was able to create a process which involved described the process of training business people as testers. The process feels very sturdy and practical, but simple enough to be implemented by all test managers trying to cope with little resources. I hope people got as much out of the track as I did even though I didn’t expect much. Eventually I learned a lot about using what resources you have and to make most out of them. It’s the context driven way, I think: “Throw in whatever you have and the kitchen sink” to get the work done. To my view, Maili succeeded in the task that was very difficult to begin with.

Kudos!

The tired travelling man (with no business-cards to share at the event)

The last day came and went at the Nordic Testing Days 2012 -conference  and feeling a bit sentimental. I feel like I left a whole bunch of new friends behind the pond: Raivo Päts, Kaspar Loog and Kristjan Karmo to name a few. The last day as a milestone for me also as I held my first conference talk which was a workshop about Heuristic testing using Mind maps. I have video and slides about the workshop, and also some pictures when I get my hands on them. Main thing was that I had a blast and I got the impression that at least some people got something out of the workshop too.

So today didn't go as planned but I feel I made the right choices overall. I saw Mr. Rex Black talk about Quality management (which I will be doing a minor post later), I held my exhausting workshop (which I will analyse with more depth and because I like to talk about myself ;) ) and a long hands-on exploratory testing workshop by Mart Toom (a great guy with a knack for ET - I'll write a good analysis fattened with constructive feedback). I was planning to take the shorter workshop and attend the tracks, but I felt that I would get/share (the experience) more if I was able to participate in the workshop.

I have all my notes in mind maps. I will try to get them online as soon as possible for all the tracks. Some are less extensive than the others but I will write posts of the most interesting ones (not to say the others were not interesting but I may not have enough to share about those.

So, tomorrow I will be heading to Technical University of Tampere to join the Testauspäivät ("Testing days") and to have a lightning talk about mind maps and exploratory testing. Let's see how that event turns out... ;) Hopefully even more friends and peers to talk about testing with! :D

- Peksi, the tired travelling man with no business-cards to share at the event

The morning after (The second day of NTD2012)

Rise and shine!

10 minutes until the event starts so I need to be quick about my writing.

I have the workshop in the morning so I'm a bit nervous right now, but that will pass after I get my slides up and the people to testing.

The Pekka's Choice for the Day 2 is the following (goes with the same formula as the yesterday's choises):

• Keynote: "Test Management and Quality Management" - Rex Black (USA)
• Workshop - Testing Real-Time IP Video Communication at Skype by Pierre Gronlier & Antonio Campos Domínguez & Marios Mpasoukos & Oksana Dementsova from Skype
• "Testing in Agile methodologies – easier or more difficult?" by Lucjan Stapp (POL) Warsaw University of Technology
• Keynote: Software stores our wisdom by Jaan Priisalu (EST)

Now I got to run to catch the Keynote by Rex Black! I'll be back!

- Peksi

Testing at Skype -track by Tiit Paananen

Lunch == Toss the food into your voice hole and write summaries about the conference talks.

First was about the testing done at Skype by Tiit Paananen. The talk was inspiring at the least and below are my notes on that session. :D

One thing that I liked about the show was the concept of testing maps. Testing maps are developed to increase visualization of the testing process. People can choose the media that they present it but they must visualize the effort that they will put into the testing.

I will update on the other tracks as soon as I can, but the time between the tracks is so short and I also have to confer – that what this is about!

- Peksi, a guy willing to borrow an iPad as this fragging laptop is so hot it makes me sweat like a pig!

The wings of procrastination

The morning came and went in the wings of procrastination. ”If I don’t go to gym I can sleep for 20 minutes longer. (Snooze!)” Now that I have rested and “broken fast” (I got that one from Game of Thrones), I can enjoy the conference fully.

Soon I will get to see the Skype keynote and see how things are done here in Tallinn. It has come to my attention that this is the first Nordic testing days –conference. The facilitation has been awesome here and hopefully the content is as good as the facilities. Stay tuned 'cause I'll be updating the  blog throughout the session.

-         - Pekka at 8:55 waiting for the conference to start.