Bringing flavor to the functional testing

"Yo! Let's bring som flava to tha functional testin'!"

I had made a plan to attend at least one workshop on the first day. So instead of going to the Beta-testing workshop I decided to build my skills in the NFT-side. To me, non-functional testing is performance testing, security testing, conformance testing, etc. Things that measure the quality criteria of the product (how fast it is, how secure is it, etc.). As I am a mostly a black box tester, I usually try to focus on the behavior of the functionalities and what the product does. This doesn’t exclude the other approaches, but that’s just my main focus.

The workshop was Non-functional testing on mobile devices by Nikolai Pavlov. I was thrilled that I would be able to test mobile devices with an expert, so I was prepared to be blown away. Sadly, my expectations were not fully fulfilled. I will describe the session and what I thought about it.

First of all, I really liked the approach that compared functional and non-functional testing, the What and the How. I have been doing functional testing and some non-functional testing, but Nikolai was able to bring out the essential in both testing approaches. I did however disagree with the test-case based approach to the functional testing, but I did not challenge him for that. I was there to focus on the NFT.

Nikolai used simple but effective examples to describe the non functional testing as a complementary approach to the functional testing. “User MUST be able to sign in (functional), and Sign-in time should be equal or less than 5 seconds (non-functional)”. So basically the NFT can bring more flavor to the functional testing and thus make it more efficient and deeper.

One aspect of non-functional testing is to keep things simple. There is a vast jungle of non-functional requirements out there, so we should focus on the ones that bring most value to the product and to the users. At Skype (Nikolai Pavlov is a Mobile QE Lead at Skype) their main focus was on 5 things: application size, start-Up time, responsiveness, memory footprint, and battery life. Some of those things had never even occurred to me as a non-functional requirement, so I was really looking forward to the hands-on-part of the workshop.

Nikolai then presented ways to test those different aspects of the product. They had really cool tools and methods to test and measure the NFR. I was really impressed by the battery-life testing tool. Also the aspect of integrating start-up time measuring to you automate tests was a really cool idea. However after the theory part was finished, the workshop ended. “No hands-on?” the inner voice in me screamed.

The workshop would have been more efficient if Nikolai would have cut out some of the theory and would have given something to tests during/after his theory part so we attendant would have brought more substance to the workshop. I’m still thinking how I could test my Android-phone the “non-functional way”, but I have no place to start as we had no hands-on testing in the workshop. I wish Nikolai will do some adjustments to the presentation and offer some hands-on testing in the next session. But all things put together, the track was inspiring and got me thinking different ways to implement to my daily work. Maybe I could focus my effort on the memory consumption, threads and stuff… What about the conformance? Can I use fuzzing to make the apps crash? ...