You gotta fight for your right to test!

I am terribly sorry for the soon to be rant and biased output that is going to happen. I have did this once, but the conversation resulting from that was rather pleasant and constructive. This is a comment / response / rant about a blog page I stumbled upon today. You can find the original post here. The page may have been outdated since it was created in 2010 but it seems that it was updated half a year ago.

My first though was “not another test case” when I read the article. I’m currently trying to figure out what I will write to my next column on the “Testaus ja Laatu” –magazine, and since the theme is juxtaposing, I thought I’d try to write some blog-stuff first. One of the topics could be “test cases vs. no test cases” and I shy away from that strict way of thinking. Exploratory testing, to me, is utilizing all the available tools and gimmicks to get to the best results. Black and white –world vision narrows too much my take on testing.

Having said that, I will use my recently found taste to logic to point out what I found could be wrong in the article. I do not know the state of mind in which the article was written or the context in which it is supposed to be fitted, so I will project it to my own workplace where that is necessary.

You are warned…

The first sentence goes like this: “A significant factor in a successful testing effort is how well the tests are written.”  I have no idea how much significant is this case. I would guess that it is either the largest part or the one after that. When I do testing, the significance of the test cases are miniscule, and I tend to do successful testing. Not all the time, but I know a certain cases where properly written test cases did not contribute to the successful testing effort. Also I don’t exactly know what qualifies as successful testing effort. It could mean having run all the tests within the schedule (which doesn’t quality for successful for many reasons), having written all the test cases (do I need to say more about this), the product is shipped to end user (lots of products have been shipped to customers and later fixed due to poor quality), etc. So I would say that well written test cases /may/ be a factor in the successful testing effort just as well as poorly or not-at-all-written test cases.

“They have to be effective in verifying that approved requirements have been met AND the test cases themselves are understandable so the tester can run the test as intended by the test author.” 

So the test cases have to be effective in verifying requirements. Granted. Do the /have to be/ written. No. Even poorly written test cases can be effective at verifying requirements. I think the ability to verify something comes not from the writing of a test case but from the skill of a tester. If a tester is skilled there may not be need for any written test cases to get the job done. “--approved requirements have been met --” So anything that is not written to requirements doesn’t get tested? We also have expectations about the product that don’t really qualify as requirements. Testers still need be aware of anything that might threaten the quality of the product.

“The 3C’s: Clear, Concise, Complete. Test Cases are not open to multiple interpretations by qualified testers. Ambiguous or nonspecific Test Cases are difficult to manage.” 

What if we do not know about the product enough at the time we write the test case? Should we wait until the product is finished and then write the cases? Isn’t that just huge waste of time and money? And when it comes to managing test cases, I think lines of text are the easiest to manage, it’s the people that might require managing. It is true that estimating coverage and depth may be difficult if the test case is ambiguous. I also think that it is difficult to estimate those with good and precise test cases. People are best to estimate the depth of their testing and the confidence in their testing. ASK IF SOMETHING IS AMBIGUOUS. -> Fight ambiguity with openness and communication

“Test Cases are easily understood.  They provide just enough detail to enable testers familiar with the functional area under test to run the test case. Note:  Detailed click by click steps are only useful for automated tests.” 

This is something that I agree! Test case can be easily understood even if it’s like this: “Play around with the product and describe the person next to you what it does.” It is both easily to understand and you already have enough familiarity because the base requirement for that is none! This could be the baseline for any testing ever to be done at any context. FAMILIARISE YOURSELF WITH THE PRODUCT.-> Fight ignorance with eagerness.

“Test cases include setup requirements (environment & data), expected results, any dependencies on other tests and tracing to the requirements they validate. Are traced to the results of each test run, including defects (bugs) discovered, test platform run on, software build version used.” 

For the data and expected results, I recommend all you read Michel Bolton’s and James Bach’s conversation and decide yourself if the test case is complete with expected results. It is necessary to document stuff that mentioned in the text so to avoid unnecessary overlapping. But traceability to test cases? Is that possible for bugs that are found during the test case execution but outside the intended observation area? BE PREPARED FOR THE UNEXPETED RESULTS WITH UNEXPETED INPUTS. -> Fight patters with chaos and vice versa.

“Measurable: For each test case, there must be a way to objectively measure that the test case has either passed or failed. All test cases must be linked to the requirements that they verify to enable impact of changes made to the requirements or solution designs to be identified (measured).” 

OK. Let’s say that the test is executed perfectly and the result is exactly as expected. A minute after that the computer crashes. Does that constitute as failed? How much do we actually know about the product to say that a test has passed or finished. And if we want to find bugs, isn’t the test passed only if it finds bugs? I think test only fails if you don’t run the test, because it didn’t test anything. TEST IS ONLY FAILED IF IT IS NOT RUN. -> Fight unnecessary documentation with rightly timed planning. I.e. keep the time between planning and doing minimum. Preferably do them at the same time.

“The test case must have been approved, prior to being run, by the key stakeholders of the requirements that the test case verifies. Any changes made to test cases caused by requirements or solution designs changes must also be approved.” 

There is ABSOLUTELY NO POINT IN THIS! Why the hell do we need approving for our testing? Only thing that requires approving is the results of our testing. If the stakeholders do not trust us to test the software, we could record everything we do and ask them to audit the test material. I would think they are happier to audit actual results than worthless, constantly changing trivial documentation. IF WE NEED EVERY TEST APPROVED, WE LOSE MONEY AND TEST COVERAGE. -> Fight über control with proper* documentation.

“Realistic test cases DO: Verify how the product will be used or could be misused, eg positive and negative tests. Verify functions and features that implement approved product requirements. Can be run using the platforms and software configurations available in the test environment.
Realistic test cases DO NOT: Verify out of scope functions and features. Verify unapproved product requirements.”

I agree with the first sentence – a good test could test how the product could be used/misused. Also how it should/will be used. How much does the product actually differ from what the customer actually wants? I also agree with the second sentence, but it should be broader. It should test the features, platforms, integrations, data integrity, security, performance, etc. I don’t understand the third sentence for if we don’t have the proper environment or setup, we should acquire them in time. If we do have a scope (referring to the fourth sentence) then I agree that we should not spend too much time on out of scope elements. They could be mutated or broken due to changes made somewhere else so regression/smoke testing should be implemented to out of scope areas. If the fifth sentence I don’t understand what is meant by “unapproved”. Unapproved by who? By the testers, client, developers, managers? It really depends, so making a claim like that is just nonsense. A REALISTIC TESTCASE IS REALISTIC BUT FLEXIBLE. -> Fight rigid descriptions with autonomy, critical thinking and challenging.

“Test Cases must be able to be successfully completed within the project scope and schedule.” 

There is no point on writing test cases that are not run. But “must”? And why do they have to be “successfully run” and is a failed test a “successfully run” test? If a test finds three hundred bugs, is it successfully run if it doesn’t reach the expected result? WE CAN ONLY RUN AS MUCH TEST CASES. -> Fight the quantification of test cases with the amount of time spent doing testing. Count time, not test cases.



To summarize, I think the whole concept of SMART test cases is wrong excluding few things that I agreed with. I also encourage to keep the writing of test cases in minimum and the amount of testing done in maximum. Use the time wisely and appropriately! But if you do, you should consider these instead of the SMART way:

• Fight ambiguity with openness and communication
• Fight ignorance with eagerness.
• Fight patters with chaos and vice versa.
• Fight unnecessary documentation with rightly timed planning. I.e. keep the time between planning and doing minimum. Preferably do them at the same time.
• Fight über control with proper documentation.
• Fight rigid descriptions with autonomy, critical thinking and challenging.
• Fight the quantification of test cases with the amount of time spent doing testing. Count time, not test cases.

I will leave you to this. I promise I will be back with more about test cases (if I have the time). I’m not saying “don’t write test cases”, but use your head! It is not smart (bun intended) to follow some rigid procedure for all context, but to adapt to situation and to make most of the time you have to test.

- Peksi

* Proper: Appropriate as automated as possible. Including video recording, automated logs, scans from scribbles, session sheets, statements, etc. What is required to get enough information to the decision makers without hindering the job of a tester.

Responce to "Oy you tester!" By ElizaF

I was strolling casually in the Twitter-land and I saw this link. It was a blog written by ElizaF (maybe that's her real name, but who knows). She had a post that was a bombardment of questions. It got me thinking and suddenly I started to write up the answers to those questions. Here they are:

What do you think when you see people talking about spending three weeks writing tightly worded test cases before they start testing a product?

• What would you think of a person refusing to do it? It there is pressure from organization is it worth your employment to refuse? Testing based on strict test cases forces you to forgo your creativity (unless exploration is allowed or better yet encouraged). But it’s not what I think when I see that kind of a thing (and boy do I see that!) but what I do. I speak up and try to my best ability steer the testing process towards enoughness, quickness, agility, exploration -> TESTING! Away from checking, towards testing.

What do you say when you read of people describing their work environments as Agile with elements of Waterfall?

• It can be agile-ish, but there no agility in waterfall! This just means that a PM or some high level dude doesn’t understand testing to any extent. I would stamp “TEST EARLY” (better yet: “TEST NOW!”) onto his/hers (from now on I speak genderless when I say “he”) forehead so he will remember it every morning he brushes his teeth. Even the most top level people should be aware of what testing is and why is it so important!

What do you comment on blogs where people write of successfully using a commercial tool time and time again project after project?

• Good question. I almost answered “I don’t.” but they may be posting things like that due to insecurity of trying things that FIT THE CONTEXT. There may be reasons behind the choise of tools that are unclear to me and they may be good ones. I however have not stumbled upon such blogs. Any hints?

How do you reply when people ask  on forums about where to get the answers to the ISEB exams?

• I say: “You don’t need the answers. Better yet you don’t need the exam!” And to be realistic the questions in ISEB tests are mostly obvious and you can answer to them without any knowledge about testing -> you just need to memorize a book. And that’s it. You will almost never use the information in the cert for anything (except you know to stay away from people that require you to). Instead of cert you need knowledge about your domain, procedures, tools, practices and make them work for you. Professionally (or CV-wise) you could take the test but relying it to bring you competence over doing testing is where the certifying goes wrong.

What impression do you have of other testers who profess to having no knowledge of who the leading speakers and thinkers in the world of QA are today?

• I know some. Not all. I know those who I think matter in the field.

How do you engage with other members of your profession with whom you disagree with or whose outlook you consider yourself to have evolved beyond years ago?

• This’ a tricky question. Be it the language barrier or whatever but this is how I understand this: “What if you think some other way to approach to something is better than the one who is suggesting an approach?” I need to know the reason why they support the chosen approach and try to understand the meaning behind it. If a valid reason is presented then it might be the best approach for the context. Or there may be some room to improve on which case the change must be found.

How do you write about QA people saying they took a long time to replicate and report a bug of which you have no knowledge but the amount of hours mentioned sound like a "long time" to you?

• Instrument (run in debugger, record GUI, kernel, etc) beforehand so less time consuming replication is needed. I would find out the context on which the system has been tested and bug found. If customer found the bug then we need to know the environment and such. We cannot say something took a long time unless we know how long it should take (mostly we don’t).

Do you talk to people who are not in perfect alignment with your way of thinking like they are below you? Do you think that evolution as tester can only be achieved by going on a certain course and anyone not following that path is below you?

• No reason to rate people by their current way of thinking. What should be done is rate them by the way they have improved their thinking. Some people do box themselves and never change POV. These people should be looked down upon. The self improvement may be just as effective as a course but sometimes preachy people make your heart turn into a certain direction which may seem the right direction for a while. Look into facts and whether the thing you are doing fits the context you’re in.

When I got to the finish of the blog I saw that this was a trap. "What do you think of yourself for thinking things like that?" I started to scroll what I wrote and came to the same conclusion as ElizaF:  

If we have the skills and the knowledge to consider ourselves as good testers we encourage others to be good testers also!  

There is no backdoor of which you can go. If you have knowledge and you don't share, your not a good tester. If you have skills that others require and it is possible to teach those, teach them or you're a bad tester. Be a good tester. Question your thinking. Think what you question!