Conditionals: Why you should avoid these two letters for better test case execution

At Qualicen, it’s often my job to check other people’s system test cases and tell the team what I think about these tests. So what do I look for? Well, in principle it is simple: After tests are written down, they are “only” executed and maintained. So this is where tests can be bad and I try to spot things that make execution and maintenance harder. For the maintenance, the largest problem here are clones, which we covered in our last blog post. For the test execution, the main problem that you want to avoid is that different testers test different things. This is called ambiguity and comes in many tastes. In this blog post, I want to explain what is structural ambiguity and why it is bad, and this way help you to create better test cases.

(Scroll to the summary, if you don’t care about the details) 😉

Ambiguous test flow

The problem for test execution that I want to discuss here, is an ambiguous test flow. This means, that for a single test case, there are multiple paths that a tester can follow when she executes the test. Let’s look at an example.

In the first case, we have the following simplified scenario: Think about a very simple version of wordpress. One core requirement here is that the system must enable the user to create a new blog post. To test whether this requirement holds in our system, there’s a set of test aspects (or test ideas) that we must test, for example, creating a new blog post as a logged in user. For each of these test ideas we create a test case, such as the shown test. Imagine you are a tester and execute this test. What do you do? You login, create a new blog post, add the title, save, and logout. And done. Easy as pie!
Now, let’s look at a second case. Again, imagine you execute this test, what do you do? Well, it depends on the status of the system!

And here we see the two letters to avoid: “IF”! When you write a conditional, such as an ‘if’, in your sentence, there’s quite a good chance that you just introduced an unclear test flow. Of course, the same holds when you replace the if through a when, in case or other conditional words, but that would make a horrible title for this blog post, so let us stick to the ‘if’. 🙂

What’s the problem?

The main problems of ambiguous test flows reveal themselves when tests are executed. The purpose of test execution is to reliably ensure that the system under test fulfils the system’s requirements. When we execute a test case, the test’s verdict (or test result) says either: “SUCCESS: Nice, this test aspect works in your system.” or “FAIL: Sorry, but this test aspect and consequently also this requirement does not work in your system.”

Now, with ambiguous test flow, we created a test case with contains multiple possible paths through the system. When we execute such a test, we leave it open which path through the system we will test. This has two main consequences:

• Test verdict is not reproducible: Take the following example underneath. Give the test into test execution. Now imagine the verdict is “FAIL”. Which combinations of input values under which conditions led to the given test result? Which of the paths did the tester follow? Now, in this example there is just four combinations, but this number doubles with each conditional! If we have just four conditionals (not uncommon in practice, I can tell you…), we would have 16 possible paths through the test case.
• Test verdict does not always ensure test aspect: Now, let’s assume we gave the test to a different tester. His verdict says “SUCCESS”. Do we know whether the test aspect works in the system? Do we know whether the requirement holds? You don’t, and the reason is that with an ambiguous test flow, the tester only reports that he ran the test, not which part of the test. Did he run all paths? Did he run just the most obvious path? You basically leave the choice to a tester. If great testers test the system, everything is fine, but you just don’t know.
• Requirements coverage becomes pointless: Lastly, because the verdict does not tell you anymore whether the requirements are fulfilled, the requirements coverage (metrics) also become pointless. Imagine you have 100% coverage of requirements by tests, and each test was executed. All your ambiguous tests say SUCCESS. Now: Do you know that 100% of your requirements are fulfilled in the system? You don’t because you don’t know whether all paths were executed during test execution.

Why would you do that?

When we ask the test engineers who write these tests, some are just not aware of this ambiguity. For those that are aware, there give us usually one of three reasons why they don’t specify the exact test flow.

• “Tester X will execute the test, he will know which path to go.” Often, test engineers tell us that the tester would know (or worse: that they should know). In other words, these test engineers leave it to the person executing the test to decide which path will best decide whether a test aspect works and a requirement holds. Is this a good excuse? In terms of risk, I argue that in most companies this is a bad idea. There is a high chance that teams change. Change inevitably leads to knowledge loss. I wish you best of luck that your next tester knows the same stuff as the one who just left. I doubt it though.
• “I don’t know which data is present at runtime.”: Many times, we hear that the data in the system under test is unclear during test execution. Why is that? In system testing, we test the system as a whole. Usually, realistic systems have tons and tons of dependencies to other systems. This could be for example a database, another system that we must access, or a webservice hosted by someone else. The core reason behind this argument is that the tester cannot control the test environment. Is this a good excuse? Well, yes and no. No, because in terms of risk, your test cases are unreproducible and don’t ensure test aspects (Especially no if you don’t have a test environment and test in a productive system). Yes, because in many cases the test engineer can’t change it, so who am I to blame the tester for a bad architecture? So, in this case we have a certain risk, and the test engineers in the system must analyze: Does the verdict guarantee test aspects and requirements? Does the team trust the tests?
• “It doesn’t matter which path the tester goes.” The core argument here is that the ambiguity does not involve the core of the test. What does that mean? As I wrote, a requirement leads to a test idea. The test engineer adds a setup and tear down that is required to test the test idea. This is often something like login, test data creation or similar. For example, imagine in the above example that you have an ambiguous test flow in the login in Step 1. Is that a problem? No, because (hopefully) the login mechanism is tested somewhere else. Instead we here only creation of the post. And here it comes back to the test aspect: What do you want to test with this test case? Is this a good excuse? We sometimes discuss this reason at Qualicen and, for sure, in the above example, it wouldn’t hurt to make it easier for the tester by specifying the exact test setup. But arguably, the risk for the three problems above is low and so I would argue that this a good reason.

So what should I do?

Here is my personal advice: First of all, be aware that conditionals in your system tests create multiple flows through the test. This means that two testers could follow two different paths through the test.

Next, check if that is ok. Here the core questions is: Does the ambiguity in any way concern the test aspect? If it doesn’t you might want to specify the exact test data and setup, but then who cares… If it does, I strongly advice to split up the test case into multiple test case. This way, you make sure that all paths are covered and each test is executed in a reproducible manner.

Phew, that was long? What’s the short version?

There are many different ways to screw up your manual system test case descriptions, of which one are ambiguous test flows. Conditional phrases, such as if, in case, and others are indicators that there are multiple paths through your test, and that you leave it to the tester to choose which path to test.

The problem with this is that tests are not reproducible anymore and, all of a sudden, even when all your tests report SUCCESS, you do not necessarily now whether your system works as expected. There are multiple reasons why people introduce these “multi-path” tests, but the only one to which I fully agree is that the ambiguity is not in the focus (the test aspect) of this test. In all other cases you should, first, be aware of such ambiguity (and use tools such as the Requirements Scout to tell you about these problems). And, second, I advice to try to split up the test case into multiple test cases. This way tests serve their purpose: Make you sleep well at night, because you know: My tests tell me whether my system does what it should.

Do you agree or disagree with my opinion? Let me know in the comments, on twitter or via direct mail.