Structured Test-Design with Specmate – Part 1: Requirements-based Testing

In this blog post  I am going to introduce Specmate, the result of a research project I have been involved into. It is an open-source tool to automate test-design, among others. This is the first post of a series in which I am going to show you some of the ideas behind Specmate.

What is test-design and why does it matter?

Test-Design is the activity to come up with the right test-cases for a piece of functionality. But what are the right test-cases? There are many criteria, depending on your focus. For me, there are two main points:

• First, they should test the right content. That means, they relate to the requirements for this functionality and cover every aspect that the requirements talk about. They should hence be able to find faults: deviations of the implementation with respect to the specification.
• Second, they should be feasible. That means, it should be possible to execute the test-cases without wasting resources.

The job of a test-designer is to create test-cases that realize these two criteria. For this, a good test-design is the basis. Creating the test-design involves understanding the functionality in-depth:

• What are the inputs?
• What are the outputs?
• How do the inputs relate to the outputs?
• What are the necessary interaction steps?
• Are there any frame-conditions that influence the whole functionality?
• Which environment variables need to be considered?
• …

From here, the the test-designer needs to do handle the combinatorics: Which combinations of inputs, frame-conditions and environment variables should be tested? Finding a sweet spot between testing a couple of ad-hoc combinations and  testing the full cross-product is the task of the test-designer (many systematic methods are available for this task, though).

Why does systematic test-design matter today? Isn’t exploratory testing the way to go to get good quality efficiently? Yes and no: (Ad-hoc) exploratory testing without explicit test-design is good at finding certain faults quickly. It leverages the experience of the testers and their gut-feeling to identify bugs. However, if you don’t combine it with an explicit  test-design (which you can!) you will most certainly rest with a bad conscience: Have you tested the system under condition X? Test-design is what gives you back your good conscience.

Why is test-design hard?

Test-Design is hard for number of reasons: First, usually not all the relevant information for creating the tests are documented (e.g. in requirements documents). For example, the information which configuration parameters influence a certain functionality is usually nothing you find in requirements documents. Hence, you need the experience of testers or developers to identify these pieces of information. Unless you can access information on how test-design has been done in similar situations. Furthermore, often requirements are ambiguous or incomplete. You need a means to clarify what the requirement specifies precisely.

A second reason why test-design is hard, is the combinatorics. As soon as you have several input variables that influence the outcome, figuring out which test-cases you need is cumbersome. Equivalence partitioning is simple for toy examples, but have you ever done it for real software projects? Building the cross-product of all variables is normally not an option: for 6 variable with two values each you already get 64 test-cases.

Specmate Overview

Specmate is an open-source tool that aims to support and automate these challenges of test-design. It is the outcome of a one-and-a-half year research project involving Technische Universität München and Allianz Deutschland. I had the joy to work on this project for the whole time. You can get the code and early releases at GitHub and an online demo here. In the current state it is a research prototype, missing several necessary features to use it productively (such as a authentication). However, it still might be worth having an early glimpse.

At the moment, Specmate provides two main workflows:

1. Creating tests from single requirements (a.k.a. requirements-based testing)
2. Creating end-to-end-tests from business process descriptions.

In this first part of this article I want give you an overview over the requirements-based testing workflow. More details on other workflows will follow.

Requirements-based Testing with Specmate

When you start Specmate you will basically see an empty screen at first. On the left side, there is the project explorer, where you can browse through the elements that make up your project. Currently, we load a couple of examples (which however are in german at the moment). You can find them when you open some of the folders of the example project. Select a requirement and Specmate shows the  details on the right.

Step 1: Model Requirements

The interesting part is towards the end of this page. Here it says “Cause-Effect-Models”. That is one way to start with the test-design in Specmate. Cause-Effect-Models (CEGs) are a graphical way to describe logical implications that are described in the requirements text. Hit the “+” button to get to the CEG editor. In this editor you can create CEGs graphically.

Let’s think about an ATM machine. A requirement might be that an ATM dispenses money in case a user inserted a bank card, entered a correct PIN and entered an amount to dispense. To model this in a CEG graph, we would create three nodes for the causes (bank card inserted, correct PIN entered, amount entered) and one node for the effect (money dispensed). Hence  we create the following diagram:

Step 2: Generate Test Specification

If we are happy with the model we can hit “Generate Testspecification” on the right in the “Links & Actions” part. And there you go: a test-specification with four logical test cases. You can tweak them if you like (for example, I removed some values, as they are irrelevant), add more test cases or remove some of the existing ones.

Step 3: Create test procedures

From here on, the next step is to create test procedures. That’s where will pick off next time!

TL;DR

• Testing makes just sense if you have the right test cases.
• Finding the right test cases is not easy.
• In this post I showcase a prototype of a tool for semi-automatic test design called Specmate
• I show how to model requirements in Specmate using cause-effect diagrams and generate test specifications