Detect more Quality Defects in your Requirements

How we investigated whether our Qualicen Scout is a useful tool for companies in the domains of software and systems engineering.

Why we wanted to answer this question

As science showed, the quality of the requirements documentation influences the subsequent activities of the software engineering process. Detecting errors late in a software engineering process leads to very expensive changes of parts of every pre-executed activity. Accordingly, we at Qualicen help our customers to assure the quality of requirements specifications before they are used in other activities.
The most common approach to assure the quality of requirements specifications is the manual review of requirements. But manual reviews need a lot of time and are therefore expensive. This is the reason why we developed a complementary approach for quality assurance of requirements documents which is called Requirements Smells. Requirements Smells are indicators of quality problems and have a detection mechanism which enables us to detect these indicators automatically with our Qualicen Scout software.

We assumed an increased efficiency and effectivity of the quality assurance process by using Qualicen Scout. But these assumptions were never investigated scientifically. This is, why I analyzed these assumptions in my master’s thesis. In my thesis, we developed an experiment which enables us to investigate whether or not the use of Qualicen Scout leads to reduced time efforts of the manual review and an increased number of detected quality defects.

Experiment Design

We asked four different companies to send us requirements authors and reviewers as participants for our experiment. The companies send us 14 participants. In our experiment, the participants had the task to review real-world specifications from their domain. Therefore the participants got the same documents but in different versions.

One group had to review the document which was analyzed by Qualicen Scout and where we removed the quality problems which were detected. The other group had to review the document in its original form. This way, we could measure the required time for the manual review and the number of detected findings by the reviewer and by Qualicen Scout for both versions.

Results of the Experiment

Our experiment showed that there is a significantly increased number of overall detected quality issues during the review process consisting of (Scout) and manual Review when Qualicen Scout is used before the manual review compared with when it is not used. In the worst case there is an increase of 7.1% and in the average case, there is an increase of 11.4% in the number of detected quality defects. Each detected quality defect decreases the probability that major and costly problems occur during an activity along the software lifecycle.

This means, that requirements smell analyses are capable of enhancing the quality of requirements documents and therefore reduce costs for the activities of the whole software lifecycle. Because our results are based on data from industry experts and different domains, we can generalize these results to industrial practice. We, therefore, recommend our customers using Qualicen Scout before conducting manual reviews.