Hey BERT, do you recognize any requirements?

Does scanning for your relevant requirements in massive documents sound familiar? In this blog post we are going to find them for you…

Why is Requirements Recognition an interesting Task?

Requirement documents contain a lot of information. Not all of this information is relevant to every stakeholder. System and Test Engineers for instance only really care for the requirements that are somewhere in that pile of information. Hence for those engineers it would be great if a document containing just the requirements would exist.

If we want to create, preferably automatically, such a document we first of all need to know which parts of the text are requirements and which aren’t.

Before we dive into the details of requirement extraction and the techniques behind it, we want to take a look at a little example:

In this figure we see an excerpt of a requirement document. The requirements contained here are highlighted in green, while the other text that does not contain a requirement is not marked.

In this example, we can see the difference between requirements and the additionally added information.

The text that is marked with “R2” says:

“When in remotely controlled mode, the speed must be limited by the rover navigation controller to 3 km/h with a target speed of 5km/h.”

This is a requirement and it describes the specific behavior that the system shall have. The next sentence is not marked as a requirement. Let’s take a look at it:

“This is because in the remote controlled mode, there is a likely delay ≥ 20 ms in information transfer from the controller to the operations centre.”

Here we have an explanation of why this requirement exists, the so-called rational. In this example we can see the difference between the classes requirement and non-requirement. The first describes the system under development. The latter adds additional information that does not describe the system but still provides valuable information for some stakeholders.

The Struggle of recognizing all Requirements manually 

Going through a massive requirement document manually, to find all requirements is time consuming and error-prone, because there are no easy ways to do it except brute-force.

The idea of looking for headlines that imply the existence of requirements in a section sounds like a good plan. We can already see from our example that this section also contains non-requirements even though the headline clearly says “requirements”. When looking for traces, we face the same problem as with the headlines. Text is a mix of requirements and non-requirements. Even though these ideas sound good in the first place they have obvious flaws.

Another idea would be to use typical keywords, like the word “note” in N5 of the example. This keywords implies the absence of a requirement. In the example we see that the next sentence is again a requirement. The keyword only marks the beginning of a non-requirement. This fact makes it difficult to use the keyword to find non-requirements.

The only valid option left is to read and understand the whole text. Reading the whole document is very time consuming. Depending on the quality of the requirements it could also be ambiguous to which category (i.e.: requirement, non-requirement) a sentence belongs, which makes manual labeling error-prone.

To sum it up recognizing requirements manually is hard, time consuming and error prone. So we need to find a better way to do this.

A better way to do it

So with manual labor out of the window the next thing is to use the power of text analytics to automate this task. Now let’s look at the different approaches and dive deeper into the techniques that can be used to solve this task.

Our Approach

When we look at the task from a more technical perspective, we see that this task resembles a classical text classification task. As mentioned before there are several approaches to solve this problem. We start with looking at possible solutions. Then we will evaluate each of the possible solutions and come up with an approach. Last but not least we will take a look at the results of our approach.

Different Classes of Approaches

In Natural Language Processing (NLP) there are two famous types of approaches for classifying sentences. The first being a rule based system and the second being a system based on machine learning. In the following, we will take a look at the differences of these two approaches and highlight their advantages and disadvantages.

Rule Based Systems

The first of the classical classification approaches in the field of NLP is to use a rule based system. Therefore, we would need to define rules to match requirements. These rules can be based on a specific syntax, special language patterns or keywords. 

Pros: 

• Easy to understand why classifications happen

Cons:

• Creating rules to cover all cases is time consuming
• Focused on a specific style of requirements document, not generally applicable

Machine Learning Based Systems

In the new area of NLP machine learning based classification approaches rise in popularity. These approaches automatically find patterns in the language and use them to classify text.

Pros: 

• Automatically learns how to execute the task
• Is able to generalize

Cons:

• The reason for a specific classification is difficult to predict

Our Use Case

In order to decide, which kind of approach to choose, we need to first define our use case. 

We want to create a system that is able to classify sentences into the categories requirement and non-requirement. The system shall work on a general natural language, so it is independent of special syntax, like for example the EARS patterns.

We want to be independent from a specific style of creating the requirement documents so we can’t use a rule-based approach, therefore we use a machine learning approach.

An Overview of our Architecture

Our machine learning model features two parts. The first part being a pre-trained version of the BERT language model and the second part is a Neural Network. 

The first part of the model is used to convert the input text into a machine understandable format. We use the result of this abstraction for the second part, the classification. 

The classification is done by a Neural Network with a single layer. 

Our model receives a sentence as input and produces probabilities for the two classes. We choose the class with the higher probability for the classification.

What is BERT and why is it so popular?

BERT (Bidirectional Encoder Representation from Transformers) is a language representation model. Researchers from Google proposed and open-sourced this project in 2018. This model can be used as a base for many different NLP tasks. The pre-trained model comes with a good understanding of general language features. This is helpful for all NLP tasks. In our case we use BERT to first of all improve the overall accuracy of our model and secondly reduce the required amount of training data and time.

Our Results

We conducted a study to evaluate whether our architecture is suitable for the task at hand. Our model reached a recall and precision of 84% for the class requirement. When we compare our result to related work, we see that your model is slightly outperformed by the competitors. The amount of data used to train our model was limited and the related work had up to three times the data that we had. Seeing that our model is outperformed only slightly suggests, that our architecture is suitable for the task.

It is expected that if we would had more data the performance of our model would have clearly exceeded the competitors.

So you see we can identify requirements hidden in massive documents automatically, faster, more consistent and probably better than anyone could by manual extracting the requirements.

TL;DR

Requirement documents that feature system requirements and additional information are common in the industry. Some stakeholders are only interested in the requirements. Extracting the requirements manual is time consuming and error prone. Therefore, we use the power of Natural Language Processing to automate this task.

We used a machine learning approach to automate the classification process. Our architecture features BERT and a classification layer to classify text on a sentence level. 

Our first results are very promising as our model reaches a good accuracy despite the fact that our data set was very small.

To answer the question: Yes, BERT can recognize requirements.