Automatic Feedback on TFS/VSTS Test Case Quality in Real Time

Team Foundation Server (TFS) and its software-as-a-service counterpart Microsoft Visual Studio Team Services (VSTS) are widely used application lifecycle management (ALM) and test management tools. They offer many great facilities to create tests, manage test plans, and execute them.  Consequently, many of our clients as well as prospective customers wanted to use our test improvement software Test Scout along with TFS/VSTS to improve the test case quality. So, here is the question that we always face: How do we get the data from a testing tool into the Test Scout? As always in life, there is a straightforward and a fancy solution. Let me show you what I mean.

First: a simple integration

Test Scout is able to process almost any kind of text format. So, integrating test management tools such as TFS/VSTS is quite straightforward: For each test management tool, we created exports, which we imported into the Scout. For HP ALM, for example, we use a simple script to create a database dump containing all currently existing test cases. We then automatically imported and processed this data in Test Scout to evaluate the test case quality. Since Test Scout keeps versions of each import in its database, the history of all test cases is available in Test Scout. Therefore, all features, such as comparing different versions of test cases and historical development of test cases still works out of the box.

 

… but imports have drawbacks

As simple and flexible this solution is it has some drawbacks: Exporting all test cases may take long time and causes high load on the test management tool server. In most cases, we could perform exports just every couple of hours. In some cases, we had to schedule the export at night when no one else was using the test management server. Obviously, this was limiting the frequency of (meaningful) exports to one export per day.

Having such long gaps between exports, the feedback about your test case quality you get from Test Scout is often not up to date. It’s just a snapshot of the last export. This is nice, but not what we had in mind when programming Test Scout: We want to help the tester when he/she needs it! (And not the next day…)

Long story short: The delay in the feedback loop makes it almost impossible to get early feedback on your tests while editing them.

Introducing: Our new TFS/VSTS connector …

To overcome these drawbacks and to promote Test Scout to a rapid feedback tool (what Test Scout is actually built for), we realized a closer integration: A dedicated connector for TFS/VSTS. Test Scout has a sophisticated plugin architecture which allows creating connectors for individual ALM tools. Having such a connector, we can connect Test Scout directly to TFS/VSTS without the intermediate step of exporting all test cases to txt/csv files. Our connector uses the REST API of the TFS/VSTS servers and is able to identify within a couple of seconds which test cases you just modified. These (and just these) test cases are then loaded into Test Scout.

Using our new TFS/VSTS connector, you can connect test scout to a TFS/VSTS project in no time by only providing the server URL, project name and credentials. (Okay, you may also want to add some filters if you want to reduce the number of analyzed test cases but let’s ignore that at the moment).

The initial import of all test cases may take a while (depending on the number of test cases and history of your TFS/VSTS project) however, you already have imported the whole history of all existing test cases then.

Immediate feedback on test case quality!

Since Test Scout is informed about changes of your test cases and has to analyze just(!) the changed part, the analysis now takes just a couple of seconds for each new version. Actually, after clicking the save button in VSTS/TFS, it’s just a matter of seconds until Test Scout has processed the modified test case and gives you feedback about the test case quality. By this, Test Scout becomes the actual rapid feedback device it was meant to be. Plus: You can use your browser’s desktop notifications to tell you what went wrong. Isn’t that great?

Long story short

To get data from a testing tool (such as TFS or VSTS) into our Test Scout, there is a straightforward and a fancy way: The easiest way to get quality feedback on your VSTS/TFS test cases is to regularly export them into csv/txt files and feed them into Test Scout. This will work out of the box, however, the waiting time until you get feedback depends on the frequency you are able to perform exports.

The fancy way is to use a dedicated connector to connect Test Scout directly to your test management tool. This way, you get feedback on your test cases in (almost) real time and can use this feedback to improve your test cases while you are writing them.

Right now, we are building connectors for more and more test management and ALM tools, such as PTC Integrity LM and IBM DOORS. You are interested in connecting a certain tool? Just let us know.