TDDD04 Software testing

Lab 5, Mutation testing

Purpose

The purpose of this lab assignment is to present a contemporary tool and techniques to conduct mutation testing on real software, in a way that it is often done in the Industry.

Introduction

Mutation testing is a software testing technique and an active research area. It was first proposed in 1979. The purpose of mutation testing in simple words is, "Does our test really test what we think that they test?".

There are numerous tools in active development to conduct mutation testing. For Java which has been the primary language in the labs of this course so far, an important tool is PIT. However, in this lab, we will introduce Dextool. Dextool is an open-source tool proposed by SAAB Aeronautics to develop and maintain high-quality tests.

Preparation

First of all, make sure that you read the README.md for the example project game tutorial thoroughly. Instructions described and presented in the README assumes that you are in the correct directory before executing any commands/scripts, plugin/mutate/examples/game_tutorial/.

The lab will involve writing a few lines of C++. It is not required that you understand all the caveats of C and C++. However, we encourage you to get acquainted with the documentation for Google test.

Dextool

Dextool is a framework for writing plugins using libclang. The main focus is on tools for testing and static analysis. In a standard installation of Dextool, six different plugins are included. The mutation testing plugin, Dextool Mutate, functions in such a way that the user provides a configuration-file where scripts, settings, and different paths are specified. The picture above shows the flow for the plugin, where the test part of mutation testing, is depicted in a more detailed manner. As shown in the image, the plugin is divided into different parts (executable commandos) - analyze, test and report. There are two more executable commandos for Dextool Mutate, admin and generate, but these can be ignored in this lab. See Dextool Mutate README for more information.

Setup

In order to set up and use Dextool, it needs to build first. See the README for a more detailed walkthrough. Below are the same steps listed, but in a shorter version.

Open your command line terminal and add the module files for the course:
```
module add courses/TDDD04
```
Clone the repository by executing the following in your command line terminal:
```
git clone https://github.com/joakim-brannstrom/dextool.git
```
Create a build directory for the installation:
```
pushd dextool
mkdir build
popd
```
Setup cmake:
```
pushd dextool/build
cmake -DCMAKE_INSTALL_PREFIX=. -DD_COMPILER=/path/to/d/compiler ..
popd
```
NOTE: Change #path/to/d/compiler to correct path to your D-compiler. If you are unsure of the path, execute the following in your terminal and use the output:
```
which dmd
```
NOTE: If you change the -DCMAKE_INSTALL_PREFIX to something else, you have to remember that path for later (specifying a certain path for your binaries could be highly individual for developers, hence the reason for the flag)
Make the project and install the binaries:
```
pushd dextool/build
make install
popd
```

The previous steps should provide you with a freshly built version of Dextool and also the executable binaries for the different plugins in a standard installation. If you encountered any problem during the setup/installation of Dextool, ask the lab-assistant to guide you (don't hesitate to create issues on the Dextool github-page if anything needs to be improved or clarified in the README-guide).

Part 1 - Manual mutation testing

Consider the following code snippet written in C:


//Simple fizzbuzz may or may not be correct
#include <stdio.h>

void fizz_buzz()
{
  for (int i = 1; i < 35; ++i) {
    //% is the modulo operator in C.
    if (i % 3 == 0 && i % 5 == 0) {
        printf("Fizz!Buzz!\n");
      } else if (i % 3 == 0) {
        printf("Fizz\n");
      } else if (i % 5 == 0) {
        printf("Buzz\n");
      } else {
        printf("%d\n", i);
      }
  }
}

void main()
{
  fizz_buzz();
}

Select five different values of the loop integer i. For these values, we would like you to give us the expected output.

Present the expected output for a specific value of i to us

We would like you to introduce five mutants of your choice. Describe how that would affect the result for the five different values of i you selected. Present the findings of this small investigation and in your report, include a table reporting these results.

For this small exercise we would like that you apart from providing a structured answer to your investigation provide us with an answer to the following questions:

What is the practical use case of introducing mutants. Say that the five pairs of input and output printed to the terminal was our tests?
Why would we in an industrial context want to automate the process of introducing mutants?

Part 2 - Automatic mutation testing

First of all, make sure that you have done the Preparation. In this section, we would like you to use mutation testing on a small roguelike game. Read the instructions for game tutorial.

There exist a few tests for this game already. After you have read the instructions, we would like you to run dextool in the way that it is described in the README. Once you have done so, you should analyze the results.

What do the results of running dextool tell us about test quality? Relate the findings from doing this experiment to the first lab. Would we make similar observations for tests for highly coupled code as in the Colony example?

Hint: Consider the test case similarities!

Part 3 - Extend the test suite

In this part, we would like you to extend the test suite. Provide an extension to the test suite with the goal of improving the mutation score and optionally coverage. For each new test, you add to provide a rationale for why this particular new test is being introduced. For passing, it is enough that the mutation score is somewhat improved.

Hint: Use the results from the previous part, i.e., the generated mutation and coverage reports.

Attach all the tests you introduced to your report along with your rationale

Part 4 - Equivalent mutants

Equivalent mutants and how to deal with these is an active research topic. We would like you to read up on what an Equivalent mutant is and then do the following tasks:

Find at least one equivalent mutant in the fizz_buzz example. You might have already found one!
If we consider mutation score when doing automated mutation testing. Why is the existence of equivalent mutants a practical problem?

Reporting your results

See the general instructions for successfully completing the lab assignment. Note that you will have to upload the results on Gitlab before demonstrating to your assistant. Write a lab report that answers the questions stated above.

For Part 2 and Part 3 we would like to see screenshots of the tool in action that motivate the answers to the questions raised in those sections.

Page responsible: Lena Buffoni
Last updated: 2019-10-02

IDA - Department of Computer and Information Science