Hide menu

Current thesis proposals

What should students learn for their theses? -- a curriculum alignment study (30hp)

During the three to five years students spend in Engineering studies, they take a number of courses that are hopefully related to their work as engineers. To test whether the right mix of courses is offered to our students, we can use openly available material on the kind of final theses students write, and courses that are available as part of their degree programmes as described in the Study Guide and on course web pages.

Using this information, this thesis will use Machine Learning techniques to explore what topics are usually selected for final theses, and how courses prepare students for those final theses by matching course content descriptions with thesis descriptions.

The thesis will contribute to our understanding of the relationship between curriculum development and final theses, hopefully for most Engineering degree programmes at LiU.

Suggested background: Student with experience in Machine Learning, natural language processing, and (some) web programming.

BS Thesis 1

Background:
Principles and practices of software engineering have been significantly improved with the introduction of agile methods instead of the traditional waterfall model, leading to customer and business focus as well as improved software quality. One of the major advantages of agile methods is the faster release of a product. Continuous Integration (CI) is the practice of automatically compiling, building and testing software. It is part of the agile method and has been widely adapted. For example, Google, Amazon and over 900k open source projects are using Travis CI. The adaption of CI provides several benefits such as faster and continuous feedback [1][2], more frequent releases [3][1][2], improved product quality and developer productivity [1][3], increased customer satisfaction [2] and early bug detection [3].

Regression testing, one of the challenges in CI [4][5] must be cost-effective to provide faster feedback to developers. To improve productivity, developers should get immediate feedback, after they submit their code to main base. A high frequency of system builds and tests requires a large amount of time and resources [3]. For example, the feedback time, after testing, to developers in Google is between 45 minutes and 9 hours [6]. It is not an easy task to achieve faster feedback during regression testing. Many researchers have investigated this challenge such as 'regression feedback time' in [4], 'long running tests' in [1][4] and 'time consuming testing' in [5][7]. RTS (Regression test selection) involves relevant and important selection of test cases, from a large repository, and TCP (test case prioritization) involves an efficient re-order of test cases to be executed to provide faster feedback. Similarity based test case selection has been reported as an effective approach for RTS and TCP [8]. Project Description:

This BSc thesis project is a part of research project named AAT (Aspects of Automated Testing), under the umbrella of Software Center (https://www.software-center.se/). At this moment, we have successfully implemented a technique that can find similarities between test cases [8] but the limitation of our implementation is that it can only be applied to specific datasets (i.e. test cases in Python), received from our partners in Software Center. The first aim of this thesis is to integrate this technique with standard input and output, so it can be applied to any other data set. Standard input/output means that we need to convert required information into the proper and general format such as XML or other, due to the fact that each company follows specific language for writing test cases or definitions. This may require an investigation of common standards or frameworks for representing test case’s information. Another important contribution of this thesis is to run the selected test case in CI environment (i.e. Jenkins). This can be achieved by writing or extending a plugin that does not only run selected test cases, but also validate the resulting reduced test suit.

Steps:

  1. Find the standard input/output that can be used for input to similarity-based test case selection technique, that we developed as part of AAT3 project. (your contribution)
  2. Integrate above methods with technique, that is already developed. (your contribution)
  3. Run the technique and identify the subset of test cases (Already developed)
  4. Run selected test cases on source code in Jenkins (i.e. we may use open source software for source code) (your contribution)
  5. Evaluate the test execution data to find the efficiency of selected test cases (your contribution)
Outcomes:
The final outcomes of this thesis can be a plugin that can be integrated within CI environment that accommodate the steps provided above.

References:
[1] M. Leppänen et al., “The highways and country roads to continuous deployment,” IEEE Softw., vol. 32, no. 2, pp. 64–72, Mar. 2015.
[2] P. Rodríguez et al., “Continuous deployment of software intensive products and services: A systematic mapping study,” J. Syst. Softw., vol. 123, pp. 263–291, Jan. 2017.
[3] M. Hilton, “Understanding and Improving Continuous Integration,” in Proceedings of the 2016 24th ACM SIGSOFT International Symposium on Foundations of Software Engineering, New York, NY, USA, 2016, pp. 1066–1067.
[4] A. Debbiche, M. Dienér, and R. B. Svensson, “Challenges When Adopting Continuous Integration: A Case Study,” in Product-Focused Software Process Improvement, 2014, pp. 17–32.
[5] F. J. Lacoste, “Killing the Gatekeeper: Introducing a Continuous Integration System,” in 2009 Agile Conference, 2009, pp. 387–392.
[6] A. Memon et al., “Taming Google-scale continuous testing,” in 2017 IEEE/ACM 39th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), 2017, pp. 233–242.
[7] A. Miller, “A Hundred Days of Continuous Integration,” in Agile 2008 Conference, 2008, pp. 289–293.
[8] F. G. de Oliveira Neto, A. Ahmad, O. Leifler, K. Sandahl, and E. Enoiu, “Improving Continuous Integration with Similarity-based Test Case Selection,” in Proceedings of the 13th International Workshop on Automation of Software Test, New York, NY, USA, 2018, pp. 39–45.

Deflaker for Python

Background:

The failure found, during regression testing, are caused by the change in the source code but sometimes, it is due to non-deterministic tests. These types of tests are called flaky tests [1]. Tests that indicate failures but do not execute any code affected by changes detract developers from investigating real faults and lowers confidence in test automation. Bell et. al. have developed a tool called “Deflaker” [2] for Java projects that detects if the test failure is due to flaky test, without rerunning the test and with a very low runtime overhead.

Project Description:

This project requires an investigation of test artefacts or other attributes in testing to detect and avoid flaky tests. You need to study the root causes of flaky tests and mitigation strategies. This BSc thesis project involves re-implementing “Deflaker” for Python. This tool will be used in an on-going study within the research project Aspects of Automated Testing, in collaboration with Software Center (www.software-center.se). You are required to understand how Deflaker works and adapt it for Python projects. The secondary contribution is to run this tool with open source Python projects to evaluate its efficiency.

References:

[1] “Flaky Tests at Google and How We Mitigate Them,” Google Testing Blog.
[2] J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “DeFlaker: Automatically Detecting Flaky Tests,” in Proceedings of the 40th International Conference on Software Engineering, New York, NY, USA, 2018, pp. 433–444.

Spam detection for test flakiness

Background:

Most of the time, regression failures are caused by the change in the source code but sometimes, it is due to non-determinism in the tests called flaky tests [1]. Given no change in source code, if the test case is failed, but earlier, it was passed can give rise to alarming situations such as less confidence in the quality of final product or test process. Researchers in [2][1][3] have identified several test smells that cause flaky tests.

Project Description:

This MS thesis project is to investigate spam email classification algorithms to detect test flakiness. The dataset for flaky test is available and the project aims to investigate which algorithms performs better in terms of flaky test detection and prediction, given test smells. This tool will be used in an on-going study with software center (www.software-center.se) called AAT3. You are required to implement spam email classification algorithms and determine their effectiveness.

References:

[1] “Flaky Tests at Google and How We Mitigate Them,” Google Testing Blog. .
[2] Q. Luo, F. Hariri, L. Eloussi, and D. Marinov, “An empirical analysis of flaky tests,” 2014, pp. 643–653.
[3] M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, “Understanding Flaky Tests: The Developer’s Perspective,” Proc. 2019 27th ACM Jt. Meet. Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng. - ESECFSE 2019, pp. 830–840, 2019.

Tuning learning algorithms for optimal regression test ordering (30hp)

Automated testing is a very important prerequisite for Continuous Integration. Automated testing ensures that changes do not cause bugs in existing software, or have otherwise unintended effects. Automated tests need to be efficient, effective and reliable in order for the continuous integration setup to provide the value needed in a modern software development environment. At LiU, we are conducting research in collaboration with several industry partners to identify key issues in software test automation and ensure that contemporary research are integrated in industry practice. So far, there has been many studies on how to optimize regression testing by means of selecting or prioritizing tests to run after individual builds, or reduce the total size of a regression test suite. However, these studies have not resulted in open software for prioritizing or selecting tests

Students are expected to create a Jenkins plugin that allows us to automatically prioritize tests in a workflow based on metrics such as APFD (Average Percentage of Failure Detection), that indicate how early in a test suite failing tests as executed, and techniques such as those introduced by Thomas, Hemmati et al [1] and empirically evaluate the results on company data from our industry partners.

Suggested background: Master's students with experience in Machine Learning and good knowledge of software testing

Reference:

[1] S. W. Thomas, H. Hemmati, A. E. Hassan, and D. Blostein, “Static test case prioritization using topic models,” Empirical Software Engineering, vol. 19, no. 1, pp. 182-212, 2014.

Software Testing Optimization with respect to hardware setup times (30hp)

When testing hardware in embedded systems, there are typically significant overhead costs to incurred by setting up and initializing the hardware. Developers who want to know if firmware changes will have adverse effects want fast feedback, which requires making optimal use of hardware resources to explore the most likely areas of failures in the current context. This will require both selection of test cases to execute as well as a priority ordering to help developers obtain fast feedback on changes.

The project aims to provide an optimization framework for automated testing that can provide optimal groups of tests given state requirements for each test and the costs of setting up hardware states that match the requirements for each test. The framework shall be evaluated against company data from either Axis Communications, Volvo Cars or Saab.

Suggested background: Good knowledge of software testing and skills in Java and Python. Knowledge in embedded systems is preferred.

Reference:

Y.-C. Huang, K.-L. Peng, and C.-Y. Huang, “A history-based cost-cognizant test case prioritization technique in regression testing,” Journal of Systems and Software, vol. 85, no. 3, pp. 626 - 637, 2012.

Software Regression Testing Optimization platform in Jenkins (15HP)

Background Regression testing, as in testing for known bugs using automatic software tests, must be cost-effective and provide fast feedback to developers. To improve productivity, developers should get immediate feedback after they submit their code to mainline. A high frequency of system builds and tests requires a large amount of time and resources. It is not an easy task to achieve fast feedback during regression testing as more and more regression tests are written. On technique used to address this problem is RTS (Regression test Selection), which involves the selection of most relevant and important test cases for a particular build, and TCP (Test Case Prioritization), which involves an efficient ordering of test cases to be executed with respect to feedback time. Similarity-based test case selection is one approach for RTS and TCP [1].

ProjectThis BSc thesis project is a part of research project named AAT (Aspects of Automated Testing), under the umbrella of Software Center. At this stage, we have successfully implemented a technique that can find similarities between test cases [1], but the limitation of our implementation is that it can only be applied to specific datasets (i.e. test cases in Python), received from our partners in Software Center. The first aim of this thesis is to integrate this technique with standard input and output, so it can be applied to any other data set. Standard input/output means that we need to convert required information into the proper and general format such as XML or other, due to the fact that each company follows specific language for writing test cases or definitions. This may require an investigation of common standards or frameworks for representing test case's information. Another important contribution of this thesis is to run the selected test case in CI environment (i.e. Jenkins). This can be achieved by writing or extending a plugin that does not only run selected test cases, but also validate the resulting reduced test suite.

Suggested background: Good knowledge of software testing and skills in Java and Python.

Reference:

[1] F. G. de Oliveira Neto, A. Ahmad, O. Leifler, K. Sandahl, and E. Enoiu, "Improving Continuous Integration with Similarity-based Test Case Selection," in Proceedings of the 13th International Workshop on Automation of Software Test, Göteborg, Sweden, 2018, pp. 39-45.

Detecting flaky tests in Python (15HP)

Background Failures found during regression testing are assumed to be caused by changes in the source code but sometimes, test fail non-deterministically. These types of tests are called flaky tests [1]. Tests that indicate failures but do not execute any code affected by changes detract developers from investigating real faults and lowers confidence in test automation. Bell et. al. have developed a tool called Deflaker [2] for Java projects that detects if the test failure is due to flaky test, without rerunning the test and with a very low runtime overhead.

Project description This project requires an investigation of test artefacts or other attributes in testing to detect and avoid flaky tests. You need to study the root causes of flaky tests and mitigation strategies. This BSc thesis project involves re-implementing Deflaker for Python. This tool will be used in an on-going study within the research project Aspects of Automated Testing, in collaboration with Software Center. You are required to understand how Deflaker works and adapt it for Python projects. The secondary contribution is to run this tool with open source Python projects to evaluate its efficiency.

Suggested background: Good knowledge of software testing and skills in Java and Python.

Reference:

[1]"Flaky Tests at Google and How We Mitigate Them," Google Testing Blog, 2016.
[2]J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, "DeFlaker: Automatically Detecting Flaky Tests," in Proceedings of the 40th International Conference on Software Engineering, New York, NY, USA, 2018, pp. 433-444.

Online, graphical constraint-based equation solving with dynamic problem generation (30hp)

Constraint-based equation solvers are powerful tools for solving optimality problems in many domains, but they are yet to become the mainstream method for solving problems where users are not experts in the use of constraint solvers. For many, using advanced mathematical models without the ability to visually inspect or manipulate the model is a major obstacle to using advanced techniques for solving constraint problems.

This thesis will explore available open tools that can be used for constraint-based, iterative equation solving, and provide a graphical framework for interacting with constraint solvers by providing graphical components that can be used to model entitites in a constraint optimization problem, and their relationships to one another. The tool will be evaluated with respect to how fast users are able to learn and use the tool to create and solve new problems, and how the proposed design for the tool will enable users to model new types of constraint problems.

Suggested background: Master's students with courses in interaction design and mathematical (linear, combinatorial, constraint-based) optimization. Possible for two students.

More will come...


Page responsible: Ola Leifler
Last updated: 2019-09-18