Login | Register

On Test Selection, Prioritization, Bisection, and Guiding Bisection with Risk Models

Title:

On Test Selection, Prioritization, Bisection, and Guiding Bisection with Risk Models

Najafi, Armin (2019) On Test Selection, Prioritization, Bisection, and Guiding Bisection with Risk Models. Masters thesis, Concordia University.

[thumbnail of Najafi_MASc_S2019.pdf]
Text (application/pdf)
Najafi_MASc_S2019.pdf - Accepted Version
Available under License Spectrum Terms of Access.
2MB

Abstract

The cost of software testing has become a burden for software companies in the era of rapid release and continuous integration. In the first part of our work, we evaluate the results of adopting multiple test selection and prioritization approaches for improving test effectiveness in of the test stages of our industrial partner Ericsson Inc. We confirm the existence of valuable information in the test execution history. In particular, the association between test failures provide the most value to the test selection and prioritization processes. More importantly, during this exercise, we encountered various challenges that are unseen or undiscussed in prior research. We document the challenges, our solutions and the lessons learned as an experience report.
In the second part of our work, we explore batch testing in test execution environments and how it can help to reduce the test execution costs. One approach to reducing the test execution costs is to group changes into batches and test them at once. In this work, we study the impact of batch testing in reducing the number of test executions to deliver changes and find culprit commits. We factor test flakiness into our simulations and its impact on the optimal batch size. Moreover, we address another problem with batch testing, how to find a culprit commit when a batch fails. We introduce a novel technique where we guide bisection based on two risk models: a bug model and a test execution history model. We isolate the risky commits by testing them individually, while the less risky commits are tested in a single large batch. Our results show that batch testing can be improved by adopting our culprit prediction models. The results we present here have convinced Ericsson developers to implement our culprit risk predictions in the CulPred tool that will make their continuous integration pipeline more efficient.

Divisions:Concordia University > Gina Cody School of Engineering and Computer Science > Computer Science and Software Engineering
Item Type:Thesis (Masters)
Authors:Najafi, Armin
Institution:Concordia University
Degree Name:M.A. Sc.
Program:Software Engineering
Date:9 January 2019
Thesis Supervisor(s):Rigby, Peter and Shang, Weiyi
ID Code:984877
Deposited By: Armin Najafi
Deposited On:27 Oct 2022 13:48
Last Modified:27 Oct 2022 13:48
All items in Spectrum are protected by copyright, with all rights reserved. The use of items is governed by Spectrum's terms of access.

Repository Staff Only: item control page

Downloads per month over past year

Research related to the current document (at the CORE website)
- Research related to the current document (at the CORE website)
Back to top Back to top