Abstract

In continuous testing, developers execute automated test cases once or even several times per day
to ensure the quality of the integrated code. Although continuous testing helps ensure the quality
of the code and reduces maintenance effort, it also significantly increases test execution overhead.
In this thesis, we empirically evaluate the effectiveness of test impact analysis from the perspective
of code dependencies in the continuous testing setting. We first applied test impact analysis to one
year of software development history in 11 large-scale open-source systems. We found that even
though the number of changed files is small in daily commits (median ranges from 3 to 28 files),
around 50% or more of the test cases are still impacted and need to be executed. Motivated by
our finding, we further studied the code dependencies between source code files and test cases, and
among test cases. We found that 1) test cases often focus on testing the integrated behaviour of the
systems; 2) 18% of the test cases have dependencies with other test cases, and test case inheritance is
the most common cause of test case dependencies; 3) We documented four dependency-related test
smells that we uncovered in our manual study. Our study provides the first step towards studying
and understanding the effectiveness of test impact analysis in the continuous testing setting and
provides insights on improving test design and execution.