Abstract

Software testing is vital for ensuring software reliability and robustness. It involves executing a program and verifying it against developer-defined criteria to identify and fix deviations, aiming for fault-free software. Despite extensive research on automated test prioritization, fault localization, and program repair, test design and maintenance remain under-explored. This dissertation aims to understand test code quality from the perspective of design and maintenance practices, exploring various aspects of test design through manual classification, quantitative methods, and automated approaches.

The first part of the dissertation examines test smells, which are design issues that impact test code comprehension and maintainability. Although widely accepted in academia, it is unclear whether developers address test smells in practice. The thesis investigates developers' awareness of test smells and their impact on defect-proneness. Findings reveal that many proposed test smells persist and are removed incidentally through code deletion and refactoring, with minimal effect on software defect density. This dissertation aims to provide empirical support for re-ranking current test smell catalogs.

As test automation grows, modern frameworks like JUnit and TestNG are increasingly used in Java-based systems. These frameworks introduce annotation-driven development, which manages crucial aspects of the test execution lifecycle. Our second aim is to use mining software repository techniques to provide insights into how modern testing frameworks are used to improve test code design and maintenance. First, we investigate annotation usage in test code, providing a manual classification of their API usage and misuses (e.g., test smells). Second, we conduct a detailed empirical analysis of one controversial practice: disabling tests using JUnit and TestNG's @Ignore annotation. While this alleviates maintenance challenges, it introduces technical debt as it does not validate software quality. Finally, we examine the tradeoffs between reusability and redundancy in test code practices, particularly through inheritance, highlighting issues in test increasing test execution time.

Through case studies and experiments, the techniques proposed in this dissertation offer new insights into test code quality that may guide the development of automated tool support in the future, which may help developers improve test maintenance.