Abstract

Automated Program Repair (APR) helps improve the efficiency of software development and maintenance. In recent years, Deep Learning (DL) approaches have been applied to the APR field and have shown promising potential in fixing software bugs automatically. The DL-based APR models translate buggy code to correct code directly. Some recent works test the general performance of various deep learning models on downstream tasks, e.g., code search and method name prediction. However, there still needs to be a fair evaluation of the deep learning models on automated program repair.

This paper aims to quantitatively and comparatively evaluate the repair performance and robustness of DL-based APR models. We first fine-tune seven pre-trained models and train two models from scratch on the unified dataset for a fair comparison of repair performance. Then, we conduct a robustness evaluation for nine trained models above against nine semantic-preserving code transformations. Our experiments show that DL-based APR models with pre-training perform better repair performance and robustness than those trained from scratch. Additionally, most APR models fine-tuned on the concrete code datasets have better repair performance than those fine-tuned on the abstract code datasets. Furthermore, most encoder-decoder-based and decoder-based APR models have better repair accuracy than encoder-based ones. Finally, compared with renaming-related code transformations, semantic-preserving transformations related to the change of syntactic structure have a more significant impact on the repair robustness of DL-based APR models. The results provide useful insights for achieving better DL-based APR approaches.

Index Terms–automated program repair, deep learning, robustness testing