Login | Register

Towards Understanding and Improving the Value of Chatbots in Software Engineering


Towards Understanding and Improving the Value of Chatbots in Software Engineering

Abdellatif, Ahmad (2021) Towards Understanding and Improving the Value of Chatbots in Software Engineering. PhD thesis, Concordia University.

[thumbnail of Abdellatif_PhD_F2022.pdf]
Text (application/pdf)
Abdellatif_PhD_F2022.pdf - Accepted Version
Available under License Spectrum Terms of Access.


Software chatbots have been around since 1966, where the first computer interacted with a human. Recently, advances in artificial intelligence and natural language processing and understanding led to the rise and widely use of chatbots in a variety of services (e.g., healthcare, e-commerce, customer service). Nowadays, chatbots have become the main conduit between humans and services. Through natural language, chatbots enable users to communicate with different services intuitively. Another reason for the increasing popularity of chatbots in several domains is their benefits in saving time, effort, and cost. These benefits and wide adoption of chatbots attract practitioners to implement chatbots that support software engineering tasks. Chatbots play an important role in various software development tasks from answering development questions to running tests and controlling services. While there are numerous chatbots and their capability of supporting software practitioners is encouraging, little is known about the development challenges and usage benefits of software engineering chatbots.

This thesis presents series of empirical studies that aim to understand the challenges of developing chatbots for the software engineering domain, highlight the value of using chatbots in software development, and proposes novel approaches to support developers at developing more efficient software engineering chatbots. More specifically, we tackle three aspects of chatbots in software engineering. First, we present an empirical study to explore the chatbot development challenges. We find that chatbot developers face several challenges that are related to chatbot integration, development, natural language understanding platforms (NLUs), user interaction, and user input.

Second, we propose a chatbot layered on top of software repositories to answer software project related questions in order to showcase the potential of chatbots in software development. We find that practitioners are able to complete their tasks more accurately (65.6% more completed tasks) and in less time (83.3% faster) when using chatbots compared to using conventional tools. During this work we find two critical challenges in chatbot development which are selecting an NLU model for the chatbot implementation and curating a high-quality dataset to train the NLU model.

Third, we propose guidelines and approaches to improve chatbots in the software engineering domain. First, we assess the performance of multiple widely used NLUs using representative software engineering tasks to guide chatbot developers in designing more efficient chatbots. We report a guideline for chatbot developers on the best performing NLUs for intents classification and entity extraction. Finally, we investigate an approach that combines synonyms replacement and paraphrasing techniques to augment the training dataset of SE chatbots, which helps chatbot developers create high-quality datasets for training the NLU models. We find that augmenting the dataset using the combined approach does not improve the NLU's performance for intents classification. Also, the results show that using the combined approach has a negligible effect on the NLU's confidence in its classification.

Divisions:Concordia University > Gina Cody School of Engineering and Computer Science > Computer Science and Software Engineering
Item Type:Thesis (PhD)
Authors:Abdellatif, Ahmad
Institution:Concordia University
Degree Name:Ph. D.
Program:Software Engineering
Date:29 October 2021
Thesis Supervisor(s):Shihab, Emad
ID Code:990118
Deposited By: Ahmad Abdel Latif
Deposited On:16 Jun 2022 15:10
Last Modified:16 Jun 2022 15:10
All items in Spectrum are protected by copyright, with all rights reserved. The use of items is governed by Spectrum's terms of access.

Repository Staff Only: item control page

Downloads per month over past year

Research related to the current document (at the CORE website)
- Research related to the current document (at the CORE website)
Back to top Back to top