Abstract

Research on LLM-based code generation is growing but often overlooks the impact of Software Engineering (SE) knowledge, such as software process models and non-functional requirements (NFRs). This thesis explores how integrating SE practices with LLMs can help bridge this gap.
We first propose FlowGen to explore the impact of software process models on code generation. We assign LLM agents to different development roles and study three models: FlowGenWaterfall, FlowGenTDD, and FlowGenScrum. We evaluate FlowGen using GPT-3.5, comparing it against several baselines on four code benchmarks (i.e., HumanEval, HumanEval-ET, MBPP, and MBPP-ET). Our results show that FlowGenScrum outperforms the other process models, achieving a 15% improve- ment in Pass@1 over RawGPT on average. Integrating a state-of-the-art technique (i.e., CodeT) further boosts Pass@1 scores. Our findings also show that development activities impact code qual- ity differently, and FlowGen enhances result stability across LLM versions and temperature settings.
Secondly, we investigate how variations in developer behavior affect how LLMs address NFRs (e.g., expressing the same NFR using different wording). Robust LLMs should generate consistent code despite such variations. We propose RoboNFR to evaluate LLM robustness in NFR-aware code generation across four key dimensions—code design, performance, readability, and reliabil- ity—using three methodologies: prompt variation, regression testing, and diverse workflows. Our experiments show that RoboNFR effectively reveals robustness issues in tested LLMs. Overall, across the three methodologies, incorporating NFRs tends to reduce Pass@1 scores while improv- ing NFR-specific metrics, but also increases the standard deviation in both correctness and quality.
This thesis highlights the significant influence of software process models and NFRs, empha- sizing the need for future work to incorporate such SE knowledge in the era of LLMs.