Abstract

This poster presents a study on deploying and evaluating a conversational agent using Retrieval-Augmented Generation (RAG) and Large Language Models (LLMs) in an academic setting. The objective was to implement a RAG-based system capable of answering reference questions and develop an evaluation protocol to measure the "usefulness" of the chatbot, comparing multiple models. The system follows a two-step approach: it first retrieves relevant documents from a curated knowledge base and then generates accurate, context-aware responses using LLMs. The evaluation protocol involved grading responses across five dimensions: accuracy, groundedness, elicitation, completeness, and further assistance. The results highlighted that while RAG significantly reduced hallucinations, challenges remained in preventing them completely. The evaluation rubric effectively differentiated between the performances of various models, despite the subjective nature of the grading process. This work emphasizes the integration of RAG and LLMs in academic reference services to provide responsible, reliable, and user-centered AI responses.