Abstract

Security incident analysis poses a difficult challenge for security operation centers, because they must respond to an overwhelming number of alerts, requiring analysis of large volumes of data, a myriad of tools, while facing shortages of experienced analysts. The job of an analyst is further complicated as incidents are dynamic and require multifaceted, multi-step analysis. While companies are keen to apply Large Language Models (LLM) to augment analysts’ efforts in security incident analysis (SIA), the lack of benchmarking of LLMs for SIA renders huge risks on their overall effectiveness and design choices. Moreover, such benchmarking becomes non-trivial as: (i) no dataset currently exists in a digestible format for LLMs that covers a wide range of SIA tasks; (ii) considering the vast diversity in analysts’ job, there is a continuous need to add new tasks; and (iii) frequent model releases must be included in evaluation. In this thesis, we aim to address these challenges while building an agentic evaluation framework, SIABENCH. Specifically, first, we build a first-of-its-kind dataset that includes two major SIA tasks: (i) deep analysis of security incidents (25 scenarios) and (ii) alert triaging (35 scenarios). Second, we build an agent to automatically conduct a wide range of SIA tasks (covering network/memory forensics, malware analysis in binary/code/PDF, phishing email/kit analysis, and log analysis) along with false alert detection. Third, we evaluate the performance of nine major LLMs (covering both open- and closed-weight) in SIA with the capability to support newer models and tasks.