Abstract

Artificial Intelligence for IT Operations (AIOps) is revolutionizing IT management by incorporating AI, machine learning, and big data analytics to automate and enhance system operations. Logs are the backbone of AIOps—they provide the fundamental data on system events, user activities, and performance metrics that AIOps needs for proactive monitoring, predictive analytics, and maintaining system health. Log management, especially log parsing, is therefore critical for identifying anomalies, diagnosing failures, and ensuring overall operational efficiency. However, challenges such as diverse log formats, insufficient logging guidelines, the sheer volume of logs, and the need for real-time insights significantly limit the precision, scalability, and effectiveness of AIOps.
This thesis develops novel methods for universal log parsing, introduces accurate evaluation metrics, proposes a comprehensive taxonomy of log characteristics, and addresses privacy compliance, collectively advancing the efficacy, scalability, and trustworthiness of AIOps.
Inaccurate log parsing can lead to inaccurate insights—misleading or incorrect conclusions drawn from analysis. Such errors may cause AIOps to misclassify events, overlook crucial anomalies, or generate noise that obscures genuine issues. To address this, the first major contribution of this thesis is the development of ULP (Universal Log Parser), which leverages a frequent token counting method to identify recurring patterns and extract log templates efficiently. By reducing computational complexity, ULP enables faster, more accurate log parsing, making it highly effective for large-scale IT environments—a key capability for the automation and responsiveness required in AIOps.
The second contribution is AML (Accuracy Metric for Log Parsing), a structured framework for evaluating the accuracy of log parsing. Traditional metrics are insufficient for heterogeneous log formats, leading to errors and subsequent misguided AIOps decisions. AML offers nuanced metrics that account for both omission and commission errors, enabling detailed and reliable comparisons across different log parsers.
The third contribution is a taxonomy of log characteristics, categorizing logs based on their structural and contextual properties. This taxonomy not only guides parser design by clarifying how logs differ across applications but also informs logging practices, helping practitioners tailor log writing strategies for better analytics.
The fourth contribution focuses on enhancing log privacy compliance, a crucial aspect in AIOps —especially as automated processes handle large volumes of sensitive log data. The thesis provides guidelines for evaluating and managing privacy risks associated with log data, ensuring that the automation capabilities of AIOps remain compliant with stringent privacy regulations and best practices.
Together, these contributions form a framework for advancing log parsing within AIOps. It is a holistic approach—encompassing efficient universal parsing, robust accuracy evaluation, a guiding taxonomy, and built-in privacy compliance—that addresses the entire life cycle of log-based analytics. This framework ultimately strengthens the capabilities of IT operations to be more proactive, responsive, and compliant, paving the way for the next generation of AIOps-driven IT management.