Abstract

In this thesis, we focus on the problem of detecting anomalies in large log data. Logs are generated at runtime and contain a wealth of information, useful for various software engineering tasks, including debugging, performance analysis, and fault diagnosis. Our anomaly detection approach is based on the multiresolution abnormal trace detection algorithm proposed in the literature. The algorithm exploits the causal relationship of events in large execution traces to build a model that represents the normal behaviour of a system using varying length n-grams and a generalizable automaton. The resulting model is later used to detect deviations from normalcy.
In this thesis, we investigate the application of this algorithm in detecting anomalies in log data. Logs and execution traces are different. Unlike traces, logs do not exhibit a causal relationship among their events, raising questions as to the effectiveness of automata to model log data for anomaly detection. Logs are unstructured data and hence require the use of parsing and abstraction techniques.
We propose a process, called LogAutomata, which uses the multiresolution abnormal trace detection algorithm as its primary mechanism. When applying LogAutomata to a large log file generated from the execution of Hadoop Distributed File System (HDFS), we show that the multiresolution algorithm can be a very effective way to detect anomalies in log data.