Abstract

Anomaly detection systems are important tools for security. Unlike signature-based systems, anomaly detection can be used to detect new attacks for which signatures are now available.

To this end, anomaly detection techniques rely on machine learning techniques to model the normal behaviour of the system. This model is used as a baseline for the detection of anomalies during system operation.

The problem is that there is no one machine learning technique that can provide good accuracy. What we need is to combine multiple techniques. This is because ensemble methods have been used to improve the overall detection accuracy in traditional machine learning.

The combination consists of combining the outputs of several accurate and diverse models. To reduce the number of combination, and hence improve the efficiency of combination, in this thesis, we propose PBC (Pruning Boolean Combination), an efficient approach for selecting and combining anomaly detectors. PBC relies on two novel pruning techniques that we have developed to prune redundant and trivial detectors. Compared to existing work, PBC reduces significantly the number of detectors to combine, while keeping similar accuracy. We show the effectiveness of PBC when applying it to benchmarks data sets.

Much of the content of this thesis is adapted and expanded from a paper published at the 2015 IEEE International Conference on Software Quality, Reliability and Security (QRS). QRS is a merger of the SERE conference (IEEE International Conference on Software Security and Reliability) and the QSIC conference (IEEE International Conference on Quality Software)