Designing an online and reliable statistical anomaly detection framework for dealing with large high-speed network traffic

Abstract

Despite a Network Anomaly Detection System (NADS) being capable of detecting existing and zero-day attacks, it is still not universally implemented in industry and real applications, with current systems producing high False Positive Rates (FPRs) and low Detection Rates (DRs). The challenges involved in designing a NADS architecture are 1) the methodology adopted for creating as comprehensive a profile as possible from diverse normal patterns and 2) in establishing an adaptive and lightweight Decision Engine (DE) which efficiently distinguishes between legitimate and anomalous activities at high speeds in large network environments. The need for such a method to be trained and validated on a decent dataset with the characteristics of current network environments is a significant challenge.

This thesis provides substantial contributions to research on the building of a scalable, adaptive and lightweight NADS framework. It considers three aspects: a data source, relevant features and observations, and new DE approaches for achieving a reliable NADS architecture.

The first key contribution is the creation of a new dataset called UNSWNB15 that has a hybrid of realistic modern legitimate and synthetic malicious activities, with statistical analyses and evaluations of it fully explained. Also, its complexity is assessed using existing techniques to demonstrate the extent of current sophisticated types of anomalous events.

The second core contribution is the development of a new theory for selecting important features and observations from network packets without redundancy to construct a legitimate profile from which any deviation is considered an attack, that is, establish an efficient NADS from analyses of the protocols and services of the OSI model.

The third major contribution is the development of two scalable frameworks with two new DE techniques for successfully detecting malicious activities in less processing times than current methods. These techniques, called the Geometric Area Analysis-ADS (GAA-ADS) and Dirichlet Mixture Model-ADS (DMM-ADS), are based on mixture models for modelling all possible normal patterns and detecting abnormal events that deviate from them using new outlier approaches.