Publication:
Enterprise Network Security via Data-driven Methods and Programmable Network Telemetry

dc.contributor.advisor Sivaraman, Vijay
dc.contributor.advisor Habibi Gharakheili, Hassan
dc.contributor.author Lyu, Minzhao
dc.date.accessioned 2022-01-30T22:48:07Z
dc.date.available 2022-01-30T22:48:07Z
dc.date.issued 2021
dc.description.abstract Enterprise networks are both complex and dynamic, with various kinds of servers (web, email, VPN, storage), clients (fixed, wireless), and Internet-of-Things devices (cameras, printers, sensors) being deployed, moved, and removed continuously. Furthermore, these assets are spread across various network segments (e.g., VLANs), often managed by different departments, with complex interconnection rules between segments, to public/private cloud services, and to the general Internet. It is therefore not surprising that organizational IT departments struggle to track their connected assets, monitor their operational health, understand the attack surface they expose, and protect them from external as well as internal threats. Current enterprise security systems such as Next-Generation-Firewalls (NGFW) and intrusion detection systems (IDS) are unable to cope with the growing volumes and diversity of emerging cyber-threats. Hardware appliance-based solutions are not just expensive, but also inflexible as their high-speed performance is optimized for relatively static rulesets. Software solutions on the other hand have great flexibility, but struggle to cope with high data rates which limit the granularity at which they analyze traffic for embedded threats. To advance the state-of-the-art of enterprise asset monitoring and distributed network attack detection, my thesis proposes a new approach that combines hardware performance with software flexibility, by leveraging the concepts of Programmable Network (PN) and Machine Learning (ML). Telemetry from Terabit-speed Programmable Switches is used to extract key attributes of traffic streams, and this is combined with ML models of enterprise asset behavior to monitor their health and to detect attacks. I make four key contributions. My first contribution focuses on the Domain Name System (DNS). I analyze DNS traffic from two large organizations to identify the behavioral aspects of various DNS assets. Using the behavioral attributes, I develop a clustering method to classify assets (e.g., recursive resolvers and authoritative name servers) and track their health through a set of well-articulated monitoring metrics. I demonstrate that my method successfully identifies over 100 key DNS assets in the two organizations and is further able to make recommendations on how these assets can be better secured against misuse. The second contribution extends my enterprise asset classification beyond DNS to include other asset types such as web servers, VPN servers, and file storage servers. For this, I develop a system that uses Programmable Network techniques to extract telemetry efficiently, feeds the attributes to a multi-grained ML-based scheme that classifies the assets in real-time, and reactively collects packet-level telemetry of suspicious hosts for forensics analysis. My method identifies hundreds of typical servers and thousands of less common assets (e.g., LDAP server and Redis proxy) across the two organizations. It additionally highlights instances of atypical behavior that provide advance warnings to IT staff on potentially anomalous assets. The third contribution detects DNS-based network attacks on enterprise hosts. To this end, I analyze incoming DNS traffic to the two organizations, and develop a hierarchical anomaly detection method that profiles incoming DNS traffic at various levels of hierarchy (e.g., host, subnet, and AS) to isolate DNS attackers that could be stealthy and distributed. The models I train detect DNS attacks in lab data with over 99% accuracy at each level of the hierarchy, and in a 1-month trial in the wild reveal hundreds of attacks that were missed by the organizational firewalls. My fourth contribution expands the attack detection from DNS to the whole dimension of network traffic. To achieve both detection effectiveness and operational practicality, I develop a multi-stage progressive inference architecture to optimally detect network attacks through a series of stages (e.g., active enterprise hosts, victims, distributed attackers, and malicious flows) each with increasing telemetry cost but narrowing focus. Evaluations using real distributed denial-of-service (DDoS) attacks and large-scale enterprise traffic traces demonstrate the ability of my system in detecting distributed network attacks to the finest flow-level with practically low computational costs as around 30% CPU and 8% RAM usage on a typical blade server, which is not achievable by its counterpart solutions. Taken together, my contributions apply Programmable Network and Machine Learning to develop new practical and effective ways that give enterprise IT departments continuous visibility of their assets, advance warning of the threat surface they expose, and real-time alarms when network attacks unfold.
dc.identifier.uri http://hdl.handle.net/1959.4/100046
dc.language English
dc.language.iso en
dc.publisher UNSW, Sydney
dc.rights CC BY 4.0
dc.rights.uri https://creativecommons.org/licenses/by/4.0/
dc.subject.other Network Security
dc.subject.other Enterprise Network Management
dc.subject.other Host Classification
dc.subject.other Distributed Attack Detection
dc.subject.other Machine Learning
dc.subject.other Programmable Networks
dc.title Enterprise Network Security via Data-driven Methods and Programmable Network Telemetry
dc.type Thesis
dcterms.accessRights open access
dcterms.rightsHolder Lyu, Minzhao
dspace.entity.type Publication
unsw.accessRights.uri https://purl.org/coar/access_right/c_abf2
unsw.contributor.advisorExternal Russell, Craig; CSIRO's Data61
unsw.date.embargo 2023-01-01
unsw.description.embargoNote Embargoed until 2023-01-01
unsw.identifier.doi https://doi.org/10.26190/unsworks/1955
unsw.relation.faculty Engineering
unsw.relation.school School of Electrical Engineering and Telecommunications
unsw.relation.school School of Electrical Engineering and Telecommunications
unsw.relation.school School of Electrical Engineering and Telecommunications
unsw.subject.fieldofresearchcode 400602 Data communications
unsw.subject.fieldofresearchcode 460407 System and network security
unsw.thesis.degreetype PhD Doctorate
Files
Original bundle
Now showing 1 - 1 of 1
Thumbnail Image
Name:
public version.pdf
Size:
16.14 MB
Format:
application/pdf
Description:
Resource type