Enabling User-Driven Rule Management in Event Data Analysis

Abstract

Increasingly in the information age, overwhelming quantities of available data has brought about opportunities as well as difficulties. Data analysis is of considerable importance to finding interesting patterns, discovering useful information and making decisions accordingly. Event data has unique characteristics including temporal dependencies, high flow rate and huge volume, which makes it more difficult to analyse than other data types. Unlike data analysts working in large companies that have IT staff and expensive software infrastructure, those working in the research sector find it more difficult to efficiently manage event data analysis by themselves. User-driven rule management is a particular challenge especially when analysis rules increase in size and complexity over time. This thesis addresses these problems by proposing a new architecture called EP-RDR aimed at enabling data analysts, with no IT experience, to manage their event data analysis. EP-RDR enables complex event processing and facilitates user-driven rule set evolution according to changing requirements. The architecture leverages event processing system (EPS) technology with a rule-based method called Ripple-Down Rules (RDR). EP-RDR has two main components: an RDR component playing the role of managing the event processing logic and of supporting incremental rule insertion that enables data analysts to define and add rules by themselves, and an EPDaaS (Event Pattern Detection as a Service) component that can invoke any EPS so that data analysts are able to conduct event processing without concern about which EPS to use. To facilitate the interoperability between components in EP-RDR, this thesis also proposes an Event Data Modelling Framework (EDMF) to assist in building data models for any application of EP-RDR. EDMF consists of an Event Data Meta-Model and its associated Operational Guidelines. Any data model built based on EDMF allows event pattern types to be defined, abstracting existing event and event pattern occurrence representation formats in a consistent manner. Finally, to evaluate the proposed new method, a prototype has been implemented and applied on real-life scenarios involving financial market data pre-processing. This case study shows that the proposed method effectively satisfies requirements of event data analysis, namely feasibility and interoperability, the capability of complex event processing, and the capability of user-driven rule set evolution.