Business process improvement with performance-based sequential experiments

Abstract

Various lifecycle approaches to Business Process Management (BPM) have a common assumption that a process is incrementally improved in the redesign phase. While this assumption is hardly questioned in BPM research, there is evidence from the field of AB testing that improvement concepts often do not lead to actual improvements. If incremental process improvement can only be achieved in a fraction of the cases, there is a need to rapidly validate the assumed benefits.

Contemporary BPM research does not provide techniques and guidelines on testing and validating the supposed improvements in a fair manner. In this research, we address these challenges by integrating business process execution concepts with ideas from a set of software engineering practices known as DevOps. We propose a business process improvement methodology named AB-BPM, and a set of techniques that allow us to enact the steps in this methodology.

As a first technique, we develop a simulation technique that estimates the performance of a new version in an offline setting using historical data of the old version. Since the results of simulation can be speculative, we propose shadow testing as the next step. Our Shadow testing technique partially executes the new version in production alongside the old version in such a way that the new version does not throttle the old version. Finally, we develop techniques that offer AB testing for redesigned processes with immediate feedback at runtime. AB testing compares two versions of a deployed product (e.g., a Web page) by observing users responses to versions A/B, and determines which one performs better. We propose two algorithms, LTAvgR and ProcessBandit, that dynamically adjust request allocation to two versions during the test based on their performance.