The research topic of Multivariate Time Series (MVTS) anomaly detection has received significant attention from both industry and academia. However, upon careful examination of the literature, it becomes apparent that the community lacks organization compared to other machine learning communities, such as Computer Vision (CV) and Natural Language Processing (NLP). Additionally, many proposed solutions have been evaluated using flawed protocols that lack scientific foundation. One popular protocol, the \\pa protocol, is so flawed that a random guess can outperform all developed algorithms to date.
In this paper, we aim to address these issues by reviewing and evaluating recent algorithms using more robust protocols. We discuss the weaknesses of commonly used protocols in the context of MVTS anomaly detection and propose methods to mitigate them. Furthermore, we express concerns regarding benchmark datasets, experiment design, and evaluation methodology observed in many works.
Additionally, we present a simple yet challenging baseline algorithm based on Principal Components Analysis (PCA) that surprisingly outperforms many Deep Learning (DL) based approaches on popular benchmark datasets. Our main objective is to stimulate more effort towards important aspects of research such as data, experiment design, evaluation methodology, and result interpretability, rather than solely focusing on the design of increasingly complex and “fancier” algorithms.