Deep Learning for Anomaly Detection

Abstract

Anomaly detection has been widely studied and used in diverse applications. Building an effective anomaly detection system requires researchers and developers to learn complex structure from noisy data, identify dynamic anomaly patterns, and detect anomalies with limited labels. Recent advancements in deep learning techniques have greatly improved anomaly detection performance, in comparison with classical approaches, and have extended anomaly detection to a wide variety of applications. This tutorial will help the audience gain a comprehensive understanding of deep learning-based anomaly detection techniques in various application domains. First, we give an overview of the anomaly detection problem, introducing the approaches taken before the deep model era and listing out the challenges they faced. Then we survey the state-of-the-art deep learning models that range from building block neural network structures such as MLP, CNN, and LSTM, to more complex structures such as autoencoder, generative models (VAE, GAN, Flow-based models), to deep one-class detection models, etc. In addition, we illustrate how techniques such as transfer learning and reinforcement learning can help amend the label sparsity issue in anomaly detection problems and how to collect and make the best use of user labels in practice. Second to last, we discuss real world use cases coming from and outside LinkedIn. The tutorial concludes with a discussion of future trends.

The previous version of this tutorial has been presented for WSDM2020, on Feb 3, 2020, at Houston, USA (WSDM tutorial website). Compared to the previous WSDM tutorial, this version puts more emphasis on the following aspects:

  1. We give a more comprehensive overview of deep model techniques for anomaly detection, updated with new contents in the latest methodologies such as flow-based generative deep models.

  2. We emphasize in this tutorial on how modern techniques help address the most practical challenge of anomaly detection --- lack of user labels. Specifically, we discuss data augmentation techniques to augment the limited labels, how to improve model robustness and avoid the impact of noisy labels, and how to collect labels in anomaly detection systems.

  3. In addition to methodologies, we would provide case studies of deep anomaly detection in practice, e.g., anomaly detection for self-driving cars, in addition to the time series detection deployed at LinkedIn.

Tutorial Outline

Part 1. Introduction (30 min)

    • 1.1. Overview of Anomaly Detection

    • 1.2. Anomaly Detection Application and Challenges

    • 1.3. Traditional Techniques and Motivation for Deep Learning

Part 2. Deep Learning for Anomaly Detection (90 min)

    • 2.1. Basic Building Blocks

      • a. MultiLayer Perceptron (MLP)

      • b. Convolutional Neural Networks (CNN)

      • c. Recurrent Neural Networks (RNN)

    • 2.2 Fundamental Model Structures Applied to Anomaly Detection Tasks

      • a. Deep One-Class Models (Deep OC)

      • b. AutoEncoder (AE)

      • c. Variational AutoEncoder (VAE)

      • d. Generative Adversarial Networks (VAE, GAN, Flow-based)

    • 2.3. Compensate for Sparse Labels

      • a. Integrated Semi-Supervised Learning

      • b. Data Augmentation and Transfer Learning

Part 3. Real-world Applications for Anomaly Detection (50 min)

    • 3.1 Anomaly Detection for Autonomous Vehicle Development

    • 3.2 Anomaly Detection at LinkedIn

      • Algorithms and Evaluation

      • System Architecture

      • Usability in Production

Part 4. Conclusion and Future Trends (10 min)


Presenters

Dr. Ruoying Wang is an AI software engineer at LinkedIn. She works on applying deep learning and statistical models for anomaly detection and capacity planning. She obtained her Ph.D. in Economics degree from UBC, focusing on empirical causal analysis with applications in International Trade. She is excited to develop deep learning algorithms for anomaly detection in production at LinkedIn.

Kexin Nie is a Sr. AI software engineer at LinkedIn, where she leads the effort to monitor AI models' online performance drift and automatic diagnose issues' root causes at scale. She has been working in the field of anomaly detection for 2+ years and launched several algorithms in LinkedIn's health monitoring service (ThirdEye). Before this, she worked for IBM to optimize its E-commercial ads' tagging. She obtained her Master of Statistics from Stanford University.

Dr. Yen-Jung Chang is a Sr. AI Software Engineer at LinkedIn. He leads the efforts on building the company-wise distributed training architecture for deep learning and the anomaly detection system for large-scale time series and metrics. He is also conducting research regarding automated data analytics of data cubes. He received his Ph.D. in Electrical and Computer Engineering at UT-Austin focusing on distributed and parallel computing, verification and debugging of distributed systems, and lattice theory.

Dr. Xinwei Gong is a Staff Software Engineer at LinkedIn, where he works on deep learning models and their applications in anomaly detection. He obtained his Ph.D. in physics from Duke University, focusing on network theory and nonlinear dynamics. Before LinkedIn, he worked on deep learning algorithms in autonomous vehicle development at Volkswagen Group Research and machine learning problems in healthcare at a biotech startup.

Dr. Tie Wang leads the AI Quality Foundation team at LinkedIn. His team owns the anomaly detection algorithm library, that supports anomaly detection for over 20 LinkedIn products. He has broad interests in machine learning/AI and its applications. He has 12 years of R\&D experience at Apple, Microsoft, LinkedIn on anomaly detection, query understanding, commercial and web search ranking algorithms and systems. He received his Ph.D. in Computer Science from Arizona State University. He has published in top journals and conferences including KDD, IEEE Transaction on Signal Process.

Dr. Yang Yang is a Senior Staff Software Engineer and Tech Lead at LinkedIn. Before joining LinkedIn, Yang worked at Yahoo! Labs as a Scientist. She obtained her Ph.D. degree at the Department of Statistics, University of Michigan. She has produced various papers and patents on applying statistical methods and machine learning approaches to real data problems involving large scale data. She has published in conferences and journals including KDD, WWW, PAM, Statistical Analysis and Data Mining, The Canadian Journal of Statistics, IIE Transactions on Healthcare Systems Engineering, and Statistical Analysis for High-Dimensional Data.

Dr. Bo Long is a Director of AI Engineering at LinkedIn, leading LinkedIn's AI Foundations team. He has 15 years of experience in data mining and machine learning with applications to web search, recommendation, and social network analysis. He holds dozens of innovations and has published peer reviewed papers in top conferences and journals including ICML, KDD, ICDM, AAAI, SDM, CIKM, and KAIS. He has served as reviewers, workshops co-organizers, conference organizer committee members, and area chairs for multiple conferences, including KDD, NIPS, SIGIR, ICML, SDM, CIKM, JSM etc.