Model validation using deep generation of stress data

A tutorial at the ACM conference on Economics and Computation 2023

Description

Many businesses, such as financial institutions, insurance companies, retailers, use predictive models to make appropriate decisions. The effectiveness of models relies strongly on the assumption that the train and test data are sampled from the same distribution or at least similar distribution, however, the dependencies in the data usually change over time in real life. The described problem is related to the ability of models to resist dataset shifts. Some techniques were proposed by researchers to assess the predictive ability of forecasting models, but the problem is the lack of data diversity, as the data history is relatively short in many cases. Data generation is a relatively new field of machine learning that explores methods for generating realistic data preserving important properties and dependencies in train data. In this tutorial, we aim to present validation methods of forecasting models based on synthetic multivariate time series data. These methods help to understand the behavior of models in case of unforeseen events.

The tutorial is aimed to present a novel procedure for model robustness validation. The key idea is to apply deep generative models that can sample unlikely events and introduce slight shifts in input data. Such an approach allows us to examine models’ reactions to input data with different levels of the likelihood. The growing number of running models highlights the importance of model validation, which determines the process of verifying the validity of input and output data, model’s performance, stability, and interpretability.

Currently, machine learning models are part of the complex pipelines in different domains and have many more possible applications in sensitive fields like medicine, production, insurance, banking and so on. In that case, we need to estimate the possible risks of the model deployment. It helps to reduce the costs of model execution in unstable environments, perform model monitoring and develop strategies with the proper reaction to specific scenarios, and reduce the costs of model refitting and adaptation.

Target audience and prerequisite knowledge

The target audience is risk managers and researchers who are interested in applying deep generative models for evaluating the model robustness. We assume the audience has basic knowledge in machine learning, probability theory, statistics and python programming.

Schedule (June 22)

Tutorial part 1: Background (10:30am-11:15am ET), Notebook

The definition of data distribution shifts.
The worst-case risk approach to estimate the potential decrease in the target metric.

Tutorial part 2: Deep generation of stress data (12:00pm-12:45pm ET), Notebook

Considering deep generative models: Generative Adversarial Networks.
Discussion of advantages and disadvantages of deep generative models.

Organizers

Artificial Intelligence Research Institute (AIRI), Moscow, Russia.

Vitaliy Pozdnyakov, researcher

Vitaliy Pozdnyakov has 7 years of experience in industrial companies as a developer of enterprise resource planning systems and 3 years in scientific research of industrial artificial intelligence methods. His master's thesis is devoted to probabilistic forecasting of multidimensional time series using deep generative models.

Dmitrii Kiselev, researcher

Dmitry Kiselev is a PhD with a solid engineering and consulting background applying machine learning and data science techniques to production in travel and financial industries.

Alexander Kovalenko, researcher

Alexander Kovalenko is an engineer specializing in microelectronics and solid-state electronics. He has more than 12 years of experience working with industrial equipment as an electronics engineer. His main area of expertise is applications of machine learning in industry, such as graph neural networks for fault detection and diagnosis based on data from multiple sensors.

Page updated

Google Sites

Report abuse