Deep Learning for Dynamical System (ODE & PDE) Seminar

  • Time: Monday 9:30 pm (Beijing Time)

  • Zoom: https://duke.zoom.us/j/95028664381?pwd=NytoMmlNaGg2c2lqcXlSdVZmTDdyZz09

  • Meeting ID: 950 2866 4381

Jan. 18th

Speaker: Dr. Gu Yiqi

Affiliation: National University of Singapore

Title: The discovery of dynamics via deep learning

Abstract: Identifying hidden dynamics from observed data is a significant and challenging task in a wide range of

applications. Recently, the combination of linear multistep methods (LMMs) and deep learning has been successfully

employed to discover dynamics. We put forward an error estimate for the deep network-based LMMs using the approximation property of ReLU networks. It indicates, for certain families of LMMs, that the l2 grid error is of O(h^p) where h is the time step size and p is the local truncation error order, as long as the network size is sufficiently large. Moreover, the numerical results of some physically relevant examples are consistent with our theory.

Jan. 25th

Speaker: Ph.D. Candidate Yihao Hu

Affiliation: University of Notre Dame

Title: NEURAL-PDE a spatiotemporal deep learning model for time-dependent PDEs

Abstract: Partial differential equations (PDEs) play a crucial role in studying a vast number of problems in science and engineering. Numerically solving nonlinear and/or high-dimensional PDEs is frequently a challenging task. Inspired by the traditional finite difference and finite element methods and emerging advancements in machine learning, we propose a sequence-to-sequence learning (Seq2Seq) framework called Neural-PDE, which allows one to automatically learn governing rules of any time-dependent PDE system from existing data by using a bidirectional LSTM encoder and predict the solutions in next n time steps. One critical feature of our proposed framework is that the Neural-PDE is able to simultaneously learn and simulate all variables of interest in a PDE system. We test the Neural-PDE by a range of examples, from one-dimensional PDEs to a multi-dimensional and nonlinear complex fluids model. The results show that the Neural-PDE is capable of learning the initial conditions, boundary conditions, and differential operators defining the initial-boundary-value problem of a PDE system without the knowledge of the specific form of the PDE system. In our experiments, the Neural-PDE can efficiently extract the dynamics within 20 epochs training and produce accurate predictions. Furthermore, unlike the traditional machine learning approaches for learning PDEs, such as CNN and MLP, which require a great number of parameters for model precision, the Neural-PDE shares parameters among all-time steps, and thus considerably reduces computational complexity and leads to a fast learning algorithm.

Mar. 1st

Speaker: Ph.D. Yiwei Wang

Affiliation: Illinois Institute of Technology

Title: Energetic Variational Inference

Abstract: We introduce a new variational inference (VI) framework, called energetic variational inference (EVI). It minimizes the VI object function based on a prescribed energy-dissipation law. Using the EVI framework, we can derive many existing Particle-based Variational Inference (ParVI) methods, including the popular Stein Variational Gradient Descent (SVGD) approach. More importantly, many new ParVI schemes can be created under this framework. For illustration, we propose a new particle-based EVI scheme, which performs the particle-based approximation of the density first and then uses the approximated density in the variational procedure, or \Approximation-then-Variation" for short. Thanks to this order of approximation and variation, the new scheme can maintain the variational structure at the particle level, and can significantly decrease the KL-divergence in each iteration. Numerical experiments show the proposed method outperforms some existing ParVI methods in terms of fidelity to the target distribution.

Mar. 8th

Speaker: Prof. Yuling Jiao

Affiliation: Wuhan University

Title: Generative Learning with Euler Particle Transport

Abstract: We propose an Euler particle transport (EPT) approach for generative learning. EPT is motivated by the problem of constructing the optimal transport map from a reference distribution to a target distribution characterized by the Monge-Ampere equation. Interpreting the infinitesimal linearization of the Monge-Ampere equation from the perspective of gradient flows in measure spaces leads to a stochastic McKean-Vlasov equation. We use the forward Euler method to solve this equation. The resulting forward Euler map pushes forward a reference distribution to the target. This map is the composition of a sequence of simple residual maps, which are computationally stable and easy to train. The key task in training is the estimation of the density ratios or differences that determine the residual maps. We estimate the density ratios (differences) based on the Bregman divergence with a gradient penalty using deep density-ratio fitting. We show that the proposed density-ratio estimators do not suffer from the “curse of dimensionality” if data is supported on a lower-dimensional manifold. Numerical experiments with multi-mode synthetic datasets and comparisons with the existing methods on real benchmark datasets support our theoretical results and demonstrate the effectiveness of the proposed method.

Mar. 15th

Speaker: Prof. Jia Zhao ,Assistant Professor of Mathematics, Utah State University

Title：Solving and learning phase field models using the modified Physics Informed Neural Networks

Email: jia.zhao@usu.edu

Abstract： Phase field models, including the Allen-Cahn type and Cahn-Hilliard type equations, have been widely used to investigate interfacial dynamic problems. Designing accurate, efficient, and stable numerical algorithms for solving the phase field models has been an active field for decades. In the meanwhile, developing reliable and physically consistent phase field models for applications in science and engineering have also been intensively investigated.

In this talk, we introduce some preliminary results on solving and learning phase field models using deep neural networks. In the first part, we focus on using the deep neural network to design an automatic numerical solver for the Allen-Cahn and Cahn-Hilliard equations by proposing an adaptive physics informed neural network (PINN). In particular, we propose to embrace the adaptive idea in both space and time and introduce various sampling strategies, such that we are able to improve the efficiency and accuracy of the PINN on solving phase field equations. In the second part, we introduce a new deep learning framework for discovering the phase field models from existing image data. The new framework embraces the approximation power of physics informed neural networks (PINN), and the computational efficiency of the pseudo-spectral methods, which we named pseudo-spectral PINN or SPINN. We will illustrate its approximation power by some interesting examples.

March 22th

Speaker: Prof. Kelin Xia, Assistant Professor, School of Physical and Mathematical Sciences, School of Biological Science, Nanyang Technological University

Title：Topological data analysis (TDA) based machine learning models for drug design

Abstract：Effective molecular representation is key to the success of machine learning models for drug design. In this talk, we will discuss a series of TDA-related models, including persistent homology, persistent spectral models, and persistent Ricci curvature and their combination with machine learning models. Unlike traditional graph/network or geometric models, these filtration-induced persistent models can characterize the multiscale intrinsic information, thus significantly reduces molecular data complexity and dimensionality. Feature vectors are obtained from various persistent attributes and inputted into machine learning models, in particular, random forest, gradient boosting tree and convolutional neural network. Our persistent representations based molecular fingerprints can significantly boost the performance of learning models in drug design.

March 29th

Speaker: Jingrun Chen (Soochow University)

Title: A deep mixed residual method for solving high-order partial differential equations

Abstract: In recent years, a significant amount of attention has been paid to solve partial differential equations (PDEs) by deep learning. For example, deep Galerkin method uses the PDE residual in the least-squares sense as the loss function and a deep neural network (DNN) to approximate the PDE solution. In this work, we propose a deep mixed residual method (MIM) to solve PDEs with high-order derivatives. In MIM, we first rewrite a high-order PDE into a first-order system, very much in the same spirit as local discontinuous Galerkin method and mixed finite element method in classical numerical methods for PDEs. We then use the residual of first-order system in the least-squares sense as the loss function, which is in close connection with least-squares finite element method. Numerous results of MIM with different loss functions and different choice of DNNs are given. In most cases, MIM provides better approximations (not only for high-derivatives of the PDE solution but also for the PDE solution itself) than DGM with nearly the same DNN and the same execution time, sometimes by more than one order of magnitude. When different DNNs are used, in many cases, MIM provides even better approximations than MIM with only one DNN, sometimes by more than one order of magnitude. Numerical results also show some interesting connections between MIM and classical numerical methods. Therefore, we expect MIM to open up a possibly systematic way to understand and improve deep learning for solving PDEs from the perspective of classical numerical analysis.

April 27th

Speaker: Dong Wang (The Chinese University of Hong Kong, Shenzhen)

Title: Consistency of archetypal analysis

Abstract: Archetypal analysis is an unsupervised learning method that uses a convex polytope to summarize multivariate data. For fixed k, the method finds a convex polytope with k vertices, called archetype points, such that the polytope is contained in the convex hull of the data and the mean squared distance between the data and the polytope is minimal. In this talk, we prove a consistency result that shows if the data is independently sampled from a probability measure with bounded support, then the archetype points converge to a solution of the continuum version of the problem, of which we identify and establish several properties. We also obtain the convergence rate of the optimal objective values under appropriate assumptions on the distribution. If the data is independently sampled from a distribution with unbounded support, we also prove a consistency result for a modified method that penalizes the dispersion of the archetype points. Our analysis is supported by detailed computational experiments of the archetype points for data sampled from the uniform distribution in a disk, the normal distribution, an annular distribution, and a Gaussian mixture model.

May 3rd

Speaker: Haizhao Yang (Purdue University)

Title: Reproducing Activation Functions for Deep Learning

Abstract: Deep learning is a powerful tool not only in computer science and data science but also in scientific computing. It has led to numerous breakthroughs in science and engineering. This talk introduces newly developed reproducing activation functions as a simple but very efficient technique to boost the performance of deep learning. Both theoretical insights and numerical evidence are provided to demonstrate the effectiveness of reproducing activation functions in terms of approximation theory and optimization analysis.