Diffusion Models and Differentiable Simulations
Abstract:
This talk focuses on the possibilities that arise from recent advances in the area of deep learning for physical simulations. In particular, it will focus on diffusion modeling and differentiable physics solvers. These solvers provide crucial information for deep learning tasks in the form of gradients, which are especially important for time-dependent processes. Also, existing numerical methods for efficient solvers can be leveraged within learning tasks. This paves the way for hybrid solvers in which traditional methods work alongside pre-trained neural network components. In this context, diffusion models and score matching will be discussed as powerful building blocks for training probabilistic surrogates. The capabilities of the resulting methods will be illustrated with examples such as wake flows and turbulent flow cases.
Bio:
My research targets deep learning methods for physical simulations. The key question here is: How can we leverage our existing knowledge about numerical methods to improve our understanding of physical systems with the help of deep learning. In this context, I believe that neural networks are an exciting area for research, even more so when going beyond the established paths of pattern recognition problems. E.g., trained deep nets are able to learn structures of numerical errors of PDEs for which we have no analytical formulations, and they’re able to anticipate the dynamics of complex physical systems. There are many fascinating topics left to explore here.
Summary:
Focus:
Differentiable simulation via deep learning methods
Fluid dynamics: gases, liquids
Given: simulation that accurately captures the dynamics of the physical system
Want:
Infer the free/unknown parameters of the simulation
I.e. invert the simulation: get parameters/inputs from the outputs
Diffusion models:
y=f(x): approximated via some neural net
Probabilistic gaussian process of noise
Learn the inverse process Infer x from y
Bayesian Diffusion Models:
Prior work:
Learned surrogate: input->output neural net
Heteroschedastic model
BNN: Bayesian distribution of neural nets
DDPN (new contribution):
Bayesian diffusion model, related to diffusion models for videos
Probability distribution conditioned on prior state of the simulation
Bayesian estimation process is based on iterative refinement
Sequence of Bayesian approximations
More expensive than one-phase Bayesian approximation
Far more accurate across different Reynolds Numbers of flows
Differentiable simulations
Train neural networks on sample runs of PDE-based simulation
Unroll the simulation runs so the model prediction from time step t is evaluated on prediction accuracy many time steps in the future (e.g. t+64)
This requires many simulations runs and expensive neural network training
But the resulting neural network PDE simulation is much faster than original
Diffusion models for differentiable simulation
Combine above ideas
SMDP: Use Bayesian Diffusion models to model physical simulation
Score matching helps improve stability of the inference
Deterministic networks can capture mean behavior but not the distribution of possible outcomes
Use of Bayesian approach makes it possible to capture the full posterior distribution. Critical when mean is a poor measure of the distribution (e.g. multi-modal distribution)