Summer 2022 (Jul - Aug)

[08/29/2022] Modeling SGD with (Stochastic/Partial) Differential Equations

On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs) (Li et al., NeurIPS 2021)

Presenter: Hojung Jung (OSI Lab @KAIST AI)
Slide and Paper

Mean-field theory for shallow neural networks

Presenter: Hanseul Cho (OptiML Lab @KAIST AI)
Slide
Papers presented:
- A mean field view of the landscape of two-layer neural networks (Mei et al., PNAS 2018)
- Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit (Mei et al., COLT 2019)

[08/22/2022] PINN & Loss Landscapes

Physics-Informed Neural Networks (Raissi et al., Journal of Computational Physics 2019)

Presenter: Minchan Jeong (OSI Lab @KAIST AI)
Slide and Paper

Visualizing the Loss Landscape of Neural Nets (Li et al., NeurIPS 2018)

Presenter: Yujeong Jung (Samsung Research)
Slide and Paper

[08/08/2022] Conservatives in Deep Learning Dynamics - Part 2

Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks (Tanaka & Kunin, NeurIPS 2021)

Presenter (online, not recorded): Yeongwoo Song (CSSPL @KAIST Physics)
Slide and Paper

[07/25/2022] Conservatives in Deep Learning Dynamics - Part 1 & Trajectory-Dependent Analysis of Stochastic Optimizers

Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics (Kunin et al., ICLR 2021)

Presenter (online, not recorded): Yeongwoo Song (CSSPL @KAIST Physics)
Slide and Paper

Deriving Generalization Bounds with Optimization Trajectory

Presenter (offline): Hanseul Cho (OptiML Lab @KAIST AI)
Slide
Papers presented:
- Hausdorff dimension, heavy tails, and generalization in neural networks (Şimşekli et al., NeurIPS 2020; Journal of Statistical Mechanics 2021)
- Generalization Bounds using Lower Tail Exponents in Stochastic Optimizers (Hodgkinson et al., ICML 2022)

[07/18/2022] Mathematical Analyses of Loss Landscapes: From Linear to Nonlinear

Exact Solutions of a Deep Linear Network (Ziyin et al., NeurIPS 2022)

Presenter (offline): Yujung Lee (Samsung Research)
Slide and Paper

Nonlinear random matrix theory for deep learning (Pennington & Worah, NIPS 2017; Journal of Statistical Mechanics 2019)

Presenter (online): Hojung Jung (OSI Lab @KAIST AI)
Slide and Paper

[07/11/2022] Entropic Perspective of SGD

Energy–entropy competition and the effectiveness of stochastic gradient descent in machine learning (Zhang et al., Molecular Physics 2018)

Presenter (offline): Minchan Jeong (OSI Lab @KAIST AI)
Slide and Paper

Entropic variants of SGD

Presenter (offline): Junghyun Lee (OSI Lab & OptiML @KAIST AI)
Slide
Papers presented:
- Entropy-SGD: biasing gradient descent into wide valleys (Chaudhari et al., ICLR 2017; Journal of Statistical Mechanics 2018)
- Entropic gradient descent algorithms and wide flat minima (Pittorino et al., ICLR 2021; Journal of Statistical Mechanics 2021)

[07/04/2022] Heavy-tailed Theory of SGD: From Continuous to Discrete

On the Heavy-Tailed Theory of Stochastic Gradient Descent for Deep Neural Networks (Şimşekli et al., ICML 2019; arXiv 2019)

Presenter (offline): Junghyun Lee (OSI Lab & OptiML @KAIST AI)
Slide and Paper

The Heavy-Tail Phenomenon in SGD (Gürbüzbalaban et al., ICML 2021)

Presenter (offline): Hanseul Cho (OptiML @KAIST AI)
Slide (FIX) and Paper

Page updated

Google Sites

Report abuse