Deep learning (DL) had gained significant popularity over the last few years, with DL-strategies being proposed for a multitude of problems arising in science and engineering. Despite the empirically demonstrated the superior performance of neural networks, it is important to acknowledge that the underlying theory to analyze such algorithms is still limited compared to traditional numerical algorithms. The goal of this RIT is to understand the idea behind neural networks and work through the key theoretical results currently available highlighting the nonlinear approximation capabilities of neural networks.
Time and Location: Every Thursday from 12:30pm-1:50pm in MTH 3206
Organizers: Deep Ray (deepray@umd.edu)
Haizhao Yang (hzyang@umd.edu)
Ricardo H. Nochetto (rhn@umd.edu)
Mailing list: Click here to join the mailing list. This will be used to make announcements about upcoming sessions and share details such as the Zoom link.
Schedule:
February 6, 2025: Introduction to Neural Network by Deep Ray
Code: MLP_Playground
References: Allan Pinkus, 1999
Patrick Kidger and Terry Lyons, 2020
Dmitry Yarotsky and Anton Zhevnerchuk, 2021
February 13, 2025: Error bounds for approximations with deep ReLU networks by Haizhao Yang
References: Dmitry Yartosky, 2017
February 20, 2025: Basic linear approximation by Di Wu
&
Nonlinear Approximations by Charles Richard Dziedzic
Notes: Slides by Di Wu
February 27, 2025: Error bounds for approximations with deep ReLU networks (cont) by Haizhao Yang
References:
Notes:
March 6, 2025: Neural network approximation based on integral representations by Haizhao Yang
&
Sharp error bound of neural network approximation based on integral representations by Cameron Austin
References: Baron, 1993
Weinan E et al., 2020
Siegel and Xu, 2024
Notes:
March 13, 2025: Nearly optimal deep network approximation for continuous functions by Haizhao Yang
References:
Notes:
March 27, 2025: Nearly Optimal Deep Network Approximation for Smooth Functions by Haizhao Yang
References: Yarotsky, 2018
Shen and Zhang, 2020
Notes:
April 3, 2025: Nonlinear Approximation in Sobolev Spaces by Mansur Shakipov
References: R. DeVore. "Nonlinear approximation", 1998.
A. Bonito, C. Canuto, R. H. Nochetto, A. Veeser. "Adaptive finite element methods", 2024.
R.H. Nochetto. "AMSC 714 Lecture Notes".
Notes: Slides by Mansur
April 10, 2025:
References:
Notes:
April 17, 2025:
References:
Notes:
April 24, 2025: Deep Network Approximation with Advanced Activation Function by Haizhao Yang
References:
Notes:
May 1, 2025: Optimal Stable Nonlinear Approximation by Yang Hong
References:
Notes:
May 8, 2025: TBA