Team: R. DeVore, K. Narayanan, G. Petrova, J. Siegel (lead) 

Synopsizing points

• Understand the set of problems in data science/scientific computing where neural networks beat other methods of function approximation

• Identify model classes for which approximation properties can be shown for various types of neural networks (ResNet, DenseNet, etc.)

• Characterize the functions in a given approximation class and assess how well these classes represent realistic functions

• Realize the approximation methods algorithmically and rigorously analyze approaches such as stochastic gradient and greedy methods

Related grants

• NSF-DMS: Neural Networks for Stationary and Evolutionary Variational Problems, 2023-26 (PI: Stephan Wojtowytsch)

• NSF-CCF CIF: Small: Interpretable Machine Learning based on Deep Neural Networks: a Source Coding Perspective, 2022-25 (local co-PI: Siegel)

• NSF SCALE-MoDL: New Perspectives on Deep Learning: Bridging Approximation, Statistical, and Algorithmic Theories, 2021-24 (local PI: Petrova, local coPI: DeVore)

• ONR MURI: Theoretical Foundations of Deep Learning, 2020-23 (local PI: DeVore, local coPIs: Foucart, Petrova)

Recent relevant papers

• R. DeVore, R. D. Nowak, R. Parhi, J. W. Siegel. Weighted variation spaces and approximation by shallow ReLU networks. (arXiv)

• J. W. Siegel. Optimal approximation of zonoids and uniform approximation by shallow neural networks. (arXiv)

• J. W. Siegel, S. Wojtowytsch. A qualitative difference between gradient flows of convex functions in finite-and infinite-dimensional Hilbert spaces. (arXiv)

• G. Petrova, P. Wojtaszczyk. Neural networks: deep, shallow, or in between? (arXiv)

• K. Gupta, J. W. Siegel, S. Wojtowytsch. Achieving acceleration despite very noisy gradients. (arXiv)

• J. W. Siegel. Sharp lower bounds on interpolation by deep ReLU neural networks at irregularly spaced data. (arXiv)

• J. W. Siegel. Optimal approximation rates for deep ReLU neural networks on Sobolev and Besov spaces. Journal of Machine Learning Research, 24/357, 1-52, 2023. (link)

• S. Wojtowytsch. Optimal bump functions for shallow ReLU networks: Weight decay, depth separation and the curse of dimensionality. Journal of Machine Learning Research, 25/27, 1−49, 2024. (link)

• S. Wojtowytsch, J. Park. Qualitative neural network approximation over R and C: Elementary proofs for analytic and polynomial activation. (arXiv)

• J. W. Siegel, J. Xu. Sharp bounds on the approximation rates, metric entropy, and n-widths of shallow neural networks. Foundations of Computational Mathematics, 24, 481-537, 2024.. (doi)

• I. Daubechies, R. DeVore, N. Dym, S. Faigenbaum-Golovin, S. Kovalsky, K.-C. Lin, J. Park, G. Petrova, B. Sober. Neural network approximation of refinable functions. IEEE Transactions on Information Theory, 69/1, 482-495, 2023. (doi)

• J. W. Siegel, J. Xu. High-order approximation rates for shallow neural networks with cosine and  ReLUk activation functions. Applied and Computational Harmonic Analysis, 58, 1-26, 2022. (doi)

• J. W. Siegel, J. Xu. Optimal convergence rates for the orthogonal greedy algorithm. IEEE Transactions on Information Theory, 68/5, 3354-3361, 2022. (doi)

• A. Cohen, R. DeVore, G. Petrova, P. Wojtaszczyk. Optimal stable nonlinear approximation. Foundations of Computational Mathematics, 22, 607–648, 2022. (doi)

• I. Daubechies, R. DeVore, S. Foucart, B. Hanin, G. Petrova. Nonlinear approximation and (deep) ReLU networks. Constructive Approximation, 55, 127-172, 2022. (doi)

• R. DeVore, B. Hanin, G. Petrova. Neural network approximation. Acta Numerica 30, 327-444, 2021. (doi)