Course 4 - Prof. Okoudjou

Functional analysis and approximation theory for deep learning

The prerequisites for this course are a course in real analysis, basic Hilbert space theory, a linear algebra course. A course in functional analysis will be a plus but not necessary.

Lectures

Lecture 1: The first lecture will partially serve as a review and will include the following topics. Lecture Notes.

A review of the theory of Hilbert spaces, orthonormal bases, frames.
Introduction to linear and nonlinear approximation theory
Reproducing Kernel Hilbert Spaces (RKHS).

[Here is the video for lecture 1]

Lecture 2: In this second lecture we will cover the following topics. Lecture Notes (include updates from the first lecture)

Brief summary on approximation by Gabor and wavelet systems with a focus on classical smoothness spaces such as Sobolev spaces, Besov spaces, and modulation spaces.
A review of some linear algebra topics, e.g., best-fit subspaces and the SVD.

[Here is the video for lecture 2]

Lecture 3: The third lecture will bring the focus on topics related to manifold learning and will include the following topics. Lecture Notes (preliminary, but include updates from the first lecture)

A survey of some nonlinear dimension reduction methods, e.g., KPCA and Diffusion maps.
Learning function for Deep Learning in Euclidean spaces including CNNs and review of some graph Fourier analysis. (Part I)

[Here is the video for lecture 3]

Lecture 4: In this final lecture, we will cover some topics in spectral graph theory including harmonic analysis and wavelet theory on graphs. The (live) lecture notes for all the four lectures are here.

[Here is the video for lecture 4]

Teaching aids – software, textbooks, etc:

M. M. Bronstein, J Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst, Geometric deep learning: going beyond Euclidean data, IEEE SIG Proc. Mag. 34 (2017), no. 4, 18--42.
M. Belkin and P. Niyogi, Towards a theoretical foundation for Laplacian-based manifold methods, Journal of Computer and System Sciences, 74(8):1289–1308 (2008).
R. R. Coifman, S. Lafon, A. B. Lee, M. Maggioni, B. Nadler, F. Warner, and S. W. Zucker, Geometric diffusions as a tool for harmonic analysis and structure definition of data: Diffusion maps, PNAS, 102 (2005), no. 21, 7426-7431.
C. Chui, and H. Mhaskar, Deep nets for local manifold learning.
R. DeVore, Nonlinear approximation, Acta Numerica (1998), pp. 51-150
T. Hofmann, B. Scholkopf, and A. J. Smola, Kernel methods in machine learning, Ann. Statist., 36 (2008), no. 3, 1171--1220.

Praticals/tutorials

The tutorial and lab for this lecture will focus on linear and nonlinear approximations with applications to data (image) compression. Data for tutorial 1 or here. A recording of the session is made available here.

The tutorial and lab for this lecture will focus on applications of linear and nonlinear dimension reduction methods. Data for tutorial 2. Notes from today's tutorial are here. A recording of the tutorial session is here.

The tutorial and lab for this lecture will focus on graph Fourier (harmonic analysis) and diffusion maps. Data for tutorial 3 can be found here. A recording of the tutorial 3 here.

@Sidarth to update

Report abuse