Wavelets, neural networks and signal processing
Schedule
PART I - FOURIER ANALYSIS
30/01 Introduction to the course. Fourier analysis, I. The Fourier transform. [RS80, Fol92]
01/02 Fourier analysis, II. Fourier series and the Cosine basis. Finite signals: the Discrete and the Fast Fourier Transform. [Rud76, Fol92, HW96, Fra99, Kat04]
06/02 Fourier analysis, III. Shannon's sampling theorem and the Paley-Wiener space. [Hig85, Dau92, Fol92]
08/02 Fourier analysis, IV. The Uncertainty Principle. [Fol92, DS98, Kat04]
13/02 Lab 1: Fourier analysis of time series and of digital images. Nyquist frequency, aliasing, convolutions, modulus and phase.
PART II - FRAMES AND TIME-FREQUENCY ANALYSIS
15/03 Crash course on frame theory. Parseval frames and orthonormal bases. Bessel sequences, dual frames and the reconstruction formula. Finite frames. [Dau92, HW96, Cas00, Chr16]
20/02 Time-frequency analysis, I. The Short-Time Fourier Transform. Time-frequency localization. [Dau92, Gro01]
22/02 Time-frequency analysis, II. Gabor frames. Existence, the density theorem and the Balian-Low theorem. [Dau92, HW96, Gro01, Jan05]
27/02 Time-frequency analysis, III. The Discrete Gabor Transform of finite signals. [Jan94, FS98, Chr16]
01/03 Lab 2: Time-frequency analysis of digitally sampled sounds with Gabor frames. [see also LTFAT]
PART III - WAVELETS
06/03 Introduction to Wavelets, I. The continuous wavelet transform. Calderón's admissibility. Cancellations and details. [Dau92, Hol95, Mal09]
08/03 Introduction to Wavelets, II. Discrete wavelets are not systems of translates. The Haar and Shannon wavelets. [HW96, GLWW03, Mal09]
13/03 Multiresolution analysis, I. Approximation, details and the scaling function. [Dau92, HW96, Mal09]
15/03 Multiresolution analysis, II. The completeness theorem. The Lowpass filter of an MRA. [HW96]
First Course Assignment (20% grade): due date 21/03
22/03 Multiresolution analysis, III. Highpass filters and Mallat's MRA orthonormal wavelets. [Dau92, HW96, Mal09]
27/03 Multiresolution analysis, IV. Compactly supported wavelets. The Decomposition Algorithm. [Dau92, HW96, Mal09, Pin02]
29/03 Lab 3: Morlet and Mexican Hat continuous wavelets. Discrete Wavelet Transforms and Multiresolution Analysis. [Fra99, VFl08]
PART IV - MACHINE LEARNING AND NEURAL NETWORKS
12/04 Introduction to Machine Learning. Unsupervised, supervised and reinforcement learning.
17/04 Introduction to neural networks: basic architectures and gradient descent. [Hay09, Nes03]
19/04 More on gradient descent: backpropagation and online learning. [Hay09, Zin03, Haz22]
24/04 NN and approximation theory, I: Cybenko's universal approximation theorem. [Pin99]
26/04 NN and approximation theory, II: Deep vs shallow approximation. [AB09, Pog17, Yar17]
Second Course Assignment (20% grade): due date 03/05
03/05 Recent news in AI [NYT23, OL23]. The bias-variance problem, overparameterization, implicit regularization in SGD. [CS01, CS18, BHM19]
08/05 A crash course on Reinforcement Learning, I. Markov Decision Processes and Dynamic Programming. [Put14]
10/05 A crash course on Reinforcement Learning, II. The Policy Gradient Theorem and Proximal Policy Optimization. [SB18, OAI17]
25/05 Final Presentations
Morning 10:30 - 13:00
Marcos Llorca - The convolutional structure of translation invariant operators.
Elena Petrunina - The Duality Theorem for Gabor frames.
Louise Ethier - Reproducing Kernel Hilbert Spaces.
Gonzalo Liniers - Kernel Principal Component Analysis and Kernel Machine Learning.
Afternoon 14:30 - 17:00
Diego González - The Retinex algorithm and the Wilson-Cowan neural activity equations.
Victor Mialdea - Signal processing and auditory perception.
Samuel Nevado - Long Short-Time Memory recurrent Neural Networks.
Esther Jerez - Image classification with Convolutional Neural Networks in Tensorflow.
Bibliography
Part I
[Bre11] H. Brezis, Functional analysis, Sobolev spaces and partial differential equations. Springer, 2011.
[DS98] D. Donoho, P. Stark, Uncertainty principles and signal recovery. SIAM J. Appl. Math. 49, pp. 906-931, 1998. link
[Con90] J. Conway, A course in functional analysis. Springer, 2nd ed. 1990.
[Fol92] G. Folland, Fourier analysis and its applications. Brooks/Cole, 1992.
[Hig85] J. Higgins, Five short stories about the cardinal series. Bull. Amer. Math. Soc. (N.S.) 12, pp. 45-89, 1985. link
[Kat04] Y. Katznelson, An introduction to harmonic analysis. Cambridge University Press, 3rd ed. 2004.
[RS80] M. Reed, B. Simon, Methods of modern mathematical physics Vol. 1 - Functional analysis. Academic Press 1980.
[Rud76] W. Rudin, Principles of mathematical analysis. McGraw-Hill, 3rd ed. 1976.
Parts II and III
[Cas00] P. Casazza, The art of frame theory. Taiwanese J. Math. 4, pp.129-201, 2000. link
[Chr16] O. Christensen. An introduction to frames and Riesz bases. Birkhäuser, 2nd ed. 2016.
[Dau92] I. Daubechies. Ten lectures on wavelets. SIAM, 1992.
[FS98] H. Feichtinger, T. Strohmer, Eds. Gabor analysis and algorithms. Springer, 1998.
[Fra99] M. Frazier. An introduction to wavelets through linear algebra. Springer, 1999.
[GLWW03] P. Gressman, D. Labate, G. Weiss, E. Wilson, Affine, quasi-affine and co-affine wavelets. In "Beyond wavelets", G. Welland (Ed.), Elsevier 2003. link
[Gro01] K. Gröchenig. Foundations of time-frequency analysis. Springer, 2001.
[HW96] E. Hernández, G. Weiss. A first course on wavelets. CRC Press, 1996.
[Hol95] M. Holschneider. Wavelets. An analysis tool. Clarendon Press 1995.
[Jan94] A. Janssen, Duality and biorthogonality for discrete-time Weyl-Heisenberg frames. Unclassified report, Philips Electronics, 002/94. link
[Jan05] A. Janssen, Classroom proof of the density theorem for Gabor systems. ESI Preprint, 2005. link
[Mal09] S. Mallat. A wavelet tour of signal processing. Academic Press, 3rd ed. 2009.
[Pin02] M. A. Pinsky. Introduction to Fourier analysis and wavelets. AMS 2002.
[SN97] G. Strang, T. Nguyen. Wavelets and filter banks. Wellesley-Cambridge Press, 1997.
[VFl08] P. Van Fleet. Discrete wavelet transformations. Wiley, 2008.
Part IV
[AB09] M. Anthony and P.L. Barlett. Neuronal network learning: Theoretical foundations. Cambridge University Press, 2009.
[BHM19] M. Belkin, D. Hsu, S. Mandal. Reconciling modern machine-learning practice and the classical bias–variance trade-off. PNAS 116:15849-15854 (2019).
[CS18] P. Chaudari, S. Soatto. Stochastic Gradient Descent Performs Variational Inference, Converges to Limit Cycles for Deep Networks. Information Theory and Applications Workshop ITA (2018).
[CS01] F. Cucker, S. Smale. On the Mathematical Foundations of Learning. Bulletin of the AMS 39:1-49 (2001).
[Hay09] S. Haykin. Neural Networks and Learning Machines. Pearson, 3rd ed. 2009.
[Haz22] E. Hazan. Introduction to online convex optimization. MIT Press, 2nd ed. 2022.
[LBRN06] H. Lee, A. Battle, R. Raina & A. Y. Ng. Efficient sparse coding algorithms. NIPS (2006).
[vL07] U. von Luxburg. A tutorial on spectral clustering. Stat Comput 17: 395–416 (2007).
[Nes03] Y. Nesterov. Introductory lectures on convex optimization. Springer, 2003.
[NYT23] ‘The godfather of AI Leaves Google and warns of danger ahead. C. Mets for The New Yourk Times, May 1 2023. link link
[OAI17] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov. Proximal Policy Optimization Algorithms. OpenAI preprint, arXiv.
[OL23] Pause Giant AI Experiments: An Open Letter. March 22, 2023. link
[Pin99] A. Pinkus. Approximation theory of the MLP model in neural networks, Acta Numerica 8:143-195 (1999).
[Pog17] T. Poggio et al. Why and When Can Deep-but Not Shallow-networks Avoid the Curse of Dimensionality: A Review. International Journal of Automation and Computing 14:503-519 (2017)
[Put14] M. L. Puterman. Markov decision processes: discrete stochastic dynamic programming, John Wiley & Sons, 2014.
[SB18] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[Yar17] D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks, 94:103-114 (2017).
[Zin03] M. Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. ICML (2003).