The aim of this course is to discuss some mathematical aspects of deep convolutional neural networks, with a special focus on scattering networks. We will motivate how several tools and results of harmonic analysis can be exploited in connection with classification problems and feature extraction. The course consists of two parallel paths with the purpose of considering complementary viewpoints on the issue.
The main track (about 20 hours) will focus on the problem of how to extract meaningful yet simple features from complex signals (e.g., images), motivated by learning problems such as classification. The main goal of the course is ultimately to discuss how some of the basic principles of harmonic analysis can be combined in an admittedly remarkable way to obtain these equivalent representations. In particular, we will show that setting as a goal the expected stability of features with respect to small natural deformations eventually leads to a convolution network architecture, which "scatters" input data through a cascading sequence of wavelet transforms and non-linear operators without deteriorating high frequency information. We will discuss the main properties of these scattering transforms and also some of their limitations. Motivated by the latter, we will finally consider more general scattering architectures and elaborate on why robustness to small deformations is yet another form of the ubiquitous uncertainty principle of harmonic analysis. This will allow us to show that stability to irregular or stochastic deformations is satisfied by a broad family of neural networks without heavy architectural assumptions.
In addition to the intriguing mathematical arguments leading to scattering transforms, it should be emphasized that scattering networks achieved state-of-the-art results in several concrete learning challenges. This is the reason why we felt the need to accompany the mathematical route with a complementary view on the implementation problems (about 10 hours). In this parallel exploration we will first have a closer look to wavelets. Following an itinerary that starts from basic examples in one dimension, ending with the challenges of high-dimensional extensions, we will present the main ideas of multiresolution analysis also by means of MATLAB hands-on sessions. The ultimate goal of this sidetrack is to see concrete scattering networks at work on classification problems and discuss their strengths and weaknesses, also in comparison with other machine learning techniques.