The past decade in computer vision research has witnessed the re-emergence of “deep learning”, and in particular convolutional neural network (CNN) techniques, allowing to learn powerful image feature representations from large collections of examples. CNNs achieve a breakthrough in performance in a wide range of applications such as image classification, segmentation, detection and annotation. Nevertheless, when attempting to apply the CNN paradigm to 3D shapes and point clouds (feature-based description, similarity, correspondence, retrieval, etc.) one has to face fundamental differences between images and geometric objects. Shape analysis and 3D vision pose new challenges that are non-existent in image analysis, and deep learning methods have only recently started penetrating into the 3D computer vision community. CNNs have been applied to 3D data in recent works using standard (Euclidean) CNN architectures applied to volumetric or view-based shape representations.  Intrinsic versions of CNNs have also been proposed very recently with the generalization of the CNN paradigm to non-Euclidean manifolds, allowing them to deal with shape deformations. These “generalized” CNNs can be used to learn invariant shape features and correspondence, allowing to achieve state-of-the-art performance in several shape analysis tasks, while at the same time allowing for different shape representations, e.g. meshes, point clouds, or graphs.

The purpose of this tutorial is to overview the foundations and the current state of the art on learning techniques for 3D shape analysis and vision. Special focus will be put on deep learning techniques (CNN) applied to Euclidean and non-Euclidean manifolds for tasks of shape classification, object recognition, retrieval and correspondence. The tutorial will present in a new light the problems of shape analysis, emphasizing the analogies and differences with the classical 2D setting, and showing how to adapt popular learning schemes in order to deal with deformable objects.

Syllabus

1.   Theoretical foundations: Images vs shapes. Basics of differential geometry. Discrete shape representations: point clouds, meshes, volumes. Laplacian operator and its spectral properties, discretization. Fourier analysis on manifolds. Isotropic and anisotropic diffusion equation on manifolds. Spectral descriptors. Functional correspondence.

2.  Learning paradigms: Introduction to machine learning. Deep vs shallow learning. Random forests. Neural networks. Convolutional neural networks. Learning methods. Backpropagation.  

3.   Learning on geometric data: Multi-view CNNs. Volumetric CNNs. Intrinsic CNNs. Random forests on manifolds.  

4.  Applications: Deformable shape similarity and correspondence. Shape retrieval, sketch-based shape retrieval. 

5. Implementation and application examples. Live demos in MATLAB \ Python to exemplify the main concepts of the tutorial. 

Background

The tutorial will assume no particular background, beyond some basic working knowledge that is a common denominator for people in the field of computer vision and pattern recognition. All the necessary notions and mathematical foundations will be described. The tutorial is targeted to graduate students, practitioners, and researchers interested in shape analysis, matching, retrieval, and big data.