Tutorial at IEEE Int. Conf. on Acoustics, Speech, and Signal Processing (ICASSP) 2016

Jiangbo Lu¹, Dongbo Min², Minh N. Do³

¹Advanced Digital Sciences Center (ADSC), Singapore

²Chungnam National University (CNU), Daejeon, Korea

³University of Illinois Urbana-Champaign (UIUC), US

Description

Resulting from light measurements of a real scene, a natural image is not a collection of random numbers simply filling up a 2D matrix. Instead, there is a rather rich amount of redundancy, self-similarity or coherence that exists locally and globally. In the same vein, visual correspondence fields or feature matches, which associate pixels (or feature points) in one image with their corresponding pixels (or feature points) in another image, possess a similar natural coherence property. However, this is just one side of the coin; on the other side, there always exist edges, boundaries or discontinuities due to e.g. the colorful yet non-flat world, independent motions of objects in the scene, and parallax induced by camera movements. As such, discontinuities in different visual “signals” are clearly roadblocks that algorithms have to effectively deal with when exploiting the coherence or smoothness property. Motivated by this, the talk is set centrally on “coherence” and “discontinuities” for images and motions. We will introduce recent work along this line, ranging from modeling and efficient solutions to wide applications.

We will start with a gentle introduction of various state-of-the-art nonlinear edge-aware image smoothing filters (both locally modeled and globally optimized versions). Thanks to their strong power in adaptively dealing with various visual signals as well as significant computational and implementation advantages, the edge-aware image smoothing techniques have found a great variety of applications in image/video processing, computer vision and computer graphics. In these applications, the smoothing techniques have been employed to allow for data adaptivity (or supports) in either local or global forms. We explain their theoretical connections, new insights and generalization. Especially, we focus on fast smoothing approaches e.g. using the bilateral grid, a color line model, multipoint aggregation, domain transform, fast global image smoothing, and so on. Then, their wide and concrete applications in image processing, computer vision and computer graphics will be discussed.

Next, we will move on to the second part of our tutorial -- estimating “motion” fields between two (or more) images, also known as “visual correspondence”, which is a fundamental problem in numerous computer vision applications. In particular, we will cover various labeling optimization techniques, including local, semi-local, and global labeling techniques, which have been developed to design visual correspondence algorithms. Although these approaches have been developed from different perspectives, they share the central goal of efficiently computing a large number of accurate matches between a given pair of images under various challenging conditions. These difficult conditions include, for instance, matching image pairs in presence of significant geometric and photometric transformation (e.g. scale, rotation, wide baseline, large and non-rigid motions, illumination changes, image quality), across different scene contents, or containing a significant number of outliers. We also introduce labeling techniques that effectively deal with the huge discrete label space and/or the high-order Markov Random Field (CRF) model by making use of efficient filtering algorithms and a smart randomized search idea. Finally, we will introduce exciting applications relying on coherent “motion” fields, including scene understanding, robot navigation, computational photography, and 3-D scene reconstruction. We will present the key ideas of these latest applications, while highlighting the essential roles that the “motion” fields are playing.

Contact

Dr. Jiangbo Lu: jiangbo (dot) lu (at) adsc (dot) com (dot) sg

Dr. Dongbo Min: dbmin (at) cnu (dot) ac (dot) kr, dbmin99 (at) gmail (dot) com

Course Material

Part 0: Introduction (ppt, pdf)

Part 1: Edge-aware image filtering and applications (ppt, pdf)

Part 2: Discontinuity-preserving visual correspondences and applications (ppt, pdf)

Code

(More codes will be updated!)