Current Research Projects
Motion is a key cue for biological visual systems, and we often detect objects when they move. Therefore, detecting changes in apparent motion in videos is a natural approach for detecting objects from video. However, motions of objects are often ambiguous to compute from limited data (few frames) due to distractions from other moving nuisances or other changes in the scene, and doing so would lead to false detections. Motion cues can be made less ambiguous by observing more data (frames), but this leads to increased computational cost. Since we are interested in real-time systems, this increased cost may not be tolerable. We are constructing algorithms that address the tradeoff between computational cost (delay after the object of interest moves) and false detections due to clutter or weak motion cues. Quickest Detection, building on optimal stopping theory, provides a framework for minimizing the delay in detecting changes in stochastic processes while constraining false detections to an acceptable tolerance, and we are building on this framework to address the problem of quickest moving object detection.
[Sample papers: arXiv 2016 Technical Report]
We are developing algorithms to label pixels of distinct objects in videos. This problem of object segmentation has importance in a variety of applications (e.g., designing a robot that picks up and manipulates an object, or in motion picture processing). Many of these applications require real-time operation, and thus we are aiming for real-time operation by constructing causal algorithms. The challenges are that objects exhibit significant geometric and appearance variability in video due to viewpoint, object deformation, illumination and occlusions. Even for rigid objects, deformations induced by change of viewpoint form an infinite dimensional group. Therefore, we are developing new computational and analytic tools for modeling objects as dense regions (to model shape) and dense radiance functions on regions (to model radiance). We are developing optimization tools on such models for segmentation, and we are developing prediction and filtering tools to estimate shape and radiance of objects across time. The basic mathematical structure to enable these computational tools is a metric, which quantifies shape changes, and we have been exploring Sobolev region metrics, which we have developed using methods of Riemannian geometry. Such metrics are natural for modeling objects, which tend to deform smoothly, because of their smoothness properties.
One of the greatest challenges in designing computer vision algorithms is the variability of appearances of the same object or material due to nuisances in image formation, for example, the geometric distortion across an image of plaid-patterned cloth due to perspective or wrinkles in the cloth, or the photometric variation due to lighting. To overcome this, invariant descriptors have been developed, which compute statistics (aggregated from local patches) that do not change with respect to nuisances. However, such statistics must be computed within objects or materials otherwise irrelevant data from other objects corrupt the descriptors. Since the purpose of descriptors is to perform higher-level tasks, e.g., segmentation, segmentation is not available. Therefore, we are generalizing such descriptors to be computed while performing higher level tasks (segmentation, matching). We call these Shape-Tailored Descriptors as they are tailored to the shape of the object. We are building on optimization theory and specifically optimization with partial differential equation (PDE) constraints to compute such descriptors efficiently, and use them in applications.
Image registration (finding pixel-wise correspondence in images) is a problem of wide importance in computer vision. The difficulty of the problem comes in part due to the aperture ambiguity, that is,
[Sample Papers: TMI 2014]
Many computer vision algorithms require labeling pixels in images with a discrete set of labels (e.g., segmentation problems) and such problems are often formulated as optimization problems.
Image correspondence (finding corresponding regions/parts/pixels between images) among images with large changes of viewpoint, occlusions, illumination change, and object deformations is a fundamental task in computer vision with a wide number of applications (segmentation of objects in video, 3D reconstruction, etc). Under such scenarios, registration methods based on optimization often fail since such methods make assumptions on the amount of displacement. While descriptor / detector methods are able to cope with large displacements, they often fail under large viewpoint change in non-planar scenes and only match sparse parts of the image. Such methods often fail since they are unable to capture the right tradeoff between invariance / covariance of the descriptor / detector and selectivity for matching. We are developing methods that are better able to address this tradeoff in the matching task by computing such invariances of regions during the matching process. We are building hierarchies of regions and hierarchies of invariant descriptors to perform matching in a coarse-to-fine manner for efficiently. To do this, we are building on methodology from group theory.
Three dimensional image datasets arise from many scientific applications, and in several applications it is of interest to extract surfaces (e.g., boundaries between various materials or structures). These structures may not enclose a volume and thus traditional methods (e.g., level set methods, graph cuts, etc) are not applicable. We are designing robust methods for extracting such surfaces with boundaries that do not enclose volumes. We are building on the Fast Marching Algorithm for propagating fronts and techniques from computational topology for extracting critical structures, under noisy conditions, from the propagating front. These critical structures form the surface. This problem has application to seismic image analysis, which is of great interest to geology and geophysics with applications to the oil industry, and we are applying our methods to this area.