Current Research Projects

Minimum Delay Moving Object Detection

Motion is a key cue for biological visual systems, and we often detect objects when they move. Therefore, detecting changes in apparent motion in videos is a natural approach  for detecting objects from video. However, motions of objects are often
ambiguous to compute from limited data (few frames) due to distractions from other moving nuisances or other changes in the scene, and doing so would lead to false detections.  Motion cues can be made less ambiguous by observing more data (frames), but this leads to increased computational cost.  Since we are interested in real-time systems, this increased cost may not be tolerable. We are constructing algorithms that address the tradeoff between computational cost (delay after the object of interest moves) and false detections due to clutter or weak motion cues.  Quickest Detection, building on optimal stopping theory, provides a framework for minimizing the delay in detecting changes in stochastic processes while constraining false detections to an acceptable tolerance, and we are building on this framework to address the problem of quickest moving object detection.

Causal Video Object Segmentation and Prediction With Sobolev Region Metrics 

We are developing algorithms to label pixels of distinct objects in videos. This problem of object segmentation has importance in a variety of applications (e.g., designing a robot that picks up and manipulates an object, or in motion picture processing).  Many of these applications require real-time operation, and thus we are aiming for real-time operation by constructing causal algorithms.  The challenges are that objects exhibit significant geometric and appearance variability in video due to viewpoint, object deformation, illumination and occlusions.  Even for rigid objects, deformations induced by change of viewpoint form an infinite dimensional group.  Therefore, we are developing new computational and analytic tools for modeling objects as dense regions (to model shape) and dense radiance functions on regions (to model radiance).  We are developing optimization tools on such models for segmentation, and we are developing prediction and filtering tools to estimate shape and radiance of objects across time. The basic mathematical structure to enable these computational tools is a metric, which quantifies shape changes, and we have been exploring Sobolev region metrics, which we have developed using methods of Riemannian geometry.  Such metrics are natural for modeling objects, which tend to deform smoothly, because of their smoothness properties.

Shape-Tailored Scale Spaces and Invariant Descriptors

One of the greatest challenges in designing computer vision algorithms is the variability of appearances of the same object or material due to nuisances in image formation, for example, the geometric distortion across an image of plaid-patterned cloth due to perspective or wrinkles in the cloth, or the photometric variation due to lighting.  To overcome this, invariant descriptors have been developed, which compute statistics (aggregated from local patches) that do not change with respect to nuisances.  However, such statistics must be computed within objects or materials otherwise irrelevant data from other objects corrupt the descriptors.  Since the purpose of descriptors is to perform higher-level tasks, e.g., segmentation, segmentation is not available.  Therefore, we are generalizing such descriptors to be computed while performing higher level tasks (segmentation, matching).  We call these Shape-Tailored Descriptors as they are tailored to the shape of the object.  We are building on optimization theory and specifically optimization with partial differential equation (PDE) constraints to compute such descriptors efficiently, and use them in applications.

Physically Motivated Motion Estimation and Medical Image Registration / Segmentation

Image registration (finding pixel-wise correspondence in images) is a problem of wide importance in computer vision.  The difficulty of the problem comes in part due to the aperture ambiguity, that is,
without additional constraints on the registration, several registrations are valid.  While generic regularization strategies, i.e., spatial smoothness of the registration, have been widely used to address the aperture ambiguity, current registration schemes are still inadequate in many applications.  In some applications, such as medical applications, which involve motions of organs, tissues, fluids, etc., constraints on the registration arise from physical laws governing the motion.  We are developing registration methods that incorporate such constraints on deforming objects and their interactions with other structures.  We are applying these methods to the problem of object segmentation in medical images (cardiac MRI and CT), where accurate registrations are needed to propagate segmentations across time and also for segmentation of objects by motion.

[Sample Papers: TMI 2014]

Fast Label: Efficient and Simplified Optimization for Multi-Label Problems

Many computer vision algorithms require labeling pixels in images with a discrete set of labels (e.g., segmentation problems) and such problems are often formulated as optimization problems.  
These problems involve both fidelity to image statistics (data term) and regularization (as such labels are often coherent across nearby pixels).  We have been exploring new regularization strategies that generalize traditional regularization terms by incorporating the scale of the interest of regions generated by the labels.  In addition, these scale-varying generalizations, leads to simpler optimization problems than traditional formulations (e.g., convex relaxations and graph cuts) with significantly simpler implementations, equally good or better segmentation, and at lower computational cost.  We have applied this idea to a variety of segmentation problems, and we are currently formulating the theory based on geometric, continuous optimization theory.

[Sample papers: CVPR 2013CVPR 2014]

Invariant and Selective Descriptors for Matching

Image correspondence (finding corresponding regions/parts/pixels between images) among images with large changes of viewpoint, occlusions, illumination change, and object deformations is a fundamental task in computer vision with a wide number of applications (segmentation of objects in video, 3D reconstruction, etc).  Under such scenarios, registration methods based on optimization often fail since such methods make assumptions on the amount of displacement.  While descriptor / detector methods are able to cope with large displacements, they often fail under large viewpoint change in non-planar scenes and only match sparse parts of the image.  Such methods often fail since they are unable to capture the right tradeoff between invariance / covariance of the descriptor / detector and selectivity for matching.  We are developing methods that are better able to address this tradeoff in the matching task by computing such invariances of regions during the matching process.  We are building hierarchies of regions and hierarchies of invariant descriptors to perform matching in a coarse-to-fine manner for efficiently.  To do this, we are building on methodology from group theory.
[Sample Papers: CVPR 2015][Project Page]

SurfCut: Free-Boundary Surface Extraction From Topological Structures

Three dimensional image datasets arise from many scientific applications, and in several applications it is of interest to extract surfaces (e.g., boundaries between various materials or structures).  These structures may not enclose a volume and thus traditional methods (e.g., level set methods, graph cuts, etc) are not applicable.  We are designing robust methods for extracting such surfaces with boundaries that do not enclose volumes.  We are building on the Fast Marching Algorithm for propagating fronts and techniques from computational topology for extracting critical structures, under noisy conditions, from the propagating front. These critical structures form the surface.  This problem has application to seismic image analysis, which is of great interest to geology and geophysics with applications to the oil industry, and we are applying our methods to this area.