Projects
Research Areas
Machine Learning, Computer Vision, Graphics, Medical Image Analysis, Biometrics, Computer Animation, and Augmented Reality.
Research Projects
Learning-based Detection and Tracking in Medical Imaging: A Probabilistic Approach
Accurate, robust and fast tracking of deformable anatomical objects, such as the heart, is a crucial task in medical image analysis. One of the main challenges is to maintain an anatomically consistent representation of target appearance that is robust enough to cope with inherent changes due to target movement, imaging device movement, varying imaging conditions, and is consistent with the domain expert clinical knowledge. To address these challenges, we propose a probabilistic framework that relies on anatomically indexed component-based object models which integrate several sources of information to determine the temporal trajectory of the deformable target. Large annotated imaging databases are exploited to encode the domain knowledge in shape models and motion models and to learn discriminative image classifiers for the target appearance. The framework robustly fuses the prior information with traditional tracking approaches based on template matching and registration...
[book chapter in Deformation Models]
Automatic 3D Cardiac Flow Quantification
Valvular heart diseases are recognized as the central cause of morbidity and mortality. Accurate quantification of cardiac flow volumes in patients is of utmost importance, since it helps estimate the progression of the disease and determine the optimal time for surgical repair or replacement. Given the recent progress on real-time echocardiography, it becomes feasible to acquire high frame rate volumetric color Doppler flow images. We propose a fully automated method to quantify the cardiac flow using instantaneous 3D+t ultrasound data. The sampling planes at the mitral annulus and left ventricle outflow tract (LVOT) are detected and tracked automatically to compensate the heart motion and to overcome aliasing happened frequently to the color Doppler flow data. Preliminary results on clinical data confirmed these findings in a quantitative manner...
[Papers: ISBI'12, ISBI'11]
Detection and Quantification of Mitral Regurgitation on TTE with Application to Assist Mitral Clip Planning and Evaluation
Mitral regurgitation (MR), characterized by reverse blood flow during systole, is one of the most common valvular heart diseases. It typically requires treatment via surgical (mitral valve replacement or repair) or percutaneous approaches (e.g., MitraClip). To assist clinical diagnosis and assessment, we propose a learning-based framework to automatically detect and quantify mitral regurgitation from transthoracic echocardiography (TTE), which is usually the initial method to evaluate the cardiac and valve function...
[Paper: MICCAI-CLIP'12 ]
Prediction Based Collaborative Trackers (PCT)
We present a robust, fast, and accurate 3D tracking algorithm: prediction based collaborative trackers (PCT). A novel one-step forward prediction is introduced to generate the motion prior using motion manifold learning. Collaborative trackers are introduced to achieve both temporal consistency and failure recovery. Compared with tracking by detection and 3D optical flow, PCT provides the best results. The new tracking algorithm is completely automatic and computationally efficient. It requires less than 1.5 s to process a 3D volume which contains millions of voxels. In order to demonstrate the generality of PCT, the tracker is fully tested on three large clinical datasets for three 3D heart tracking problems with two different imaging modalities: endocardium tracking of the left ventricle (67 sequences, 1134 3D volumetric echocardiography data), dense tracking in the myocardial regions between the epicardium and endocardium of the left ventricle (503 sequences, roughly 9000 3D volumetric echocardiography data), and whole heart four chambers tracking (20 sequences, 200 cardiac 3D volumetric CT data)...
[Paper: TMI'11 ]
Patient-Specific Modeling and Quantification of the Aortic and Mitral Valves from 4D Cardiac CT and TEE
As decisions in cardiology increasingly rely on non-invasive methods, fast and precise image processing tools have become a crucial component of the analysis workflow. To the best of our knowledge, we propose the first automatic system for patient-specific modeling and quantification of the left heart valves, which operates on cardiac computed tomography (CT) and transesophageal echocardiogram (TEE) data. A novel physiological model of the aortic and mitral valves is introduced, which captures complex morphologic, dynamic and pathologic variations. This holistic representation is hierarchically defined on three abstraction levels: global location and rigid motion model, non-rigid landmark motion model and comprehensive aorticmitral model. Measurements efficiently computed from the aortic-mitral representation support an effective morphological and functional clinical evaluation. Extensive experiments on a heterogeneous data set, cumulated to 1516 TEE volumes from 65 4D TEE sequences and 690 cardiac CT volumes from 69 4D CT sequences, demonstrated a speed of 4.8 seconds per volume and average accuracy of 1.45mm with respect to expert defined ground-truth...
[Paper: TMI'10 ]
Non-rigid Face Tracking with Local Appearance Consistency Constraint
By utilizing both spatial and temporal appearance coherence at the patch level, the proposed approach can reduce ambiguity and increase accuracy. Recent research demonstrates that feature based approaches, such as constrained local models (CLMs), can achieve good performance in non-rigid object alignment/tracking using local region descriptors and a non-rigid shape prior. However, the matching performance of the learned generic patch experts is susceptible to local appearance ambiguity. Since there is no motion continuity constraint between neighboring frames of the same sequence, the resultant object alignment might not be consistent from frame to frame and the motion field is not temporally smooth. In this paper, we extend the CLM method into the spatio-temporal domain by enforcing the appearance consistency constraint of each local patch between neighboring frames. More importantly, we show that the global warp update can be optimized jointly in an efficient manner using convex quadratic fitting....
[paper:FG'08]
Enforcing Convexity for Improved Alignment with Constrained Local Models
Constrained local models (CLMs) have recently demonstrated good performance in non-rigid object alignment/ tracking in comparison to leading holistic approaches (e.g., AAMs). We propose a new approach for optimizing the global warp update in an efficient manner by enforcing convexity at each local patch response surface. Furthermore, we show that the classic Lucas-Kanade approach to gradient descent image alignment can be viewed as a special case of our proposed framework....
[paper:CVPR'08]
Non-Rigid Object Alignment with a Mismatch Template Based on Exhaustive Local Search
Non-rigid object alignment is especially challenging when only a single appearance template is available and target and template images fail to match. Two sources of discrepancy between target and template are changes in illumination and non-rigid motion. Because most existing methods rely on a holistic representation for the alignment process, they require multiple training images to capture appearance variance. We developed a patch-based method that requires only a single appearance template of the object. Specifically, we fit the patch-based face model to an unseen image using an exhaustive local search and constrain the local warp updates within a global warping space. Our approach is not limited to intensity values or gradients, and therefore offers a natural framework to integrate multiple local features, such as filter responses, to increase robustness to large initialization error, illumination changes and non-rigid deformations...
[paper:NRTL'07]
Face Re-Lighting from a Single Image under Harsh Lighting Conditions
We present a new method to change the illumination condition of a face image, with unknown face geometry and albedo information. This problem is particularly difficult when there is only one single image of the subject available and it was taken under a harsh lighting condition. Recent research demonstrates that the set of images of a convex Lambertian object obtained under a wide variety of lighting conditions can be approximated accurately by a low-dimensional linear subspace using spherical harmonic representation. However, the approximation error can be large under harsh lighting conditions thus making it difficult to recover albedo information. In order to address this problem, we propose a subregion based framework that uses a Markov Random Field to model the statistical distribution and spatial coherence of face texture, which makes our approach not only robust to harsh lighting conditions, but insensitive to partial occlusions as well...
3D Surface Matching and Recognition Using Conformal Geometry
3D surface matching is a fundamental issue in computer vision with many applications such as shape registration, 3D object recognition and classification. However, surface matching with noise, occlusion and clutter is a challenging problem. In this paper, we analyze a family of conformal geometric maps including harmonic maps, conformal maps and least squares conformal maps with regards to 3D surface matching. As a result, we propose a novel and computationally efficient surface matching framework by using least squares conformal maps...
[ demo:avi], [ Paper:CVPR'06, ICCV'07, PAMI ]
High Resolution Tracking of Non-Rigid 3D Motion of Densely Sampled Data Using Harmonic Maps
We present a novel fully automatic method for high resolution, non-rigid dense 3D point tracking. The novelty of this paper is the development of an algorithmic framework for 3D tracking that unifies tracking of intensity and geometric features, using harmonic maps with added feature correspondence constraints. While the previous uses of harmonic maps provided only global alignment, the proposed introduction of interior feature constraints guarantees that non-rigid deformations will be accurately tracked as well. The harmonic map between two topological disks is a diffeomorphism with minimal stretching energy and bounded angle distortion. The map is stable, insensitive to resolution changes and is robust to noise. Due to the strong implicit and explicit smoothness constraints imposed by the algorithm and the high-resolution data, the resulting registration/deformation field is smooth, continuous and gives dense one-to-one inter-frame correspondences. Our method is validated through a series of experiments demonstrating its accuracy and efficiency...
High Resolution Acquisition, Learning and Transfer of Dynamic 3-D Facial Expressions
Synthesis and re-targeting of facial expressions is central to facial animation and often involves significant manual work in order to achieve realistic expressions, due to the difficulty of capturing high quality dynamic expression data. We address fundamental issues regarding the use of high quality dense 3-D data samples undergoing motions at video speeds, e.g. human facial expressions. In order to utilize such data for motion analysis and re-targeting, correspondences must be established between data in different frames of the same faces as well as between different faces.
We present a data driven approach that consists of four parts: 1) High speed, high accuracy capture of moving faces without the use of markers, 2) Very precise tracking of facial motion using a multi-resolution deformable mesh, 3) A unified low dimensional mapping of dynamic facial motion that can separate expression style, and 4) Synthesis of novel expressions as a combination of expression styles.
The accuracy and resolution of our method allows us to capture and track subtle expression details. The low dimensional representation of motion data in a unified embedding for all the subjects in the database allows for learning the most discriminating characteristics of each individual's expressions as that person's "expression style". Thus new expressions can be synthesized, either as dynamic morphing between individuals, or as expression transfer from a source face to a target face, as demonstrated in a series of experiments...
[demo:avi1, avi2], [paper:EG'04]
A Hierarchical Framework For High Resolution Facial Expression Tracking
We present a novel hierarchical framework for high resolution, nonrigid facial expression tracking. The high quality dense point clouds of facial geometry moving at video speeds are acquired using a phase-shifting based structured light ranging technique. To use such data for temporal study of the subtle dynamics in expressions and for face recognition, an efficient nonrigid facial tracking algorithm is needed to establish intra-frame correspondences. We propose such an algorithmic framework that uses a multi-resolution 3D deformable face model, and a hierarchical tracking scheme...
[paper:ANM'04]
Estimation of Multiple Directional Light Sources for Synthesis of Augmented Reality Images
We present a new method for the detection and estimation of multiple directional illuminants, using a single image of any object with known geometry and Lambertian reflectance. We use the resulting highly accurate estimates to modify virtually the illumination and geometry of a real scene and produce correctly illuminated Augmented Reality images. Our method obviates the need to modify the imaged scene by inserting calibration objects of any particular geometry, relying instead on partial knowledge of the geometry of the scene. Thus, the recovered multiple illuminants can be used both for image-based rendering and for shape reconstruction. Our method combines information both from the shading of the object and from shadows cast on the scene by the object...
Estimation of Multiple Illuminants from a Single Image of Arbitrary Known Geometry
We present a method for the detection and estimation of multiple illuminants, using one image of any object with known geometry and Lambertian reflectance. Our method obviates the need to modify the imaged scene by inserting calibration objects of any particular geometry, relying instead on partial knowledge of the geometry of the scene...
[paper:ECCV'02]
Brookhaven National Laboratory (BNL) BME Project
Developing an integrated CCD-based image capture and motion tracking system, a core research of novel technologies for Positron Emission Temography (PET) and functional Magnetic Resonance Imaging (MRI) that will allow imaging of the awake animal brain in real-time and in natural physiological conditions. Knowing the brain?s position at any point in time will make is possible to compensate for its motion, either during the data acquisition or reconstruction phase of MRI and MicroPET scans.
Single View Metrology Project
In the light of the algorithm described in "Single View Metrology" by Criminisi, Reid, and Zisserman, ICCV 1999, we developed a system that can reconstruct 3D texture-mapped models from a single image and user interactions...
