Research

Current Research Interest: Deep Learning, Deep Reinforcement Learning, Machine Learning, Computer Vision solving diagnostics and interventional imaging problems.

Selected previous projects:

Deep Reinforcement Learning for Robust Multi-modal Medical Image alignment (Appear at ACCV 2018). Recurrent Network and convolution neural network encode state space. Our method is robust in the multi-modal medical image alignment task even for data missing. We proposed a Monte Carlo Rollout method further improved accuracy.

6-DOF device tracking in 2D+time fluoroscopy video. This technology enables Siemens' True Fusion Product:

Core analytics algorithm contributor using deep learning in Siemens' easy detect and our team won 1st Prize in the Innovation Award in Siemens Mobility

PhD work: Medical computer vision and graphics, lung image analysis, robust model-based segmentation methods, efficient interactive segmentation methods, and hybrid virtual reality/desktop user interface.

The following demonstrates previous work from year 2008 to 2013.

- Lung Segmentation

A Novel Robust Active Shape

Model (RASM) based segmentation of lungs with high density pathology in combination with Optimal Surface Finding (OSF) approach.

Example of segmentation result

- Medical Virtual Reality

Interactive exploration of medical image data in a developed virtual reality (VR) environment.

VR setup - Hybrid user interface for interactive visualization and manipulation of medical image data and corresponding segmentations. (a) The user inspects the segmentation result by utilizing a 3D user interface. Circled devices are shown in enlarged sub-figures: (1) tracking cameras, (2) shutter glasses with head tracking targets, (3) stereo display with tracking targets, and (4) tracked input device. (b) The user operates the 2D user interface.

Advanced interactive visualization methods utilizing general-purpose computing on graphics processing units (GPGPU).

- Interactive Lung Segmentation Refinement using Developed Hybrid VR/Desktop User Interface

Example of interactive segmentation refinement - a real-time “dialog” between segmentation algorithm and user. Segmentation refinement of a lung with a small lung mass adjacent to the lung boundary. (a) The user inspects the lung segmentation and locates a segmentation error. (b) In a cross-section, the user selects a point on the correct boundary location with a virtual pen. Note that the incorrect portion of the contour is highlighted in light blue, which was automatically generated based on the selected point. (c) and (d) Refinement result after calculating the updated segmentation. (d) The corrected surface region is highlighted in green.

- Intravascular Ultrasound Image Segmentation

Graph-based approach for segmentation of luminal and external elastic lamina (EEL) surface of coronary vessels in intravascular ultrasound (IVUS) image sequences (volumes). The approach consists of a fully automated segmentation stage and a user-guided computer-aided refinement stage.

Illustration of interactive segmentation refinement of an automatically generated IVUS segmentation. (a) The user inspects the IVUS segmentation produced by the automated approach and discovers a local segmentation inaccuracy of the inner (arrow 1) and outer (arrow 2) surface. The outer boundary segmentation got “distracted” by a high density (calcified) region inside of the vessel wall and the associated shadow. (b) The user roughly indicates the correct location of the outer wall by drawing a polygon line (arrow 3, purple line) in proximity to the desired surface location. This single polygon line is used to locally modify the cost function for the outer boundary. (c) Refinement result after “recalculating” the segmentation result. Note that outer (arrow 4) and inner boundary (arrow 5) are simultaneously corrected due to the mutually interacting dual-surface segmentation approach. (d) Corresponding independent standard.

- Other projects-Brain connectivity visualization

US patent for this method was filed.

Collaborator: Mariappan Nadar and Sandra Sudarsky, Siemens Corporate Research.

Example of brain connectivity visualization. (a) A conventional method using straight lines between connected brain region. In this example, It is very difficult to see the connectivity since tons of connections. (b) Utilized 3D hierarchy edge bundling method significantly improves the visual clarity. (c) Brain connectivity visualization combined with volume rendering.

Demos:

Movie 1: Developed a Hybrid Desktop/Virtual Reality System

Movie 2: Advanced interactive visualization methods utilizing GPGPU. A user uses 3D input device exploring the lung and airway segmentation of CT image. Note that only monocular image was recorded for this movie.

Movie 3: 3D lung segmentation refinement using hybrid VR/Desktop user interface. Note that only monocular image was recorded for this movie.

Movie 4: 4D lung segmentation refinement using hybrid VR/Desktop user interface. User select inspiration scan and Note that only monocular image was recorded for this movie.

Movie 5: Interactive segmentation for 3D IVUS data.

Movie 6: Comparison between a standard ASM and proposed RASM. (Top video) A standard ASM fails to converge to the lung boundary in the data with missing scans. (Bottom video) Proposed RASM successfully converges to lung boundary.