Research

Bayesian approach for automatic joint parameter estimation in 3D image reconstruction from multi-focus microscope

We present a Bayesian approach for 3D image reconstruction of an extended object imaged with multi-focus microscopy (MFM). MFM simultaneously captures multiple sub-images of different focal planes to provide 3D information of the sample. The naive method to reconstruct the object is to stack the sub-images along the z-axis, but the result suffers from poor resolution in the z-axis. The maximum a posteriori framework provides a way to reconstruct a 3D image according to its observation model and prior knowledge. It jointly estimates the 3D image and the model parameters. Experimental results with synthetic and real experimental data show that it enables the high-quality 3D reconstruction of an extended object from MFM.

Fig. 1. 3D image reconstruction from experimental MFM data. (a) Tile stacking, (b) RLS-TV, (c) MAP1, and (d) MAP2.

Fig. 2. Estimation of noise level

Sinogram image completion for limited angle tomography with deep neural networks

We present a novel approach based on machine learning for solving the limited angle tomography problem. The limited angle views in tomography cause severe artifacts in the tomographic reconstruction. We use deep convolutional generative adversarial networks (DCGAN) to fill in the missing information in the sinogram domain. First, we train the DCGAN to generate sinograms for a target data. It is used to generate a full angle sinogram that corresponds to the partial information from the limited angle tomography. By using the continuity loss and two-ends method, the image completion in the sinogram domain is done effectively, resulting in the high-quality reconstruction with less artifacts. The sinogram completion method can be applied to different problems such as region-of-interest tomography and ring removal problems.

3D image reconstruction from multi-focus microscopy: multiple-frame processing

Multi-focus microscope (MFM) provides a way to obtain 3D information by simultaneously capturing multiple focal planes. The naive method for MFM reconstruction is to stack the sub-images with alignment. However, the resolution in the z-axis in this method is limited by the number of acquired focal planes. In this work, we present a new reconstruction algorithm for MFM images to improve the resolution in the z-axis. For further improvement of the reconstruction quality, we also propose two multiple-frame MFM image reconstruction algorithms: batch and recursive approaches. In the batch approach, we take multiple MFM frames and jointly estimate the 3D image and the motion for each frame. In the recursive approach, we utilize the reconstructed image from the previous frame. Experimental results show that the proposed algorithms produce a sequence of 3D object reconstruction with high quality that enable reconstruction of dynamic extended objects.

3D image deconvolution and particle tracking with I⁵M

In spite of many advantages of optical microscopy, it suffers from the poor axial resolution. In order to overcome the limit, we have set up I⁵M, which stands for I³ (incoherent interference illumination) + I²M (image interference microscopy). This interferometric fluorescent microscope enables us to obtain super-resolution images of live biological samples and to achieve dynamic real-time tracking. We perform 3D image deconvolution by inverse filtering and constrained least squares filtering followed by projection onto non-negative orthant. The tracking utilizes the information stored in the interference pattern of both the illuminating incoherent light and the emitted light. By periodically shifting the interferometer phase and a phase retrieval algorithm we obtain information that allows localization with sub-2 nm axial resolution at 5 Hz.

Alignment and image reconstruction from uncalibrated tomography

As x-ray and electron tomography is pushed further into the nanoscale, the limitations of rotation stages become more apparent, leading to challenges in the alignment of the acquired projection images. Here we present an approach for rapid post-acquisition alignment of these projections to obtain high-quality three-dimensional images. Our approach is based on a joint estimation of alignment errors, and the object, using an iterative refinement procedure. With simulated data where we know the alignment error of each projection image, our approach shows a residual alignment error that is a factor of a thousand smaller, and it reaches the same error level in the reconstructed image in less than half the number of iterations. We then show its application to experimental data in x-ray and electron nanotomography. (paper)

Compressive reconstruction for 3D incoherent holographic microscopy

Incoherent holography has recently attracted significant research interest due to its flexibility for a wide variety of light sources. In this work, we use compressive sensing to reconstruct a three-dimensional volumetric object from its two-dimensional Fresnel incoherent correlation hologram (FINCH). We show how compressed sensing enables reconstruction without out-of-focus artifacts when compared to conventional back-propagation recovery. Finally, we analyze the reconstruction guarantees of the proposed approach both numerically and theoretically and compare that with coherent holography. (project, paper)

Video super-resolution using convolutional neural network

Convolutional neural networks (CNN) are a special type of Deep Neural Networks (DNN). They have so far been successfully applied to image super-resolution (SR) as well as other image restoration tasks. We propose a CNN that is trained on both the spatial and the temporal dimensions of videos to enhance their spatial resolution. Consecutive frames are motion compensated and used as input to a CNN that provides super-resolved video frames as output. We investigate different options of combining the video frames within one CNN architecture. While large image databases are available to train deep neural networks, it is more challenging to create a large video database of sufficient quality to train neural nets for video restoration. We show that by using images to pre-train our model, a relatively small video database is sufficient for the training of our model to achieve and even improve upon the current state-of-the-art. We compare our proposed approach to current video as well as image super-resolution algorithms. (project, paper)

Video super-resolution using dictionary learning

We have developed a video super-resolution (SR) algorithm using the dictionary technique. First of all, we use a multiple-dictionary scheme for the image SR. By using multiple dictionaries, we are able to tune dictionaries specifically to reconstruct a certain type of structure or feature, which enables us to create more accurate SR images. Second, we propose two approaches to exploit neighboring frames using optical flow algorithm - sequential and recursive methods. The proposed SR approaches first apply the estimated optical flow to obtain multiple-frame registration with high accuracy and then jointly reconstruct an HR frame from multiple LR frames. Finally, we suggest a novel dictionary training algorithm for video super-resolution, which improve the performance even more. This research was funded by Samsung Electronics. (paper)

Fig 1. Multiple-frame super-resolution scheme. The batch approach (left) and the recursive approach (right).

Light field rendering

Light field represents all the light information in free space. Once you have light field data, you can synthesize any view in the space. We used the dynamic re-parameterization technique introduced by Isaksen et al. (SIGRAPH 2000). which is suitable for light field data captured by camera array. It allows us to render views with variable depth of focal plane and aperture size as well as any translation and rotation of novel views. We have implemented the light field renderer in three different ways, (1) pixel-based ray tracing method, (2) memory coherent ray tracing method, and (3) texture mapping based method. Experimental results show the successful rendering with different parameters. This work was done during the internship at Dolby Laboratories.

Visual search system

We have developed visual search system in order to support Northwestern University building search. We proposed a novel client-server content-based-image-retrieval (CBIR) application that combines Laplacian-SIFT for feature descriptor, multiple kd-trees for indexing, and two levels of geometric verification. We developed back-end and front-end Application Programming Interfaces (APIs) for client-server CBIR application, and we proposed a distributed system architecture to support multiple client requests. The application consists of two user interfaces: a web interface and a mobile interface.I worked on geometric verification algorithm and development of the Android mobile application for this system. (paper)

Automatic red-eye removal using inpainting in digital photograph

When we take pictures with flash, red-eye effect often appears in photographs. We proposes a red-eye removal algorithm using inpainting and eye-metric information, which is largely composed of two parts: red-eye detection and red-eye correction. For red-eye detection, face regions are detected first. Next, red-eye regions are segmented in the face regions using multi-cues such as redness, shape, and color information. By region growing, we select regions, which are to be completed with iris texture by an exemplar-based inpainting method. Then, for red-eye correction, pupils are painted with the appropriate radii calculated from the iris size and size ratio. Experimental results with a large number of test photographs with red-eye effect show that the proposed algorithm is effective and the corrected eyes look more natural than those processed by the conventional algorithms. (paper)

Design and analysis of decoder of H.264 and MPEG-2 in multi-core system

As demands for high-definition television (HDTV) increase, the implementation of real-time decoding of high-definition (HD) video becomes an important issue. The data size for HD video is so large that real-time processing of the data is difficult to implement, especially with software. In order to implement a fast H.264/MPEG-2 decoder for HDTV, we compose five scenarios that use parallel processing techniques such as data decomposition, task decomposition, and pipelining. Assuming the multi digital signal processor environments, we analyze each scenario in three aspects: decoding speed, L1 memory size, and bandwidth. By comparing the scenarios, we decide the most suitable cases for different situations. We simulate the scenarios in the dual-core and dual-central processing unit environment by using OpenMP and analyze the simulation results. This research was funded by Samsung Advanced Institute of Technology. (paper)

Fig 3. Two schemes for the parallel processing (left, right)

Face recognition in different color spaces

We evaluate the PCA-based face recognition algorithms in various color spaces and analyze their performance in terms of the recognition rate. Experimental results with a large number of face images (CMU and FERET databases) show that color information is beneficial for face recognition and that the SV, YCbCr, and YCg‘Cr‘ color spaces are the most appropriate spaces for face recognition. The SV color space is shown to be effective for illumination variation, the YCbCr color for facial expression variation, and the YCg‘Cr‘ color space for aged faces. From experiments, we found that the recognition rate in all the color spaces with independent processing is higher than that with concatenated processing. (paper1, paper2)