Sparse color sensor for better low-light imaging

Sparse Color Sensor using Convolutional Neural Network-based Image Colorization

Despite the advances in image sensors, mainstream RGB sensors are still struggling from low quantum efficiency due to the low sensitivity of the Bayer color filter array (CFA). To address this issue, a sparse color sensor uses mostly panchromatic white pixels and a smaller percentage of sparse color pixels to provide better low-light photography performance than a conventional Bayer RGB sensor. However, due to the lack of a proper color reconstruction method, sparse color sensors have not been developed thus far. This study proposes a deep-learning-based method for sparse color reconstruction that can realize such a sparse color sensor. The proposed color reconstruction method consists of a novel two-stage deep model followed by an adversarial training technique to reduce visual artifacts in the reconstructed color image. In simulations and experiments, visual results and quantitative comparisons demonstrate that the proposed color reconstruction method can outperform existing methods. In addition, a prototype system was developed using a hybrid color-plus-mono camera system. Experiments using the prototype system reveal the feasibility of a very sparse color sensor in different lighting conditions.

Paper link: https://www.osapublishing.org/oe/abstract.cfm?uri=oe-27-17-23661

• Our Sparse Color Sensor is designed, based on the distribution of cones and rods in human retinas. 

• In particular, look at the peripheral vision in the following figure: 

• Deep learning model-based image colorization algorithm can realized the development of such "Sparse Color Sensors". 

• In our work, we verify the possibility of the sparse color sensor. 

Architecture of the Deep Convolutional Neural Network for Color Reconstruction from Sparse Color Pixels (1%):

The main reconstruction networks comprise of two building blocks (Generator and Discriminator). In our paper, it has been denoted as sGAN (structure GAN). The color reconstruction network takes three inputs: 1) RGB pixels, 2) panchromatic pixels, and 3) edge image. Please refer to the actual manuscript for more details.  The overview of the deep color reconstruction and the architectures of the sGAN have been shown in the following Figures. 

Fig. 1. The overall framework of the proposed color reconstruction method. LRN: luminance recovery network. CRN: color reconstruction network.

Fig. 2. The model architecture of color reconstruction network (CRN)

Fig. 3. The model architecture of the adversarial block (i.e., discriminator)

Experimental Results:

The color reconstruction model have been demonstrated with their respective prerequisites (i.e, two types of CFA and hybrid camera system). 

• • Results with a fixed sparse CFA pattern (i.e., extended CFZ [1]):

Examples of color filter patterns. (a) Bayer pattern. (b) Existing CFZ-14 pattern with a spare ratio (89% panchromatic pixels and 11% RGB pixels). (c) Extended version of CFZ-14 with a very sparse ratio (98.86% panchromatic pixels and 1.13% RGB pixels) used in our experiments. In the Figs., W denotes panchromatic white pixels.

Comparison of sparse color reconstruction with  Extended version of CFZ-14. Here, the CFA comprises with only 1% color pixels. (a) Ground truth (the inset is a zoomed-in image of the marked area). (b) Input Image (zoom). (c) Chakrabarti-14 . (d) Zhang. (e) Chakrabarti-16. (f) sNet (Proposed). (g) sGAN (Proposed).

• • Results with a Random CFA (color filter array):

Example illustration of the random CFA pattern with 1% color pixels used in our experiments. W denotes panchromatic white pixels. R, G, and B denote red, green, and blue color pixels, respectively

Prototype with a Mono-plus-RGB camera:

Hybrid camera system using two cameras and a beam splitter. (a) Side-view. (b) Top-view.

Extreme low-light comparison at 6 lux. In each pair, the left image shows the ground-truth image captured with an RGB sensor and the right image shows the result obtained by the hybrid camera system and the sparse color reconstruction method.