Abstract

In recent years, the importance of vision sensors has been emphasized as deep learning-based vision applications have demonstrated superior performance. Despite the importance, visible-light cameras suffer from fundamental hardware limitations such as narrow dynamic range, small aperture size, and low sensor sensitivity. To compensate for this problem, the conventional camera performs a built-in Image Signal Processing (ISP) that improves image quality and provides an aesthetically pleasing image. However, the camera built-in ISP usually does not guarantee an optimal quality image for the various computer vision tasks.

In this paper, we propose a multi-objective camera ISP framework that utilizes Deep Reinforcement Learning (DRL) and camera ISP toolbox that consist of network-based tools and conventional tools. The proposed DRL-based camera ISP framework iteratively selects an image processing tool from the toolbox and applies it to the image to maximize a given vision task-specific reward function. For this purpose, we implement total 52 ISP tools that include exposure correction, color-and-tone correction, white balance, sharpening, denoising, and others. We also propose an efficient DRL network architecture that can extract the various aspects of an image and make a rigid mapping relationship between images and a large number of actions. Our proposed DRL-based ISP framework effectively improves the image quality according to the vision tasks such as RAW-to-RGB image restoration, 2D object detection, and monocular depth estimation.

Contribution

• We propose a novel framework for camera ISP that effectively performs a suitable action according to the image state and reward function based on DRL.

• We propose a camera ISP toolbox along with its training method. The toolbox consists of light-weight CNN tools and traditional tools that can represent each block of the camera ISP pipeline.

• We propose an efficient DRL network architecture that can extract the various aspects of an image and make rigid mapping relationships between images and a large number of actions.

• We validate the proposed method for RAW-to-RGB image restoration, 2D object detection, and monocular depth estimation. Throughout the experiments, the proposed method sequentially enhances the target task's performance by modifying images suitably.

Publication

"DRL-ISP: Multi-Objective Camera ISP with Deep Reinforcement Learning" [PDF]

Ukcheol Shin*, Kyunghyun Lee*, and In So Kweon
(*Equal contribution)

IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022