We are trying to solve the problem of portrait photo retouching (PPR) which aims at enhancing the visual quality of a collection of flat-looking portrait photos. There is a large demand for portrait photo retouching, as many people like to take portrait photos, but simply taking a picture with a nice camera does not produce as high quality of an image as can be achieved by an expert photographer editing such a picture. Of course, this would require hiring a professional to perform such work, which can be expensive and time-consuming. If there were an automated system to transform high-quality pictures into the desired final image, as if they were edited by a professional, then such images would become more accessible.

There do exist some machine learning models for image retouching, but for portrait photo retouching specifically, to our knowledge this is the only example, Liang et al. [4], which we intend to replicate. This is largely due to the fact that PPR has its special and practical requirements such as human-region priority (HRP) and group-level consistency (GLC). These are difficult to achieve using machine learning without a portrait-specific dataset, which Liang et al. have developed and shown for their model [4]. Importantly, this model [4] is also better than other state-of-the-art techniques when applied to this dataset. As this is relatively new territory and the dataset they used is now publicly available, we are interested in attempting to replicate their findings, as well as improve on their model where possible.

Figure 1: Two examples of groups of photos from the PPR10K dataset. The top half are raw; the bottom half retouched by an expert for better visual quality and group-level consistency.

For this task, we are using the PPR10K dataset developed by Liang et al. [3]. While there do exist many general-purpose datasets for retouching or enhancing photos, as well as models to automate this process, they do not fulfill the requirements mentioned previously that are specific to PPR, HRP and GLC. HRP requires that the photo retouching pays attention to the human regions of the photo, and GLC requires that the tone of photos be consistent within a group that shares the same subjects and setting. To achieve these requirements, Liang et al. created a PPR-specific dataset with thousands of 4k and 8k photos, including high-resolution segmentation masks for the human region, and all manually retouched by experts in the field for high-quality ground truth.

The dataset consists of 11,161 4k and 8k images divided into 1,681 groups, with each group containing 3-18 photos with the same subjects in the same setting across the group. The photos were taken with various high quality DSLR devices and are very diverse in terms of scene, subject, lighting, and camera settings. As the name ”portrait photo retouching” suggests, the main focus of the images in the dataset is people. This means the diversity comes through changing the people and the location, including subjects both old and young, locations both indoor and outdoor, and lighting conditions day and night, as well as at different times of the year. In order to achieve the high-quality ground truth, 3 different experts were hired to retouch each of the raw photos, so every photo in the dataset has 3 possible ground-truths that can be used in training the model to achieve the goals of HRP and GLC. Before they are used for either training or validation, the raw and retouched photos are scaled down from 4k/8k to 360p for faster training times. Figure 1 shows what a group of photos looks like and how the raw photos compared to those by experts, and figure 2 showcases the diversity of the dataset.

The dataset is also bolstered via data augmentation. This is where transformations and modifications are made to existing images in the dataset to add more diversity and increase the number of training and validation points for creating the model. Here, it is performed with 6 main visual attributes: temperature, tint, exposure, highlights, con- trasts, and saturation. There are ranges set for all of these values based on advice from the expert retouchers to ensure they are still realistic; each image is given random modifica- tions within these ranges on each attribute to create the additional images to augment the dataset. Modifying these attributes contributes to the dataset having more diverse and representative lighting and color distributions. After data augmentation, the dataset consists of 53,250 training pairs.

Figure 2: Examples to show the diversity of the dataset

We learn image-adaptive 3-dimensional lookup tables to achieve fast photo enhancement. 3D LUTs are widely used for manipulating color and tone of photos, but they are usually manually tuned and fixed in camera imaging pipeline or photo editing tools. We learn multiple 3D LUTs and importance of each 3D LUT using Convolution Neural Network(CNN) architecture. An intuitive idea is to learn a classifier to perform scene classification, and then use different 3D LUTs to enhance different images. The CNN weight predictor aims to understand the global context such as brightness, color and tones of the image to output content-dependent weights. Therefore, it only needs to work on the down-sampled version of the input image to largely save computational cost. We used the human region weighted mean squared loss on predicted color images to optimize a model. For the human-centered measures, we set w = 1 for backgrounds and w = 5 for human-regions for faster convergence.

Figure 3: Baseline 3D LUT Model

Figure 4: Deep Guided Filter

Figure 5: Guided Filter Example

Figure 6: Guided Filter Results