Semantic Colorization with Internet Images


ACM Transactions on Graphics (Proceedings of SIGGRAPH ASIA 2011)


Alex Yong-Sang Chia,   Shaojie Zhuo,   Raj Gupta Kumar,   Yu-Wing Tai,   Siu-Yeung Cho,   Ping Tan,   Stephen Lin

Our system takes a grayscale photo with labeled and segmented foreground objects (a) as input and generates a set of colorization results (b) using reference image regions automatically searched from the internet and filtered to obtain the most suitable examples. Some of these examples are shown in (c). 


Colorization of a grayscale photograph often requires considerable effort from the user, either by placing numerous color scribbles over the image to initialize a color propagation algorithm, or by looking for a suitable reference image from which color information can be transferred. Even with this user supplied data, colorized images may appear unnatural as a result of limited user skill or inaccurate transfer of colors. To address these problems, we propose a colorization system that leverages the rich image content on the internet. As input, the user needs only to provide a semantic text label and segmentation cues for major foreground objects in the scene. With this information, images are downloaded from photo sharing websites and filtered to obtain suitable reference images that are reliable for color transfer to the given grayscale photo. Different image colorizations are generated from the various reference images, and a graphical user interface is provided to easily select the desired result. Our experiments and user study demonstrate the greater effectiveness of this system in comparison to previous techniques.


Download the supplementary video (.wmv 53M)

Download the fast forward video (.wmv 15M)


Download the paper

High quality PDF (47M)

Low quality PDF (2M)

Supplementary materials PDF (32M)

author = “Alex Yong-Sang Chia and Shaojie Zhuo and Raj Gupta Kumar and Yu-Wing Tai and Siu-Yeung Cho and Ping Tan and Stephen Lin”, 
title = “Semantic Colorization with Internet Images”, 
journal = “ACM Trans. Graph. (Proceedings of ACM SIGGRAPH ASIA) ”, 
year = “2011”, 
pages = “to appear”, 
volume = “30”, 
number = “6”}


Download the slides (.ppt 53M)


We thank the anonymous reviewers for their valuable input, and Yianling Cheong for her help in preparing the supplementary video. This work is supported by Institute for Infocomm Research and by Microsoft Research Asia. Shaojie Zhuo and Ping Tan are supported by Singapore Grant R-263-000-555-112. Yu-Wing Tai is supported by the NRF-2011-0013349 of Korea.

Last updated : 17 December 2011