DAMO Academy, Alibaba Group, Hangzhou, China. (yunfuliu@gmail.com)
Da-Wei Jaw
National Taipei University of Technology, Taipei, Taiwan. (jdw.davidjaw@gmail.com)
National Taipei University of Technology, Taipei, Taiwan. (schuang@ntut.edu.tw)
University of Washington, Seattle, WA, USA. (hwang@uw.edu)
Our DesnowNet automatically localizes the translucent and opaque snow particles from realistic winter photographs (top) and removes them to achieve better visual clarity on corresponding resultants (bottom). Image credits (left to right): © Flickr users yooperann, Michael Semensohn, and Robert S.
Abstract
Existing learning-based atmospheric particles removal approaches such as the ones for rainy and hazy images are designed with strong assumptions regarding spatial frequency, trajectory, and translucency. However, removing snow particles is more complicated because not only it possesses additional attributes of particle size and shape, but all of these attributes could vary inside a single image. In the meantime, hand-crafted features are still the mainstream for snow removal so that they are hard to achieve great generalization.
We design a multistage network codenamed DesnowNet to in turn deal with translucent and opaque snows. We also disentangle snow into attributes of translucency and chromatic aberration for accurate estimations. Moreover, residual complements of the snow-free images are individually predicted to recovery the details covered by opaque snows. Additionally, the multi-scale design is utilized throughout entire network to model the diversity of snow.
As demonstrated in experimental results, our approach outperforms state-of-the-art learning-based atmospheric phenomenon removal methods and one semantic segmentation baseline on the proposed Snow100K dataset in both of qualitative and quantitative comparisons. Our network would benefit the applications involving computer vision and graphics.
Snow100K dataset
This dataset consists of 1) 100k synthesized snowy images, 2) corresponding snow-free ground truth images and 3) snow masks, and 4) 1,329 realistic snowy images. The snow-free and snowy realistic images 2) and 4) are downloaded from Flickr, and we manually validate each sample to see whether it is snow-free or not. The largest size of the image boundary is 640 pixels.
This dataset consists of three subsets as per the variations inside single image. 1) Snow100K-S: Samples in this subset are only synthesized with small snow particles. 2) Snow100K-M: It contains the samples that are of both snow particles in small and medium sizes. 3) Snow100K-L: Snow particles of sizes small, medium, and large are all used for synthesizing samples. Each subset contains around 33k images.
Download links: 1) Training set (50,000 images, 7.8GB), 2) Test set (50,000 images, 7.8GB), and 3) Realistic snowy images (1,329 images, 67MB).
Comparison
First column (left): Realistic snowy image. Second column: DerainNet [1]. Third column: DehazeNet [2]. Forth column: DeepLab [3]. Fifth column (right): Ours.
[1] X. Fu et al., "Clearing the Skies: A Deep Network Architecture for Single-Image Rain Removal," IEEE Transactions on Image Processing, vol. 26, no. 6, pp. 2944-2956, April 2017.
[2] B. Cai et al., "DehazeNet: An End-to-End System for Single Image Haze Removal," IEEE Transactions on Image Processing, vol. 25, no. 11, pp. 5187-5198, Aug. 2016.
[3] L.-C. Chen et al., "DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs," arXiv:1606.00915, May 2017.
More results
Left: Realistic snowy image. Right: Ours.
More results
Each GIF image repeatedly shows the 1) realistic snowy image, 2-3) two gradually estimated snow-free results of ours, and 4) the corresponding estimated snow mask.
Failure cases
Some of our failure cases happening on realistic images as exhibited below are of speckle-like artifacts. They are mainly caused by the confusing and blur backgrounds (the non-covered areas) so that hard to recover the covered regions with informative and distinct predictions.
However, it still reaches an essential purpose of our work that makes distinguishing the details behind falling snow a lot easier, e.g., the number of persons and their behaviors.
Left: Realistic snowy image. Right: Ours.
Citation
@article{liu2018desnownet, title={DesnowNet: Context-Aware Deep Network for Snow Removal}, author={Liu, Yun-Fu and Jaw, Da-Wei and Huang, Shih-Chia and Hwang, Jenq-Neng}, journal={IEEE Transactions on Image Processing}, volume={27}, number={6}, pages={3064--3073}, year={2018}, publisher={IEEE}}