Learning-based Denoising & Deblur

under construction...

Learning-based Image Denoising & Deblur

•Noise reduction: various assumptions of content internal structures;

•Learning-based

•Field of experts (MRF), CRF, NN (MLP, CNN);

•Sparse coding: K-SVD, LSSC,….

•Self-similarity

•Gaussian, Median;

•Bilateral filter, anisotropic diffusion;

•Non-local means.

•Sparsity prior

•Wavelet shrinkage;

•Use of both Redundancy and Sparsity

•BM3D (block matching 3-d filter)-benchmark;

•Can ‘Deep Learning’ compete with BM3D

•Sparse coding-based denosing by K-SVD [Elad and M. Aharon'06]

•Extract overlapping patches from a single image;

• clean or corrupted, even reference (multiple frames)?

•for example, 100k of size 8x8 block patches;

•Applied the K-SVD, training a dictionary;

•Size of 64x256 (n=64, dictionary size k).

•Lagrange multiplier namda = 30/sigma of noise;

•The coefficients from OMP;

•the maximal iteration is 180 and noise gain C=1.15;

• the number of nonzero elements L=6 (sigma=5).

•Denoising by normalized weighted averaging.

The Sparseland Model?

Image Denoising by K-SVD

Denoising by K-SVD SC

•Locally Learned Dictionaries (K-LLD) [Chatterjee and P. Milanfar'09]

•Identify dictionary which best capture underlying geometric structures;

•Similar structures will have similar dictionary, similar weights;

•Cluster image based on geometric similarity (K-Means on the SKR weights);

•Learn dictionary and order of regression for each cluster;

•Performance is between K-SVD and BM3D.

K-LLD based image denoising

•Image denoising is a learning problem to training ConvNets [Jain and Seung'08];

•Parameter estimation to minimize the reconstruction error.

•Online learning (rather than batch learning): stochastic gradient

•Gradient update from 6x6 patches sampled from 6 different training images

•Run like greedy layer-wise training for each layer.

CNN-based image denoising

•Denoising by MLP NN learning [Burger, Schuler, Harmeling'12]

•Denoising as learning: map noisy patches to noise-free ones;

•Patch size 17x17;

•Training with different noise types and levels:

•Sigma=25; noise as Gaussian, stripe, salt-and-pepper, coding artifact;

•Feed-forward NN: MLP;

•input layer 289-d, four hidden layers (2047-d), output layer 289-d.

•input layer 169-d, four hidden layers (511-d), output layer 169-d.

•40 million training images from LabelMe and Berkeley segmentation!

•1000 testing images: Mcgill, Pascal VOC 2007;

•GPU: slower than BM3D, much faster than KSVD.

•Deep learning can help: unsupervised learning from unlabelled data.

MLP of feed forward network

•Motion blur (camera or object): degradation by convolution of a latent image with a blur kernel during exposure;

•Averaging of unaligned images along the motion trajectory;

•Deblurr is an inverse problem: estimate point spread function, i.e. PSF;

•Multiple images or single image:

•Hardware-based: hybrid imaging (camera motion), coded aperture (blur kernel);

•Presence of noise is a big problem (how to detect blur and noise?);

•Ringing artifacts or amplification of noise in deblurring.

•Non-blind deconvolution for single image deblur (kernel known):

•Blind deconvolution (BD) for single image deblur (no kernel clue)

•Ill-posed, only solved by assumptions or priors;

•Spatially invariant: uniform BD

•Spatially variant: non-uniform BD

•Optical aberration: lens imperfection

•Image Restoration by CNN [Eigen, D Krishnan, R Fergu'13]

•Collect a dataset of clean/corrupted image pairs then used to train a specialized form CNN (AlexNet).

•Given a noisy image x, predict a clean image y close to the clean image y*

•The input kernel p1 = 16, the output kernel pL = 8.

•Two hidden layers (i.e. L = 3), each with 512 units, the middle layer kernel p2 = 1.

•W1 as 512 kernels of size 16x16x3, W2 as 512 kernels of size 1x1x512, and W3 as size 8x8x512.

•This learns how to map corrupted image patches to clean ones, implicitly capturing the characteristic appearance of noise in natural images.

•Train the weights Wl and biases bl by minimizing the mean squared error.

•Minimize with SGD.

•First patchifying the input, applying a FCN to each patch, and averaging the resulting output patches.

AlexNet in ImageNet

•Image Deconvolution with Deep CNN [Xu et al.'14]

•Establish the connection between traditional optimization-based schemes and a CNN architecture;

•A separable structure is used as a reliable support for robust deconvolution against artifacts;

•The deconvolution task approximated by a CNN by nature, based on the kernel separability theorem;

•Kernel separability is achieved via SVD;

•An inverse kernel with length 100 is enough for plausible deconv. results;

•Image deconvolution CNN (DCNN);

•2 hidden: 38 1-d kernels 121×1 + 381x121 convolution kernels to each in h1-> output 1×1×38 kernel;

•Random-weight initialization or got from the separable kernel inversion;

•Concatenation of deconvolution CNN module with denoising CNN;

•It is called “Outlier-rejection Deconvolution CNN (ODCNN)”;

•2 million sharp patches together with their blurred versions in training.

ODCNN for image deconvolution

Denoising

Deblur