Noise Reduction

Image/Video Denoising and Sharpening by Self Similarity and Sparsity

CCD (Charge Couple Device) Camera:

• digital image sensor (microlenses, CFA, ADC);

• CFA(color filter array): Bayer pattern, demosaicing;

• CCD sensor: correlated double sampling (CDS) to remove reset noise;

• store the electron charge and move them out of photo sensor converted to voltage signal.

Fig. 1, CCD Sensor and CCD Camera

CCD Noise Model:

• photon&dark shot noise

• amplifier noise

• readout (reset, kT/C thermal and 1/f flicker) noise;

• quantization noise.

CMOS (Complementary Metal Oxide Silicon) Camera and Noise Model:

• 10 times more fix pattern noise than CCD;

• 10 ~100 times faster then CCD;

• all the logic and control circuit be build on the same silicon wafer dice (small in size);

• power consumption is 1/2 to 1/4 less then CCD;

• convert electron generated by photo diode into voltage signal.

Fig. 2, CMOS Sensor and CMOS Camera

Noise Types:

• digital (flickering, salt-and-pepper, band/block/ringing, mosquito, bleeding);

• analog (film grain, Gaussian/Poisson);

Fig. 3, Gaussian noise, Artifacts

Goal of Noise Reduction:

• based on various assumptions of content internal structures

• smooth the content while preserving its details;

Classic Noise Estimation Methods:

• blocking-based [Bosco'03, Amer'02];

• smoothing-based [Immerkær'96, Rank'99];

• blocking+filtering-based [Shin'05, Foi'08];

State of Art Noise Estimation (Learning-based) Methods:

• wavelet-based [Stefano04];

• piecewise linear model [Russo06];

• segmentation-based picewise image model [Liu06];

• Kurtosis-based DCT bandpass filtering [Zoran & Weiss'09].

Fig. 4, Noise Properties[Liu06]

Basic/Classical Denoising Methods:

• mean/median/gaussian filtering

• bilateral filtering [Tomasi98]

• mean shift [Comaniciu&Meer02]

• anisotropic diffusion [Perona&Malik90, Black98]

• wavelet-based shrinkage in transform domain (thresholding/coring)

• adaptive sigma filtering

State-of-Art Denoising (Patch-based) Methods:

• NLM [Buades05]: Non Local-Means (self similarity for averaging);

• K-SVD [Elad&Aharon06]: Optimize the sparse coding in a trained dictionary;

• BM3D [Dabov06]: Block Matching 3-D (self similarity in image domain and sparsity in transform domain);

• K-LLD [Chatterjee&Milanfar09]: Clustering and Learning Local Dictionaries with kernel regression;

• LSSC [Mairal08]: combine grouping and dictionary learning to recursively find the optimized ones;

• CSR [Dong11]: combine global sparsity (dictionary learning) and local sparsity (structural clustering).

Fig. 5, Gray Image Denoising [Debov2007a]

Fig. 6, Color Image Denoising [Debov2007b]

Fig. 7, Video Denoising [Debov2007c]

Besides of Noise Level, What Else Used for Filters' Strength Adaptation?

• SSIM (structural similarity) learned from training data (original-filtered pair)[Zhou'04];

• Saliency map(importance of patches)[Itti'97] ;

• JND (just noticeble distortion from visual masking) [Zhang'08];

• Artifact index (blockiness, ringing, or color bleeding metric) [Lu'03].

Fast Search for BM3D (in place of exhaustive search)

• Three step search (TSS) [Koga'81];

• New three step search (NTSS) [Li'94];

• Four step search (4SS)[Po'96];

• Diamond search (DS)[Zhu'00];

Predictive Search for BM3D (motion smoothness and inertia)

• 3DRS[deHann'93];

• MVFAST[Ma & Hosur'99];

• PMVFAST[Tourapis, Au & Liou'01];

Fig. 8, Predictive Search in Video BM3D [Debov2007c]

Constraint and Expedients in Implementation of BM3D:

• Use sliding of step 3~5 pixels in horizontal/vertical directions;

• Restrict max size of a patch group with upper bound 16 or 32;

• Search for matching in a local neighborhood 15x15 or 33x33;

• Integral image in fast block matching (not usable for predictive search);

• Predictive search (especially efficient for video data);

• Use 2d + 1d transform for 3d transform;

• Simplify transform coefficients, like sparse integral transform;

• Precompute 2d spectra before grouping (may not suitable for video data).

Fig. 9, Video Denoising Results (Wall)

Fig. 10, Video Denoising Results(Lepord)

Improvement of BM3D:

• BM4D [Magioni12]: motion trajectory-based volume grouping;

• Shape-adaptive by PCA [Dabov'09];

• Locally adaptive regression kernel (LARK) [Takeda, Farsiu, Milanfar'07].

Fig. 11, BM4D: volume extraction by motion trajectory[Magioni12]

Fig. 12, BM3D-Shape Adaptive PCA [Dabov'09]

Fig. 13, Locally Adaptive Regression Kernel [Takeda, Farsiu, Milanfar'07]

Acceleration in Filter Design:

• Pipeline: splatting-blurring-slicing based on importance sampling;

• Bilateral grid [Chen, Siggraph’07]: a new data structure that enables fast edge-aware image processing;

  • x and y correspond to pixel position;

  • z corresponds to pixel intensity;

  • Euclidean distance accounts for edges;

  • Grid can be coarsely sampled;

  • To apply bilateral grid with three steps: grid creation, processing on bilateral grid and 2d map extraction by slicing.