Objective Visual Quality

Objective Non-Reference Visual Quality Assessment of Image/Videos

Under construction!

Applications of Visual Quality Assessment (VQA):

• • Image/Video Coding;

  • Image/Video Watermarking;

  • Image/Video Denoising;

  • Image/Video Artifact removal;

  • Image/Video Error Protection;

  • Image/Video Synthesis/Rendering.

Categories of VQA:

• • Subjective rating:

    • • MOS (Mean opinion score);

      • DMOS (Difference Mean opinion score);

Subjective tests:

• • DSCQS (Double stimulus continuous quality scale);

  • DSIS (Double stimulus continuous Impairment scale);

  • SSCQE (Single stimulus continuous Quality Evaluation);

  • PC(Pair Comparison), ACR(Absolute Category Rating).

Objective methods:

• • Full Reference;

  • Reduced Reference;

  • Non Reference.

Fig. 1, Subjective methods by MOS

Objective VQA Metrics

• • Signal fidelity measures;

    • • PSNR.

  • Perceptual visual quality metrics (PVQMs);

    • • Vision-based modeling;

        • • Subband Decomposition;

          • Luma Adaptation;

          • Visual masking.

Signal-driven approach.

• • Statistical features;

  • distortion;

  • artifact;

  • structural similarity.

VQEG: ITU.

Fig. 2, Vision-based Modeling for VQM

Non Reference VQM

• • Contrast/Sharpness: a bandpass-filtered/lowpass-filtered image;

    • • Luminance, color, motion, texture.

  • Blockiness: dicontinuties at block boundaries(horiz. and vert.), harmonic analysis;

    • • Profiles along vert. and horiz. and local peaks detection;

      • Luminance adaptation and texture masking as well.

Noise: noise engergy after denoising;

Blurring: contrast decrease on edges;

Color Bleeding: color contrast higher at the edges between areas;

Ringing/Mosquito: ratio for deviation of the noise spectrum after edge preserving filter;

Jerkiness: frame dropping/freeze measure.

Human Visual System (HVS)

• • Visual masking: masking (visibility is reduced by the background) effect in HVS;

    • • Physiological & Psychological mechanism.

      • Luminance adaptation;

      • Pattern masking;

        • • Contrast masking;

          • Noise masking;

          • Edge masking;

          • Texture masking (semi-local, entropy or activity masking).

Contrast sensitivity function (CSF);

Visual Attention model (saliency map);

Mono/Multi-channel model:

• • Frequency-based signal decomposition to create a spatial-frequency hierarchy: steerable pyramid, wavelet, cortex filter, QMF, HOP, ...

Fig. 3, Campbell and Robson chart

Fig. 4, Spatial CSF

Fig. 5, Velocity CSF

Fig. 6, Spatial-Temporal CSF

Fig. 7, Luminance Adaptation

Cortex Filter [Watson'87]

• • Gabor function: Gaussian by a sinusoid, to provide a good albeit preliminary description of V1 simple cell receptive fields;

  • Cortex transform mimics HVS's critical band (Gabor shape);

    • • Radial freq. selectivity is symmetric on a log freq. axis with bandwidths nearly constant at one octave.

      • Orientation selectivity is symmetric about a center peak angle with tuning bandwidths varying w.r.t. the radial freq.;

      • Scalability means identical processing for each resolution/layer.

Cortex filter models the space frequency localization aspects of HVS by splitting the original image spectrum into many spatial images;

• • Mesa filter: a 2d low pass filter with a blurry disc at frequency domain and a impulse response as a Gaussian by a Bessel function;

  • Difference of Mesa (dom) filter for radial frequency selectivity: subtraction of a smaller disk from a large one;

  • Bisection filter: bisect the frequency space, created by a 2d step function convolved with scaled Gaussian;

  • Fan filter for orientation selectivity: a set of fan shaped regions in frequency space, created by repeated bisection filters;

Cortex filter is multiplying the dom filter with the fan filter to get a small band of 2d spatial frequency, resembling the 2d Gabor function;

DCT-cortex filter mapping: How dct coeff. contribute to cortex band;

Threshold elevation (masking) model for texture energy (spatial details).

Fig. 8, Typical cortex transform decomposition in frequency domain

Fig. 9, Cortex Filter from dom filter and fan filter

Fig. 10, Cortex Filter response (right) similar to Gabor function (left) of a cortical receptive field

Fig. 11, DCT Cortex Filter Mapping

Fig. 12, A Simple Threshold Elevation Model

Visual Difference Predictor (VDP) [Daly'93, Bradley'99, Ninassi'08]

• • Extended from the Cortex filter;

    • • Hanming function in transition for mesa filter;

      • Polar degree in dom filter;

      • Gaussian baseband;

      • Unuse high frequency residual.

Conditions of a patial-frequency hierarchy:

• • invertible, unit frequency response, orientation sensitivity to at least 4 directions;

  • minimal overlap btw adjacent frequency channels;

  • shift invariant, orthogonal;

  • limited spatial extent, linear phase.

Masking: phase coherent/incoherent masking, mutual masking;

• • Distortions: blurring and contouring (banding) in addition.

  • Noise: minimal threshold on WT coeff.

  • Activities: smooth or highly textured.

Wavelet Visible Difference Predictor: Approximate correspondence b.t.w. HVS and WT, more comput. efficient than cortex transform;

WQA: Wavelet-based Quality Assessment proposed by Thomson;

• • Separable wavelets: weaker match to HVS;

  • Non-separable wavelets: sampling pattern alternates by level (rectangular or quincunx), but no increase of regularity;

  • HVS wavelets: rotated of -45 degree and scaled after separable WD (input sampling rectangularly, but output sampling quincunx).

CSF for spatial frequency and orientation over the DWT subband;

Masking:

• • Contrast masking;

  • Semi-local masking;

Error pooling.

How DWT-based VQA is applied for image coding?

• • Codestream quality layer in JPEG 2000 [Zeng'02].

Multiscale geometric analysis:

• Wavelet, curvelet, bandelet, contourlet.

Fig. 13, DWT-based subband detection (noise) threshold

Fig. 14, DWT-based subband threshold elevation model

Fig. 15, Spatial frequency dependent DWT (separable) and its rotation to match HVS better.

Fig. 16, DWT-based CSF and rotated DWT-based CSF

Fig. 17, WQA Flowchart

Visual Masking

• • Visibility threshold: contrast value for which a visual cell responds;

  • Visual Masking: sensitivity of human eyes for natural pictures can be modulated (inc./dec. ) by stimulus (background).

  • Sub-band decomposition:

    • • intra-channel (same orientation) masking, intra-component (same frequency) masking;

      • inter-channel (different orientation) masking and inter-component (different frequency) masking.

JND (Just-Noticeable-Distortion) Model

• • Visibility threshold can be detected by majority (75%): minimally-noticeable-distortion (MND) profile;

  • Pixel-based:

    • • Edge detection & Block type classification.

        • • Edge density for plain, edge, texture;

      • Texture masking

        • • Spatial and disorder degree (gradient distribution).

Subband-based:

• • DWT domain (others: Laplacian pyramid...).

  • DWT coeff: LL, HL, LH, HH;

Frequency-based:

• • DCT domain

    • • DCT coeff: DC, L(low freq), E(edge), H(high freq.);

      • Edge energy: L+E;

      • Texture energy: E+H;

Intra-band masking: coef direct comparing;

Inter-band masking: coef. combination and block classification (see the figure) followed by low-medium-high masking for smooth-edge-texture block resp.;

Luminance adaptation (Weber's law).

• • Mean luminance in the block;

Contrast masking

• • structural and textural masking.

Pixel-domain decomposition by edge preserving filters (separation of structure and texture).

Foveation (peripheral vision): distance to calculate weight.

JND for video: take into account temporal contrast sensitivity function (TCSF), eye movement and object motion (local).

Fig. 18, JND in Psycho Curve

Fig. 19, DCT-based Block Classification

Fig. 20, Block Classification and JND Masking Results

Note: more results are shown in the following webpage.