Deinterlacing & FrameUpconversion

Motion Compensated Video Deinterlacing and Framerate Conversion

Under construction!

Display Technolgies of TV:

    • CRT - Pleasing color balance performance and wide viewing angles;

    • LCD – Less expensive than CRTs and come in both flat panel and rear projection varieties; bright, use less electricity than CRTs;

    • Plasma – Thin and made up of cells that correspond to pixels and contain three gas-fill sub-cells, for each color;

    • DLP – Light is reflected off an array of microscopic hinged mirrors. Each tiny mirror corresponds to a visible pixel;

    • OLED - A LED the emissive electroluminescent layer is a film of organic compound emitting light in response to an electric current.

State-of-Art LCD TV:

    • High frame rate (120Hz-240Hz);

    • Backlight dimming;

    • 3-D TV (without/with glasses).

    • UDTV (4Kx2K or 8K-4K?).

Video Formats:

    • Interlaced: Analog;

    • Progressive: Digital;

Deinterlacing Methods:

    • Intra field de-interlacing;

    • Inter field de-interlacing;

    • Spatio-temporal Interpolation.

    • Motion adaptive de-interlacing;

    • Motion compensated de-interlacing.

Fig. 1, Deinterlacing [1]

Motion Compensated Deinterlacing Methods:

    • Interpolate in the direction with the highest correlation, i.e. along the motion trajectory;

    • Combined with scene change detection, otherwise finding motion between two completely different scenes;

    • Combination of edge directed spatial interpolation and motion compensated temporal interpolation in practice;

    • Motion estimation is performed on the fields of same parity.

        • Motion vectors are halved and the blocks moved simultaneously to the sampling grid of the current interpolated field.

Fig. 2, Diagram and illustration of Motion Compensated Deinterlacing [1]

Fig. 3, Block Shifting to match the sampling grid [1]

Deinterlacing Artifacts:

    • Mouth-teeth;

    • Ghosting;

    • Blur.

Fig. 4, Mouth teeth artifact in de-interlacing [1]

Rendering Problem in LCD Display:

    • Motion judder due to slow response time of liquid crystal;

    • Perceptual blurring due to “sampling-and-hold” display;

    • Temporal integration of Human Visual System (HVS).

Frame Rate Up-Conversion:

    • Pull-down by drop/repeat: 3:2 for 24 to 60 Hz, 2:2 for 30 to 60Hz;

    • Interpolation.

        • Smoothing

        • Motion Compensation

Fig. 5, Diagram of Pull-down (Repeat) and Motion Compensated (MC) Frame Rate Conversion (FRC)

Fig. 6, Illustration [4] and Example (4x) [9] of Motion Compensated Frame rate conversion

Note: (a)(b) original frames, (c)(g) zoomed original, (d)-(f) zoomed interpolated.

Framerate Conversion Artifacts:

    • Halo (cover/uncover);

    • Patch (repeated pattern);

    • Judder (motion range);

    • Shadow (large motion);

    • Broken (small obj, boundary wrong labeling).

    • Blocking (block-based interpolation).

Fig. 7, Halo/Patch/Judder/Broken Artifacts (covered in yellow, uncovered in blue) in MC-FRC [2].

Motion Estimation:

    • Spatial Domain Techniques (Block Matching);

        • Full Search

        • Fast search

        • Hierarchical search

        • Predictive Search

        • 3-D Recursive Search (3DRS): nice for IC field.

            • Philip NXP.

    • Pixel-based methods

        • Half-pixel

        • Qua-pixel

    • Bidirectional Motion Estimation (Fwd/Bwd MVs for object boundaries)

    • Parametric Model-based Motion Estimation (camera rotation or zooming)

Fig. 8, Hierarchical Motion Estimation (HME) [3]

Fig. 9, Sub-pixel Motion Estimation [8]

Fig. 10, Full Search for Motion Estimation [8]

Fig. 11, Fast Search for Motion Estimation [8]

3DRS: Bias/Penalty for Each Candidate

    • Global Motion Vector (GMV) candidates;

    • Spatial motion vector candidates (up, left);

    • Temporal motion vector candidates (constant velocity);

    • Zero motion vector (ZMV) candidates;

    • Hierarchical motion vector (HMV) candidates


Fig. 12, Illustration [4] and Dialgram [6] of 3D Recursive Search in Motion Estimation


Fig. 13, Illustration Difference of Full Search and 3D Recursive Search in Motion Estimation (Pseudo Color) [10]

Reliability of Motion Estimation (The Aperture Problem)

    • SAD/SSD;

    • Edge-weighted (SED);

    • SAD/SSD surface (error distribution)

Fig. 14, SAD Surface to evaluate the reliability of Motion Estimation [11]

Post-processing of ME

    • Smoothing (Vector Median Filter);

    • FG/BG segmentation for MV Field Refinement;

        • Global Motion Vector;

        • Bidirectional ME: Forward/Backward Motion Vector;

        • Matching errors: SAD or SAD for halved MV.

    • Occlusion (Cover/Uncovered) Detection for MV Field Refinement;

        • Bidirectional ME: Forward/Backward Motion Vector;

        • FG/BG: covered by FG and uncovered BG based on MV direction;

        • Motion Vector Field Discontinuities;

        • Matching errors: SAD or SAD for halved MV.


Fig. 15, Smoothing Motion Field [2]

Fig. 16, Occlusion Detection (OCC: Occlusion Constraints, LRC: Left-Right-Failures-Checking, and the 4-Way Method) [7].

Motion Compensation:

    • Unidirectional MC:

        • Overlapping

        • Holes

        • Blocking artifacts

        • Cover/Uncover (Recommended)

    • Bidirectional MC:

        • OBMC (overlapped block MC) for blocking removal

        • No info. of occlusion

        • Bi-directional MVs (Compete or Merge).

    • Error Concealment in MC:

        • Motion Compensated Average (MCA);

        • Dynamic Median;

        • Static Median;

        • Cascaded Median.

Fig. 17, Uni-/Bi-directional MC [5]

Fig. 18, Static/Dynamic MC [6]

Fig. 19, Cascaded Median MC [6]


Fig. 20, MCA and cascaded median MC (halo removal) [10]


Fig. 21, Blocking artifact and illustration of deblocking with OBMC (Overlapped Block MC)

Other Issues for Discussion

    • Scene change detection;

    • Fade-in/Fade-out detection;

    • Film/Movie detection (3:2 or 2:2 pull down);

    • Logo/Caption detection;

    • Saliency map to replace edge map;

    • Cartoon identification;

    • Color to Grayscale conversion;

    • .....

References

1. R. Prasad, De Interlacing Techniques, Slideshare, 2011.

2. N. Balram, Motion-Compensated Frame-Rate-Conversion for Perceptual Blur Reduction, Seminar ar Stanford, 2010.

3. A. Heinrich et al., Optimization of Hierarchical 3DRS Motion Estimators for Picture Rate Conversion, IEEE J. OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 2, APRIL 2011.

4. O. Erdler, Frame Rate Conversion in the HD era, European Technology Center, 2007.

5. D Wang et al., Motion-Compensated Frame Rate Up-Conversion—Part II: New Algorithms for Frame Interpolation, IEEE TRANSACTIONS ON BROADCASTING, 56(2), JUNE 2010.

6. J. van de Waerdt et al., TEMPORAL VIDEO UP-CONVERSION ON A NEXT GENERATION MEDIA-PROCESSOR, SIP, 2005.

7. C. Bartelsa and G. de Haanab, Occlusion Classifiers for Picture Rate Conversion, IS&T/SPIE Electronic Imaging; Visual Communications and Image Processing, January 2009.

8. M. Phadtare, Motion estimation techniques in video processing, Electronic Engineering Times India, August 2007.

9. Yen-Lin Lee, Truong Nguyen, HIGH FRAME RATE MOTION COMPENSATED FRAME INTERPOLATION IN HIGH-DEFINITION VIDEO PROCESSING, IEEE ICIP'10, 2010.

10. C. N. Cordes, G. de Haan, Key Requirements for High Quality Picture-Rate Conversion, Invited paper, SID Symposium Digest of Technical Papers, June 2009.

11. M. Chappalli, Y-T Kim, Z Zhou, Method and System for Determining the reliability of estimated motion vectors, US patents application, June 2006.

12. Y-C Chen, Y Gong, I-M Pao, Apparatus and method for frame rate conversion, US Patent application, June 2011.