Deinterlacing & FrameUpconversion

Fig. 7, Halo/Patch/Judder/Broken Artifacts (covered in yellow, uncovered in blue) in MC-FRC [2].

Motion Estimation:

• • Spatial Domain Techniques (Block Matching);

    • • Full Search

      • Fast search

      • Hierarchical search

      • Predictive Search

      • 3-D Recursive Search (3DRS): nice for IC field.

        • • Philip NXP.

  • Pixel-based methods

    • • Half-pixel

      • Qua-pixel

  • Bidirectional Motion Estimation (Fwd/Bwd MVs for object boundaries)

  • Parametric Model-based Motion Estimation (camera rotation or zooming)

Fig. 8, Hierarchical Motion Estimation (HME) [3]

Fig. 9, Sub-pixel Motion Estimation [8]

Fig. 10, Full Search for Motion Estimation [8]

Fig. 11, Fast Search for Motion Estimation [8]

3DRS: Bias/Penalty for Each Candidate

• • Global Motion Vector (GMV) candidates;

  • Spatial motion vector candidates (up, left);

  • Temporal motion vector candidates (constant velocity);

  • Zero motion vector (ZMV) candidates;

  • Hierarchical motion vector (HMV) candidates

Fig. 12, Illustration [4] and Dialgram [6] of 3D Recursive Search in Motion Estimation

Fig. 13, Illustration Difference of Full Search and 3D Recursive Search in Motion Estimation (Pseudo Color) [10]

Reliability of Motion Estimation (The Aperture Problem)

• • SAD/SSD;

  • Edge-weighted (SED);

  • SAD/SSD surface (error distribution)

Fig. 14, SAD Surface to evaluate the reliability of Motion Estimation [11]

Post-processing of ME

• • Smoothing (Vector Median Filter);

  • FG/BG segmentation for MV Field Refinement;

    • • Global Motion Vector;

      • Bidirectional ME: Forward/Backward Motion Vector;

      • Matching errors: SAD or SAD for halved MV.

  • Occlusion (Cover/Uncovered) Detection for MV Field Refinement;

    • • Bidirectional ME: Forward/Backward Motion Vector;

      • FG/BG: covered by FG and uncovered BG based on MV direction;

      • Motion Vector Field Discontinuities;

      • Matching errors: SAD or SAD for halved MV.

Fig. 15, Smoothing Motion Field [2]

Fig. 16, Occlusion Detection (OCC: Occlusion Constraints, LRC: Left-Right-Failures-Checking, and the 4-Way Method) [7].

Motion Compensation:

• • Unidirectional MC:

    • • Overlapping

      • Holes

      • Blocking artifacts

      • Cover/Uncover (Recommended)

  • Bidirectional MC:

    • • OBMC (overlapped block MC) for blocking removal

      • No info. of occlusion

      • Bi-directional MVs (Compete or Merge).

  • Error Concealment in MC:

    • • Motion Compensated Average (MCA);

      • Dynamic Median;

      • Static Median;

      • Cascaded Median.

Fig. 17, Uni-/Bi-directional MC [5]

Fig. 18, Static/Dynamic MC [6]

Fig. 19, Cascaded Median MC [6]

Fig. 20, MCA and cascaded median MC (halo removal) [10]

Fig. 21, Blocking artifact and illustration of deblocking with OBMC (Overlapped Block MC)

Other Issues for Discussion

• • Scene change detection;

  • Fade-in/Fade-out detection;

  • Film/Movie detection (3:2 or 2:2 pull down);

  • Logo/Caption detection;

  • Saliency map to replace edge map;

  • Cartoon identification;

  • Color to Grayscale conversion;

  • .....

References

1. R. Prasad, De Interlacing Techniques, Slideshare, 2011.

2. N. Balram, Motion-Compensated Frame-Rate-Conversion for Perceptual Blur Reduction, Seminar ar Stanford, 2010.

3. A. Heinrich et al., Optimization of Hierarchical 3DRS Motion Estimators for Picture Rate Conversion, IEEE J. OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 5, NO. 2, APRIL 2011.

4. O. Erdler, Frame Rate Conversion in the HD era, European Technology Center, 2007.

5. D Wang et al., Motion-Compensated Frame Rate Up-Conversion—Part II: New Algorithms for Frame Interpolation, IEEE TRANSACTIONS ON BROADCASTING, 56(2), JUNE 2010.

6. J. van de Waerdt et al., TEMPORAL VIDEO UP-CONVERSION ON A NEXT GENERATION MEDIA-PROCESSOR, SIP, 2005.

7. C. Bartelsa and G. de Haanab, Occlusion Classifiers for Picture Rate Conversion, IS&T/SPIE Electronic Imaging; Visual Communications and Image Processing, January 2009.

8. M. Phadtare, Motion estimation techniques in video processing, Electronic Engineering Times India, August 2007.

9. Yen-Lin Lee, Truong Nguyen, HIGH FRAME RATE MOTION COMPENSATED FRAME INTERPOLATION IN HIGH-DEFINITION VIDEO PROCESSING, IEEE ICIP'10, 2010.

10. C. N. Cordes, G. de Haan, Key Requirements for High Quality Picture-Rate Conversion, Invited paper, SID Symposium Digest of Technical Papers, June 2009.

11. M. Chappalli, Y-T Kim, Z Zhou, Method and System for Determining the reliability of estimated motion vectors, US patents application, June 2006.

12. Y-C Chen, Y Gong, I-M Pao, Apparatus and method for frame rate conversion, US Patent application, June 2011.