Research & Tech Transfer (selected)

My research has been transferred into numerous products with millions of users

Super Quality AI Portrait (2019)

Got a bad selfie? No worries! We present a deep network that fixes all the problems in your face images - noise, blur, out-of-focus, low light, low-res, compression, etc. Click here to see examples.

Lenovo S5 Pro: AI 3D Face Reshaping (2018)

We shipped this brand-new face-transforming feature to Lenovo mobile phones announced on March 20, 2018. We trained a network that analyzes a huge collection of "beautiful faces" and generates a 3D face model that transforms the input image. We made a lot of optimization so that the algorithm runs at 60 fps on a mobile phone!

Press conference

Face-aware Wide-angle Lens Correction (2017)

We developed a technique that corrects the distortion in wide-angle lens while preserving the faces. Note the faces near the image edges. The algorithm is super efficient and runs in real-time.


Burst Image Denoising (2018)

Super fast multi-frame denoising algorithm on mobile phones. The technique has been adopted by multiple phone companies to enhance pictures taken by the telephoto-lens.

Adobe After Effects: Camera Shake Deblur (2017)

The new video filter, Camera-Shake Deblur, reduces motion blur artifacts resulting from camera shake. This effect can save shots that might not otherwise be usable due to slight camera bumps or camera motion blur that remains after stabilization.

Video

Portrait Beautify Kit for Face++ API (2017)

A comprehensive feature set that includes skin clean-up, skin tone adjustment, digital makeup and face reshape, etc. Everything runs in real time on mobile!

Adobe Dimension CC: Auto IBL (2017)

This is a key feature in Dimension, a 3D compositing software that makes 3D easy for 2D graphics designers. It completes background images to 360-degree panoramas and approximates lighting and tone-mapping simultaneously.

Lightroom & Photoshop CC: Boundary Warp (2016)

Boundary Warp is designed to solve the issue of irregular boundaries when creating stitched panoramas. It analyzes the boundary of your panorama and then warps the image so the edges fit nicely inside a rectangular frame while all pixels are preserved.

GIF examples Tutorial Video Media Coverage

Premiere Elements 15: Fast Video Dehaze (2016)

Front page feature! This new Haze Removal enables you to easily remove haze to make the background of your scenes as crisp as what’s up front. We have done tremendous optimization to the algorithm. The filter processes 1080p videos in real-time!

Link

After Effects CC: Face Tracking (2015)

The new face tracker lets you track specific parts of the face such as pupils, mouth, and nose, allowing you to isolate and work on these facial features with greater detail. For example, change colors of the eyes or exaggerate mouth movements without frame-by-frame adjustments.

Tutorial Video

After Effects CC: Key Cleaner (2014)

The Key Cleaner effect recovers detail from a scene keyed by a typical keying effect, including recovering detail from a keyed scene with compression artifacts.

The Advanced Spill Suppressor effect removes color spill on a foreground subject from a colored background used for color keying.

Tutorial Press

AE & Premiere Pro: Detail-preserving Upscale (2013)

The Detail-preserving Upscale effect is capable of scaling up images by large amounts while preserving details in the image. The sharpness of sharp lines and curves is preserved. For example, you can scale up from SD frame sizes to HD frame sizes, or from HD frame sizes to digital cinema frame sizes.

Tutorial

After Effects CS6: Refine Edge Tool (2012)

Internal code name: "Hair Brush". It is a video matting tool that keeps the details when separating complicated foreground elements like frizzy hair or motion-blurred edges from complex backgrounds. This means you can create natural-looking composites without keying specially shot footage. When combined with Roto Brush we developed earlier, they give you a powerful, brand new workflow for rotoscoping.

Video Link Press

After Effects CS5: Roto Brush (2010)

The all-new Roto Brush tool in After Effects CS5 is largely based on our research on Video SnapCut. This is the KILLER feature of this AE version. it provides an alternative, faster workflow for this segmentation and creation of a matte, saving professional roto artist hours of work on a single footage.

Video Press

Other Projects/Tech Transfers

Camera, Low-level vision

    • Multi-frame Alignment, Denoising, HDR, Super-res
    • Video Denoising, HDR
    • Night Scene RAW Denoising
    • Multi-lens Fusion, Lens Distortion Removal
    • Single Camera Depth Estimation using Focus Stacking
    • 2PD Depth Estimation

Face Applications

Animation

Document

Image/Video Editing