Research & Tech Transfer
My research has been transferred into numerous products with millions of users
Adobe’s Firefly Video Model
A journey of text-to-video:
Generate Video on Firefly Web App
https://blog.adobe.com/en/publish/2024/10/14/generate-video-beta-on-firefly-web-app
Adobe’s AI video model is here, and it’s already inside Premiere Pro
https://www.theverge.com/2024/10/14/24268695/adobe-ai-video-generation-firefly-model-premiere-pro
Project Blink (2023)
After three years of development, our team announced Project Blink, the next generation video app powered by AI. Search for words, images, people, and moments in a video then cut and paste just as they do with a text document.
Super Quality AI Portrait (2019)
Got a bad selfie? No worries! We present a deep network that fixes all the problems in your face images - noise, blur, out-of-focus, low light, low-res, compression, etc. Click here to see examples.
OPPO Reno: Ultra Night Mode (2019)
With the OPPO Reno Ultra Night Mode, you can now capture amazing dark night scenes of the city, sunsets, parties and dinners without compromising on the quality of your photos.
Lenovo S5 Pro: AI 3D Face Reshaping (2018)
We shipped this brand-new face-transforming feature to Lenovo mobile phones announced on March 20, 2018. We trained a network that analyzes a huge collection of "beautiful faces" and generates a 3D face model that transforms the input image. We made a lot of optimization so that the algorithm runs at 60 fps on a mobile phone!
Face-aware Wide-angle Lens Correction (2017)
We developed a technique that corrects the distortion in wide-angle lens while preserving the faces. Note the faces near the image edges. The algorithm is super efficient and runs in real-time.
Burst Image Denoising and Super-res (2018)
Super fast multi-frame denoising algorithm on mobile phones. The technique has been adopted by multiple mobile phone companies to enhance pictures taken by the telephoto-lens.
Adobe After Effects: Camera Shake Deblur (2017)
The new video filter, Camera-Shake Deblur, reduces motion blur artifacts resulting from camera shake. This effect can save shots that might not otherwise be usable due to slight camera bumps or camera motion blur that remains after stabilization.
Portrait Beautify and Virtual Makeup SDK (2017)
A comprehensive feature set that includes skin clean-up, skin tone adjustment, digital makeup and face reshape, etc. Everything runs in real time on mobile!
Adobe Dimension CC: Auto IBL (2017)
This is a key feature in Dimension, a 3D compositing software that makes 3D easy for 2D graphics designers. It completes background images to 360-degree panoramas and approximates lighting and tone-mapping simultaneously.
Lightroom & Photoshop CC: Boundary Warp (2016)
Boundary Warp is designed to solve the issue of irregular boundaries when creating stitched panoramas. It analyzes the boundary of your panorama and then warps the image so the edges fit nicely inside a rectangular frame while all pixels are preserved.
Premiere Elements 15: Real-time Video Dehaze (2016)
Front page feature! This new Haze Removal enables you to easily remove haze to make the background of your scenes as crisp as what’s up front. We have done tremendous optimization to the algorithm. The filter processes 1080p videos in real-time!
After Effects CC: Face Tracking (2015)
After Effects CC: Key Cleaner (2014)
The Key Cleaner effect recovers detail from a scene keyed by a typical keying effect, including recovering detail from a keyed scene with compression artifacts.
The Advanced Spill Suppressor effect removes color spill on a foreground subject from a colored background used for color keying.
AE & Premiere Pro: Detail-preserving Upscale (2013)
The Detail-preserving Upscale effect is capable of scaling up images by large amounts while preserving details in the image. The sharpness of sharp lines and curves is preserved. For example, you can scale up from SD frame sizes to HD frame sizes, or from HD frame sizes to digital cinema frame sizes.
After Effects CS6: Refine Edge Tool (2012)
Internal code name: "Hair Brush". It is a video matting tool that keeps the details when separating complicated foreground elements like frizzy hair or motion-blurred edges from complex backgrounds. This means you can create natural-looking composites without keying specially shot footage. When combined with Roto Brush we developed earlier, they give you a powerful, brand new workflow for rotoscoping.
After Effects CS5: Roto Brush (2010)
The all-new Roto Brush tool in After Effects CS5 is largely based on our research on Video SnapCut. This is the KILLER feature of this AE version. it provides an alternative, faster workflow for this segmentation and creation of a matte, saving professional roto artist hours of work on a single footage.
Other Projects/Tech Transfers
Camera, Low-level vision
Multi-frame Alignment, Denoising, HDR, Super-res
Video Denoising, HDR
Multi-lens Fusion, Lens Distortion Removal
Single Camera Depth Estimation using Focus Stacking
2PD Depth Estimation
Face Applications
Perfect Group Photo for Live Photos
Virtual Face Light (Android App)
Animation
Animating Still Images
Object Tracker for Character Animator
Document
Document Image Parsing and Enhancement for Acrobat Pro
Image/Video Editing
Soft Shadow Removal
Halide Acceleration for After Effects
Light Wrap Fg/Bg Compositing for After Effects