The goal of this project is to create a video editor with some intelligent behavior. Specifically it will allow the user to automatically remove unwanted items from a scene. Optionally it could allow the user to pull out specific items and remove everything else similar to a green screen. We believe the students can accomplish this by combining two existing neural networks in a novel way. Specifically using a mask-rcnn, the user will be presented with objects it has identified in the scene. Viewing the segmented video, the user can select an item they want to remove. Using the selected mask from the mask-rcnn network the students can feed each frame into an auto-encoder trained for the task of image in-painting. The auto-encoder will essentially "paint over" the selected objects using the contextual information of the image to fill in the gaps. If done well this software could greatly speed up video editing time and essentially create automatic special effects.
To see the code for this project we have a Github where it is located:
https://github.com/yfrayer/magic-eraser
YouTube channel with example output videos: