Assignment 8

I continued to explore the first-order motion for image animation model I used in last Tuesday's class.

app.runwayml.com/models/runway/First-Order-Motion-Model

Model biography:

Image animation is able to generate a video sequence in which a source image is animated according to the motion of a driving video.

The model is trained with a large collection of video sequences containing objects of the same object category. The model is trained to "reconstruct the training videos by combining a single frame and a learned latent representation of the motion in the video. Observing frame pairs (source and driving), each extracted from the same video, it learns to encode motion as a combination of motion-specific keypoint displacements and local affine transformations".(First Order Motion Model for Image Animation)

For my experiment, I tried two different things: using the camera as the driving video and using video as the driving video.The source image is always the anime image.

This is the one I used my camera as the driving video. The outcome is very unstable but I can see that the image somehow moving along with my motion.

I then changed it to meme video that provided maybe more stable input. Yet it didn't work very well. So I thought maybe it is because the driving video is not a human face and my source image is an anime character. But the documentation of the model said that the model was trained with non human dataset(robotarm.etc) and was able to capture a variety of motions.

In the end, I changed my source video to a man singing. But the outcome was like the previous ones. The motion was very lagging. Maybe it was caused by internet lagging or perhaps the issue was with my source image being an anime character. The model can't process it as a real human face.