Sample Animated Videos

For each example, we show the ground truth video, the reanimated video using the flow predicted from the ground truth initial frame, and the reanimated video using the flow predicted from the initial frame generated by MindVideo. The MindVideo initial frame represents the diffusion image as predicted by MindVideo [1] from fMRI data. As such, this frame is often quite different in content from the ground truth, although the objects in each are in similar locations. However, the motion predicted by our model is still consistent with ground truth motion, for example, in the first video the jellyfish retracts backwards at the end clip. The same backwards motion can be seen in the reanimated videos for with both the ground truth initial frame and the MindVideo generated initial frame (which happens to be a boat). This establishes that the motion visualized using DragNUWA [2] is not derived solely from the diffusion model, but rather, incorporates our predicted motion.

Encoding Models Voxel-wise Prediction Performance on Inflated Cortex

Here we show the voxel-wise fMRI prediction performance, quantified as the Pearson correlation (r) between measured and predicted responses, for the remaining visual encoding models in alphabetical order. We include the HCP-MMP Parcellation Map on an inflated cortex for reference here.

Note that all following plots of encoding accuracy on inflated cortical maps are on the same scale for ease of comparison. 

AF (averaged frames) models consist of model features averaged over frames spanning the video.