A Naturalness Evaluation Database for Video Prediction Models

Downloads

Paper: arXiv

Database: Link

Code: Link

IISc VINE Database

By applying various Video Prediction models on various datasets, we generate a large collection of predicted videos. We select 300 videos from these to form the Indian Institute of Science - VIdeo Naturalness Evaluation (IISc VINE) Database.

In the following videos, we show example distortions observed in our Database. These videos correspond to the videos shown in the paper.

Different Distortions

Different Shape Distortions

Video Naturalness Evaluation - Our Proposed Method

For a given video, we compute deep features of pretrained networks such as VGG-19, ResNet-50 and Inception-v3. We further process them to get 2 sets of features. We use a shallow feed forward neural network to learn naturalness score from the computed features (Figure A).

1. Motion-compensated Cosine Similarity (MCS) features: We compute cosine similarity between deep features of last context frame and motion-compensated deep features of predicted frames (Figure B).

2. Rescaled Frame Difference (RFD) features: We compute difference of successive frames and rescale the diff frames in the range [0,255]. We then compute deep features of them and average spatially (Figure C).

Model Architecture

Illustration of Rescaled Frame Differences

The following videos illustrate how rescaled frame differences (RFD) capture the contour of objects under motion and shape distortions.

Shortcomings of Full Reference Measures

If you use our work, please cite our paper:

Nagabhushan Somraj, Manoj Surya Kashi, S P Arun, Rajiv Soundararajan, "A Naturalness Evaluation Database for Video Prediction Models", arXiv e-prints, p.arXiv:2005.00356

Bibtex:

@article{somraj2020vine,

title = {A Naturalness Evaluation Database for Video Prediction Models},

author = {Somraj, Nagabhushan and Kashi, Manoj Surya and Arun, S. P. and Soundararajan, Rajiv},

journal = {arXiv e-prints},

eid = {arXiv:2005.00356},

pages = {arXiv:2005.00356},

archivePrefix = {arXiv},

eprint = {2005.00356},

year = {2020}

}