A Naturalness Evaluation Database for Video Prediction Models
Nagabhushan Somraj, Manoj Surya Kashi, Dr S P Arun and Dr Rajiv Soundararajan
IISc VINE Database
By applying various Video Prediction models on various datasets, we generate a large collection of predicted videos. We select 300 videos from these to form the Indian Institute of Science - VIdeo Naturalness Evaluation (IISc VINE) Database.
In the following videos, we show example distortions observed in our Database. These videos correspond to the videos shown in the paper.
Different Distortions
Blur
Shape Distortion
Disappearance
Color Change
Different Shape Distortions
Deformation
Splitting
Partial Disappearance
Elongation
Video Naturalness Evaluation - Our Proposed Method
For a given video, we compute deep features of pretrained networks such as VGG-19, ResNet-50 and Inception-v3. We further process them to get 2 sets of features. We use a shallow feed forward neural network to learn naturalness score from the computed features (Figure A).
1. Motion-compensated Cosine Similarity (MCS) features: We compute cosine similarity between deep features of last context frame and motion-compensated deep features of predicted frames (Figure B).
2. Rescaled Frame Difference (RFD) features: We compute difference of successive frames and rescale the diff frames in the range [0,255]. We then compute deep features of them and average spatially (Figure C).
Model Architecture
Illustration of Rescaled Frame Differences
The following videos illustrate how rescaled frame differences (RFD) capture the contour of objects under motion and shape distortions.
Video 1 - Partial Disappearance
Video 1 - Rescaled Frame Differences
Video 2 - Deformation
Video 2 - Rescaled Frame Differences
Shortcomings of Full Reference Measures
Ground Truth 1
Predicted Video 1
Ground Truth 2
Predicted Video 2
If you use our work, please cite our paper:
Nagabhushan Somraj, Manoj Surya Kashi, S P Arun, Rajiv Soundararajan, "A Naturalness Evaluation Database for Video Prediction Models", arXiv e-prints, p.arXiv:2005.00356
Bibtex:
@article{somraj2020vine,
title = {A Naturalness Evaluation Database for Video Prediction Models},
author = {Somraj, Nagabhushan and Kashi, Manoj Surya and Arun, S. P. and Soundararajan, Rajiv},
journal = {arXiv e-prints},
eid = {arXiv:2005.00356},
pages = {arXiv:2005.00356},
archivePrefix = {arXiv},
eprint = {2005.00356},
year = {2020}
}