Image, Video Annotation/Retrieval
The goal of the project was to study how partially labelled training data could be leveraged for the task of image or video annotation or retrieval. To address this challenge, we proposed a novel algorithm based on Active Learning. The algorithm outputs an order of labeling the training data, which allows us to obtain a good performance at annotation/retrieval of images or videos even before the training data is fully annotated. At the heart of our algorithm is the idea of "informativeness" as shown in the figure. We conducted experiments to test the effectiveness of our algorithm. For our experiments, we created a new dataset, which is open to the community for purposes of conducting academic research. The videos in the dataset were processed using Computer Vision techniques to compute low-level visual descriptors which were then used for experimentation.
We created a live demo of our system as well. Here, is a snapshot from the same. The image shows a sample video retrieval for a query with the concept "arms".
In further extension to our work, we did the following:
We explored a novel ranking function for performing active learning based image/video retrieval. This new ranking function, the core of which is shown in the equation above, mitigates some of the weaknesses of the previous ranking models.
The project was being executed under my leadership and the results have been published in multiple conferences of international repute.
Publications:
M. Chatterjee, A. Leuski, “CRMActive: An Active Learning Based Approach for Effective Video Annotationand Retrieval”, ACM Int’l Conf. on Multimedia Retrieval 2015(ACM ICMR 2015).
M. Chatterjee, A. Leuski, “A Novel Statistical Approach for Image and Video Retrieval and ItsAdaption for Active Learning”, ACM Int’l Conf. on Multimedia 2015(ACM MM 2015).
Non-Peer Reviewed:
M. Chatterjee, A. Leuski, “An Active Learning Based Approach for Effective Video Annotation and Retrieval”, arXiv preprint arXiv:1504.07004 (CoRR.abs 2015).
M. Chatterjee, "Minimizing Image or Video Annotation Effort Using Effective Relevance Models", Technical Report, 2015
Dataset Link:
Project Funding Support:
We are extremely grateful towards Dreamworks Animations SKG and the U.S. DARPA for providing funding support to the project.