Abstract

We present a novel problem of text-based visual question generation or TextVQG in short. Given the recent growing interest of the document image analysis community in combining text understanding with conversational artificial intelligence, e.g., text-based visual question answering, TextVQG becomes an important task. TextVQG aims to generate a natural language question for a given input image and an automatically extracted text also known as OCR token from it such that the OCR token is an answer to the generated question. TextVQG is an essential ability for a conversational agent. However, it is challenging as it requires an in-depth understanding of the scene and the ability to semantically bridge the visual content with the text present in the image. To address TextVQG, we present an OCR-consistent visual question generation model that Looks into the visual content, Reads the scene text, and Asks a relevant and meaningful natural language question. We refer to our proposed model as OLRA. We perform an extensive evaluation of OLRA on two public benchmarks and compare them against baselines. Our model – OLRA automatically generates questions similar to the public text-based visual question answering datasets that were curated manually. Moreover, we ‘significantly’ outperform baseline approaches on the performance measures popularly used in text generation literature.

Citation

If you find OLRA useful, please consider citing:

@InProceedings{10.1007/978-3-030-86549-8_22,

author="Jahagirdar, Soumya

and Gangisetty, Shankar

and Mishra, Anand",

editor="Llad{\'o}s, Josep

and Lopresti, Daniel

and Uchida, Seiichi",

title="Look, Read and Ask: Learning to Ask Questions by Reading Text in Images",

booktitle="Document Analysis and Recognition -- ICDAR 2021",

year="2021",

publisher="Springer International Publishing",

address="Cham",

pages="335--349"

}

Contact

• Soumya Shamarao Jahagirdar - soumyasj22@gmail.com