Slides Download Link

Abstract

As an interesting research problem in conversational AI, Question Answering (QA) has been surveyed from various perspectives recently [1]. In this tutorial, we focus on goal-oriented QA systems [2, 3, 4], which aim at guiding users to complete a specific task like finding a job, booking a ticket etc. via a conversational bot, in contrast to the large amount of social bots in Alexa [5], and the engineering designs and all of the components involved in building an end-to-end system in practice.

Depending on the setting of answer retrieval, QA systems can be categorized as structured data based systems [6, 16, 17, 18] and unstructured data based systems [7, 15]. Structured data based QA systems produce answers from a closed and structured data source, including modern knowledge graphs and traditional databases. Unstructured data based QA systems extract answers from free text by combining the information retrieval and the information extraction. While structured data based QA systems can provide well-formatted accurate answers in closed domains and are thus more suitable for goal-oriented bots, the unstructured data based QA systems can cover wider user intents via search. In industry, a practical goal-oriented bot leverages both with a focus on structured data based QA systems.

Structured data based QA systems semantically parse the question into a formal query language executable in knowledge graph or database. Varied by different semantic parsing process, structured data based QA systems are grouped as ontology-based systems [8] and intent classification based systems [9]. Ontology-based QA systems take questions and an ontology as input, and associate question features with answer patterns of a knowledge base via the ontology file. Despite their generic framework in semantic understanding, the ontology-compliant systems [10] tend to not outperform simpler models in public benchmark datasets [13, 14]. On the other hand, intent classification simply classifies questions into predefined answer patterns needed by goal oriented bots in a knowledge base. The challenge boils down to the training data preparation for each predefined answer pattern, which can be more easily tackled in an industrial environment.

In this tutorial, we first introduce a variety of QA systems based on knowledge graph and intent classification proposed by pioneer researchers. The audience can easily comprehend what common technical components remain challenging and what unique engineering heuristics are useful. For the first time, the audience can learn in-depth not only the scientific methods that boost the precision of question understanding and answer retrieval / generation [11], but also our practical experiences as well as engineering designs that enable an end-to-end system [12]. Then, on top of three LinkedIn real scenarios, we share our hands-on experiences in the end-to-end process of building goal oriented bots, including problem analysis from scratch, architecture design, training data collection, paraphrase generation, intent modeling and dialogue management. Our goal is that, after this tutorial, the audience knows how to efficiently build a goal-oriented bot without getting stuck in unrealistic solutions.

References

[1] Question Answering with Knowledge Base, Web and Beyond. Scott Wen-tau Yih and Hao Ma, SIGIR 2016.

[2] POMDP-Based Statistical Spoken Dialog Systems: A Review. Young, S., Gasic, M., Thomson, B., and Williams, J. D., Proceedings of the IEEE, 2013.

[3] Learning End-to-end Goal-oriented Dialog. Antoine Bordes, Y-Lan Boureau and Jason Weston, ICLR 2017.

[4] End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient. Li Zhou, Kevin Small, Oleg Rokhlenko and Charles Elkan, https://arxiv.org/pdf/1712.02838.pdf

[5] Alexa Skills. https://www.amazon.com/alexa-skills/b/ref=skillsrw_surl?ie=UTF8&node=1 3727921011

[6] Information Extraction over Structured Data: Question Answering with Freebase. Xuchen Yao and Benjamin Van Durme, ACL 2014.

[7] Natural language question answering: The view from here. L. Hirschman and R. Gaizauskas, Journal of Natural Language Engineering, 2001.

[8] Is Question Answering fit for the Semantic Web?: a Survey. Vanessa Lopeza, Victoria Urenb, Marta Sabouc and Enrico Motta. Semantic Web, 2011.

[9] More Accurate Question Answering on Freebase. Hannah Bast and Elmar Haussmann, CIKM 2015.

[10] Survey on Challenges of Question Answering in the Semantic Web. Konrad Höffner, Sebastian Walter, Edgard Marx, Ricardo Usbeck, Jens Lehmann and Axel-Cyrille Ngonga Ngomo. Semantic Web, 2017.

[11] End-To-End Memory Networks. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston and Rob Fergus. NIPS, 2015.

[12] Building Watson: An Overview of the DeepQA Project. David Ferrucci et al. AAAI Magazine, 2010.

[13] Webquestions benchmark leaderboard https://worksheets.codalab.org/worksheets/0xba659fe363cb46e7a505c5b6 a774dc8a/

[14] SQUAD benchmark leaderboard https://rajpurkar.github.io/SQuAD-explorer/

[15] Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M. Rush, Bart van Merriënboer, Armand Joulin and Tomas Mikolov. ICLR, 2016

[16] Learning to parse database queries using inductive logic programming. M. Zelle and R. J. Mooney. AAAI, 1996

[17] Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. L. S. Zettlemoyer and M. Collins. UAI, 2005.

[18] Semantic Parsing on Freebase from Question-Answer Pairs. Jonathan Berant, Andrew Chou, Roy Frostig and Percy Liang. EMNLP, 2013.