Projects
Research Topics
Research in IR, NLP, Text Mining and Machine Learning/Deep Learning. See Publications page and selected research projects for more details. The main topics are Deep Learning in QA/Conversations and IR/Ranking/Search/Ads/Recommender System.
Deep Learning in QA/Conversations and IR/Ranking/Search/Ads
Deep Learning for Information-seeking Conversations [SIGIR 2018a, SIGIR 2018b, CIKM 2018, WSDM 2019, ACL 2018, SIGIR 2017 NeuIR, CHIIR 2019]
Deep Learning for Question Answering [CIKM 2016]
Deep Learning for Ad-hoc Retrieval [SIGIR 2016, ICTIR 2016]
Non-factoid Answer Retrieval [ECIR 2016, SIGIR 2018, CHIIR 2019]
User Modeling & Personalization & Recommendation
Model User Reply Behavior in Enterprise Email Systems [SIGIR2017]
Model User Interests for Proactive Ranking [ECIR 2016]
Model User Topical Expertise and Interests [CIKM 2013]
Model User Interactions in Online Discussions [CIKM 2013, NAACL 2013]
Online User Profiling [PAKDD 2014, SocInfo 2013]
User Generated Content Mining [ICDM 2014, COLING 2014]
Selected Research Projects
Deep Learning for Information-seeking Conversations
Intelligent personal assistant systems (e.g. Apple Siri, Google Now, Amazon Alexa, and Microsoft Cortana) with either text-based or voice-based conversational interfaces are becoming increasingly popular around the world. Retrieval-based conversation models have the advantages of returning fluent and informative responses. Most existing studies in this area are on open domain ''chit-chat'' conversations or task / transaction oriented conversations. More research is needed for information-seeking conversations. In this project, we investigated:
1) External knowledge in information-seeking conversations: We proposed a learning framework on the top of deep neural matching networks that leverages external knowledge for response ranking in information-seeking conversation systems. We incorporate external knowledge into deep neural models with pseudo-relevance feedback and QA correspondence knowledge distillation. Extensive experiments with three information-seeking conversation data sets including both open benchmarks and commercial data show that, our methods outperform various baseline methods including several deep text matching models and the state-of-the-art method on response selection in multi-turn conversations.
2) User intent characterization in information-seeking conversations: We introduce a new customer service dialog dataset MSDialog and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns.
3) Transfer learning in information-seeking conversations: The current neural conversational models are generally not efficient for industrial applications, and they rely on a large amount of labeled data, which may not be available. We study transfer learning for multi-turn information seeking conversation systems. We propose an efficient and effective multi-turn conversation model based on convolutional neural networks. We further extend our model to adapt the knowledge learned from a resource-rich domain to further boost our model performance. We have deployed our model in an industrial bot application and observed a significant improvement over the existing online model.
4) Next question prediction in information-seeking conversations: We study the effectiveness of neural matching models for predicting the next question in conversations with the publicly available Ubuntu dialog data.
Related Publications
Liu Yang, Minghui Qiu, Chen Qu, Cen Chen, Jiafeng Guo, Yongfeng Zhang, Bruce Croft and Haiqing Chen. IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems. In Proceedings of The Web Conference 2020 (WWW 2020), Taipei, China, April 20-24, 2020. Short Oral Paper. Acceptance rate=24.6% (98 out of 397).
Liu Yang, Junjie Hu, Minghui Qiu, Chen Qu, Jianfeng Gao, W. Bruce Croft, Xiaodong Liu, Yelong Shen, Jingjing Liu. A Hybrid Retrieval-Generation Neural Conversation Model. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM 2019), Beijing, China, November 03-07, 2019. Full Oral Paper. Acceptance rate=19.4% (200 out of 1030).
Liu Yang, Response Retrieval in Information-seeking Conversations. PhD thesis, University of Massachusetts Amherst, 2019. [PDF][Bibtex][Talk Slides][Talk PDF]
Chen Qu, Liu Yang, W. Bruce Croft, Yongfeng Zhang, Johanne R Trippas and Minghui Qiu. User Intent Prediction in Information-seeking Conversations, To appear in Proceedings of the 2019 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2019), Glasgow, Scotland, UK, March 10-14, 2019. Full Paper.
Chen Qu, Feng Ji, Minghui Qiu, Liu Yang, Zhiyu Min, Haiqing Chen, Jun Huang and W. Bruce Croft. Learning to Selectively Transfer: Reinforced Transfer Learning for Deep Text Matching. To appear in Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM 2019), Melbourne, Australia, February 11-15, 2019. Full Oral Paper. Acceptance rate=16% (84 out of 511).
Minghui Qiu, Liu Yang, Feng Ji, Wei Zhou, Weipeng Zhao, Jun Huang, Haiqing Chen, W. Bruce Croft, Wei Lin. Transfer Learning for Context-Aware Question Matching in Information-seeking Conversation Systems in E-commerce. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), Melbourne, Australia, July 15-20, 2018. Short Paper. Acceptance rate=24% (126 out of 526) (CCF Rank A)
Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, Jun Huang, Haiqing Chen. Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems, In Proceedings of the 41th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, U.S.A. July 8-12, 2018. Full Oral Paper. Acceptance rate=21% (86 out of 409). (CCF Rank A)
Chen Qu, Liu Yang, W. Bruce Croft, Johanne R Trippas, Yongfeng Zhang, Minghui Qiu. Analyzing and Characterizing User Intent in Information-seeking Conversations, In Proceedings of the 41th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, U.S.A. July 8-12, 2018. Short Paper. Acceptance rate=30% (98 out of 327). (CCF Rank A)
Liu Yang, Hamed Zamani, Yongfeng Zhang, Jiafeng Guo, W. Bruce Croft. Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation, In Neu-IR 2017: The SIGIR 2017 Workshop on Neural Information Retrieval (SIGIR Neu-IR 2017), Tokyo, Japan, August 7-11, 2017. Oral Presentation. [Arxiv Version][Slides][Poster][Bibtex]
Deep Learning for Question Answering
As an alternative to question answering methods based on feature engineering, deep learning approaches such as CNNs and LSTMs have recently been proposed for semantic matching of questions and answers. To achieve good results, however, these models have been combined with additional features such as word overlap or BM25 scores. Without this combination, these models perform significantly worse than methods based on linguistic feature engineering. In this paper, we propose an attention based neural matching model for ranking short answer text. We adopt value-shared weighting scheme for combining different matching signals and incorporate question term importance learning using question attention network. Using the popular benchmark TREC QA data, we show that the relatively simple aNMM model can significantly outperform other neural network models that have been used for the question answering task, and is competitive with models that are combined with additional features. When aNMM is combined with additional features, it outperforms all baselines.
Related Publications
Liu Yang, Qingyao Ai, Jiafeng Guo, W. Bruce Croft. aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model, In Proceedings of the 25th ACM International Conference on Information and Knowledge Management (CIKM 2016), Indianapolis, IN, USA. October 24-28, 2016. Full Oral Paper. Acceptance rate=17.6% (165 out of 935). [PDF][Slides][Bibtex][Code in Java][Code with TensorFlow][ACL Wiki on QA][Arxiv Version]
Chen Qu, Liu Yang, Minghui Qiu, W. Bruce Croft, Yongfeng Zhang and Mohit Iyyer. BERT with History Answer Embedding for Conversational Question Answering, In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2019), Paris, France. July 21-25, 2019. Short Paper. Acceptance rate=24% (108 out of 443)
Chen Qu, Liu Yang, Minghui Qiu, Yongfeng Zhang, Cen Chen, W. Bruce Croft and Mohit Iyyer. Attentive History Selection for Conversational Question Answering. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM 2019), Beijing, China, November 03-07, 2019. Full Oral Paper. Acceptance rate=19.4% (200 out of 1030).
Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft and Mohit Iyyer. Open-Retrieval Conversational Question Answering. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020), Xi'an, China. July 25-30, 2020. Full Oral Paper. Acceptance rate=26% (147 out of 555)
Non-Factoid Question Answering / Answer Retrieval
Retrieving finer grained text units such as passages or sentences as answers for non-factoid Web queries is becoming increasingly important for applications such as mobile Web search. In this project, we introduce the answer sentence retrieval task for non-factoid Web queries, and investigate how this task can be effectively solved with learning to rank approaches and deep learning approaches. We made the following contributions:
Semantic and context features for non-factoid answer retrieval: We designed semantic and context features, beyond traditional text matching features for non-factoid answer retrieval. Results show that features used previously to retrieve topical sentences and factoid answer sentences are not sufficient for retrieving answer sentences for non-factoid queries, but with semantic and context features, we can significantly outperform the baseline methods.
A benchmark collection for non-factoid answer retrieval: Currently, there are no comparable collections that address non-factoid question answering within larger documents while simultaneously providing enough examples sufficient to train a deep neural network. We introduce a new Wikipedia based collection specific for non-factoid answer passage retrieval containing thousands of questions with annotated answers and show benchmark results on a variety of state of the art neural architectures and retrieval models.
Related Publications
Liu Yang, Qingyao Ai, Damiano Spina, Ruey-Cheng Chen, Liang Pang, W. Bruce Croft, Jiafeng Guo and Falk Scholer. Beyond Factoid QA: Effective Methods for Non-factoid Answer Sentence Retrieval. In Proceedings of the 38th European Conference on Information Retrieval (ECIR 2016), Padova, Italy, March 20-23, 2016. Full Oral Paper. Acceptance rate = 21%.[PDF][Data][Code][Slides][Poster][Bibtex]
Daniel Cohen, Liu Yang, W. Bruce Croft. WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval, In Proceedings of the 41th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, U.S.A. July 8-12, 2018. Short Paper. Acceptance rate=30% (98 out of 327). (CCF Rank A)
Chen Qu, Liu Yang, W. Bruce Croft, Falk Scholer and Yongfeng Zhang. Answer Interaction in Non-factoid Question Answering Systems, To appear in Proceedings of the 2019 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2019), Glasgow, Scotland, UK, March 10-14, 2019. Short Paper.
Deep Learning for Conversational Recommendation
A joint project in progress to enable online chat-bots to recommend products to users with conversation interactions. Typical applications include shopping guidance chat-bots in E-commerce websites.
Related References
Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016. Towards Conversational Recommender Systems. In KDD'16.
Yueming Sun, Yi Zhang. 2018. Towards Conversational Recommender System. In SIGIR'18.
Yueming Sun, et al. 2016. Conversational Recommendation System with Unsupervised Learning. In RecSys'16.
Related Publications
Yongfeng Zhang, Xu Chen, Qingyao Ai, Liu Yang, and W. Bruce Croft. Towards Conversational Search and Recommendation: System Ask, User Respond. To appear in Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM 2018), Turin, Italy, October 22-26, 2018. Full Oral Paper. Acceptance rate=17% (147 out of 862).[PDF]
Deep Learning for Ad-hoc Retrieval
Ad-hoc Retrieval is one of the most fundamental IR tasks. Incorporating topic level estimation into language models has been shown to be beneficial for information retrieval (IR) models such as cluster-based retrieval and LDA-based document representation. Neural embedding models, such as paragraph vector (PV) models, on the other hand have shown their effectiveness and efficiency in learning semantic representations of documents and words in multiple Natural Language Processing (NLP) tasks. However, their effectiveness in information retrieval is mostly unknown. In this project, we study how to effectively use the PV model to improve ad-hoc retrieval. We propose three major improvements over the original PV model to adapt it for the IR scenario: (1) we use a document frequency-based rather than the corpus frequency-based negative sampling strategy so that the importance of frequent words will not be suppressed excessively; (2) we introduce regularization over the document representation to prevent the model overfitting short documents along with the learning iterations; and (3) we employ a joint learning objective which considers both the document-word and word-context associations to produce better word probability estimation. By incorporating this enhanced PV model into the language modeling framework, we show that it can significantly outperform the state of-the-art topic enhanced language models.
Related Publications
Qingyao Ai, Liu Yang, Jiafeng Guo, W. Bruce Croft, Analysis of the Paragraph Vector Model for Information Retrieval, In Proceedings of The 2nd ACM International Conference on the Theory of Information Retrieval (ICTIR 2016). Newark, DE, USA. September 12-16, 2016. Full Oral Paper. [PDF][Slides][Bibtex]
Qingyao Ai, Liu Yang, Jiafeng Guo, W. Bruce Croft, Improving Language Estimation with the Paragraph Vector Model for Ad-hoc Retrieval, In Proceedings of the 39th Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2016). Pisa, Italy. July 18-10, 2016. Short Paper.[PDF][Bibtex] (CCF Rank A)
Modeling and Predicting Online User Topical Expertise and Interests
In this project, we investigate machine learning models for mining and predicting online user topical expertise and interests. Our work included:
Modeling user topical interests and expertise in CQA: Community Question Answering (CQA) websites, where people share expertise on open platforms, have become large repositories of valuable knowledge. To bring the best value out of these knowledge repositories, it is critically important for CQA services to know how to find the right experts, retrieve archived similar questions and recommend best answers to new questions. To tackle this cluster of closely related problems in a principled approach, we proposed a novel probabilistic generative model TEM with GMM hybrid to jointly model topics and expertise by integrating textual content model and link structure analysis. Based on TEM results, we proposed CQARank to measure user interests and expertise score under different topics. Experiments carried out on Stack Overflow data, the largest CQA focused on computer programming, show that our method achieves significant improvement over existing methods on multiple metrics.
Modeling user interests in proactive search systems: We study user modeling in proactive search systems such as information cards ranking system in Microsoft Cortana and propose a learning to rank method for proactive ranking. We explore a variety of ways of modeling user interests, ranging from direct modeling of historical interaction with content types to finer-grained entity-level modeling, and user demographic information. To reduce the feature sparsity problem in entity modeling, we propose semantic similarity features using word embedding and an entity taxonomy in knowledge base. Experiments performed with data from a large commercial proactive search system show that our method significantly outperforms a strong baseline method deployed in the production system.
Related Publications
Liu Yang, Minghui Qiu, Swapna Gottipati, Feida Zhu, Jing Jiang, Huiping Sun and Zhong Chen. CQARank: Jointly Model Topics and Expertise in Community Question Answering. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM 2013), San Francisco, CA, USA. October 2013. Full Oral Paper, Top 3 Cited Papers in CIKM'13 . Acceptance rate=16.8% (143 out of 848). [PDF][Slides][PPT][Bibtex][Code]
Liu Yang, Qi Guo, Yang Song, Sha Meng, Milad Shokouhi, Kieran McDonald and W. Bruce Croft. Modelling User Interest for Zero-query Ranking. In Proceedings of the 38th European Conference on Information Retrieval (ECIR 2016), Padova, Italy, March 20-23, 2016. Full Oral Paper. Acceptance rate = 21%.[PDF][Slides][Bibtex]
Modeling and Predicting Online User Behavior and Attribute/Profile
In this project, we investigate machine learning models for mining and predicting online user behavior and profile. Our work included:
Characterizing and predicting user email reply behavior: Email is still among the most popular online activities. People spend a significant amount of time sending, reading and responding to email in order to communicate with others, manage tasks and archive personal information. We extend previous work on predicting email reply behavior by looking at enterprise settings and considering more than dyadic communications. We characterize the influence of various factors such as email content and metadata, historical interaction features and temporal features on email reply behavior. We also develop models to predict whether a recipient will reply to an email and how long it will take to do so. Experiments with the publicly-available Avocado email collection show that our methods outperform all baselines with large gains. We also analyze the importance of different features on reply behavior predictions. Our findings provide new insights about how people interact with enterprise email and have implications for the design of the next generation of email clients.
Predicting user attribute and profile: We study how to model user online debate and discussions to predict user attribute/profile like political affiliations.
Related Publications
Liu Yang, Susan T. Dumais, Paul N. Bennett and Ahmed Hassan Awadallah. Characterizing and Predicting Enterprise Email Reply Behavior, In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017), Tokyo, Japan, August 7-11, 2017. Full Oral Paper. Acceptance rate=22% (78 out of 362).[PDF][Data][Slides][Bibtex] (CCF Rank A)
Swapna Gottipati, Minghui Qiu, Liu Yang, Feida Zhu, Jing Jiang. An Integrated Model for User Attribute Discovery: A Case Study on Political Affiliation Identification. In Proceedings of the 18th Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2014), Tainan, Taiwan, May 2014. Full Oral Paper. Acceptance rate=10.8% (40 out of 371).[PDF][Bibtex]
Swapna Gottipati, Minghui Qiu, Liu Yang, Feida Zhu and Jing Jiang. Predicting User's Political Party using Ideological Stances. In Proceedings of the 5th International Conference on Social Informatics (SocInfo 2013), Kyoto, Japan. November 2013. Full Oral Paper, Best Paper Runner-ups .[PDF][Slides][Bibtex]
Modeling and Predicting Online User Interactions and Relations
As the fast development of Web 2.0, millions of users generated vast amount of posts/messages in online forums (Reddit, Createdebate, Debatepedia, Douban), CQA sites (Yahoo Answers, Baidu Zhidao, Quora, Zhihu) and microblogs (Twitter, Weibo) everyday. In this project, we study how to mine user debate interactions/stances and user relations in online forums with machine learning models. Our work include:
Modeling user interactions in online forums: To automatically identify the sides/stances of posts or users from textual content in online discussion forums, it is important to exploit user posts that implicitly contain support and dispute (interaction) information and to mine such interactions from the content of posts. We proposes a two-stage solution based on latent variable models: an interaction feature identification stage to mine interaction features from structured debate posts with known sides and reply intentions; and a clustering stage to incorporate interaction features and model the interplay between interactions and sides for debate side clustering.
Modeling user relations in online forums: Advances in sentiment analysis have enabled extraction of user relations implied in online textual exchanges such as forum posts. However, recent studies in this direction only consider direct relation extraction from text. As user interactions can be sparse in online discussions, we propose to apply collaborative filtering through probabilistic matrix factorization to generalize and improve the opinion matrices extracted from forum posts. Experiments with two tasks show that the learned latent representation can give good performance on a relation polarity prediction task and improve the performance of a subgroup detection task.
Online forums text mining and summarization: We study how to summarize travel-related information in forum threads to generate supplementary travel guides. We propose to use a latent variable model to align forum threads with the section structure of well-written travel guides. The model also assigns section labels to named entities in forum threads. We then propose to modify an ILP-based summarization method to generate section-specific summaries. Evaluation on threads from Yahoo! Answers shows that our proposed method is able to generate better summaries compared with baselines based on ROUGE scores and coverage of named entities.
Related Publications
Minghui Qiu, Liu Yang and Jing Jiang. Mining User Relations from Online Discussions using Sentiment Analysis and Probabilistic Matrix Factorization. In Proceedings of the 2013 Conference of North American Chapter of Association for Computational Linguistics: Human Language Technologies (NAACL 2013), Atlanta, GA, USA. June 2013. Long Paper, Acceptance rate=30% (88 out of 293). [PDF][Bibtex][Code][Data]
Minghui Qiu, Liu Yang and Jing Jiang. Modeling Interaction Features for Debate Side Clustering. In Proceedings of the 22nd ACM International Conference on Information and Knowledge Management (CIKM 2013), San Francisco, CA, USA. October 2013. Short Paper, Acceptance rate=12.5% (106 out of 848). [PDF][Bibtex]
Jianguang Du, Jing Jiang, Liu Yang, Dandan Song, Lejian Liao. ShellMiner: Mining Organizational Phrases in Argumentative Texts in Social Media. In Proceedings of the 14th IEEE International Conference on Data Mining (ICDM 2014 ), Shenzhen, China, December 14-17, 2014. Short Paper. Acceptance rate = 19.7% (143 out of 727). [PDF][Bibtex]
Liu Yang, Jing Jiang, Lifu Huang, Minghui Qiu and Lizi Liao. Generating Supplementary Travel Guides from Social Media. In Proceedings of the 25th International Conference on Computational Linguistics (COLING 2014), Dublin, Ireland, August 23-29 2014. Full Oral Paper. [PDF][Slides][Data][Bibtex]
Professors and Friends
UMass&SMU&CAS&PKU Prof. W. Bruce Croft, Prof. Jing Jiang, Prof. Feida Zhu, Prof. Jiafeng Guo, Prof. James Allan, Dr. Hang Li(Toutiao), Prof. Xueqi Chen, Prof. Jun Xu, Prof. Yanyan Lan, Prof. Andrew McCallum, Prof.Subhransu Maji, Prof. Benjamin M. Marlin, Prof. Daniel R. Sheldon, Prof. Mohit Iyyer, Prof. Brendan O'Connor, Prof. Ee Peng Lim, Prof. Hady W. Lauw, Prof. David Lo, Prof. Zhong Chen, Prof. Huiping Sun, Prof. Xiaojun Wan, Prof. Zhihong Deng, etc.
Friends Minghui Qiu(Alibaba), Swapna Gottipati(SMU), Qiming Diao(SMU), Ming Gao(SMU), Ying Ding (SMU), Qi Guo(Google), Wayne Xin Zhao(PKU), Ennan Zhai(Yale), Kangjie Lu(GIT), Da Yu(Brown), Lilong Jiang(OSU), Jitong Chen(Baidu), Wenpeng Yin(UMunich), Lifu Huang(RPI), Junjie Yao(UCSB), Jiepu Jiang(UMass), Qingyao Ai (UMass), Weize Kong (UMass), Chia-Jung Lee(UMass), Pan Hu (UMass), Xiang Li(UMass), Hamed Zamani(UMass), Chenyan Xiong (CMU&MSR), Liang Pang(CAS), Yixing Fan(CAS), Lixin Su(CAS), Zhaochun Ren(JD), Zhiyuan Chen(Google), Siyuan Liu(PSU), Qiang Qu(Aarhus), Wei Xie(SMU), Zhiyong Cheng(NUS), Cheng Li(Umich), Linfeng Song (Rochester), etc.
Active Researchers in IR/NLP/DM/ML: Yi Chang (Huawei), Liangjie Hong blog(LinkedIn), Dawei Yin(Baidu), William Wang(UCSB), Rui Yan (PKU), Jiwei Li(Stanford), Arvind Neelakantan (UMass), Mostafa Dehghani (UA), Rishabh Mehrotra (UCL), Yuening Hu(Yahoo!), Guangbin Huang(NTU), Ting Liu (Google), Kangjie Lu (Gatech&UMN), Ennan Zhai (Yale), Yida Wang (Princeton & Amazon), Honglei Liu (Facebook), Jian Tang (MILA), Yu Zheng (Microsoft & JD), Mu Li (MSRA), Xiangnan He (NUS), Heng Ji (RPI), Yi Tay (NTU), Tuomas Sandholm (CMU), Furu Wei (MSRA), Sam Zhang (NTU), Hongbo Deng (Alibaba), Jingbo Zhu (NEU)