Projects

Research Topics


Research in IR, NLP, Text Mining and Machine Learning/Deep Learning. See Publications page and selected research projects for more details. The main topics are Deep Learning in QA/Conversations and IR/Ranking/Search/Ads/Recommender System.

  • Deep Learning in QA/Conversations and IR/Ranking/Search/Ads

    • Deep Learning for Information-seeking Conversations [SIGIR 2018a, SIGIR 2018b, CIKM 2018, WSDM 2019, ACL 2018, SIGIR 2017 NeuIR, CHIIR 2019]

    • Deep Learning for Question Answering [CIKM 2016]

    • Deep Learning for Ad-hoc Retrieval [SIGIR 2016, ICTIR 2016]

    • Non-factoid Answer Retrieval [ECIR 2016, SIGIR 2018, CHIIR 2019]

  • User Modeling & Personalization & Recommendation

    • Model User Reply Behavior in Enterprise Email Systems [SIGIR2017]

    • Model User Interests for Proactive Ranking [ECIR 2016]

    • Model User Topical Expertise and Interests [CIKM 2013]

    • Model User Interactions in Online Discussions [CIKM 2013, NAACL 2013]

    • Online User Profiling [PAKDD 2014, SocInfo 2013]

    • User Generated Content Mining [ICDM 2014, COLING 2014]

Selected Research Projects

  • Deep Learning for Information-seeking Conversations

    • Intelligent personal assistant systems (e.g. Apple Siri, Google Now, Amazon Alexa, and Microsoft Cortana) with either text-based or voice-based conversational interfaces are becoming increasingly popular around the world. Retrieval-based conversation models have the advantages of returning fluent and informative responses. Most existing studies in this area are on open domain ''chit-chat'' conversations or task / transaction oriented conversations. More research is needed for information-seeking conversations. In this project, we investigated:

      • 1) External knowledge in information-seeking conversations: We proposed a learning framework on the top of deep neural matching networks that leverages external knowledge for response ranking in information-seeking conversation systems. We incorporate external knowledge into deep neural models with pseudo-relevance feedback and QA correspondence knowledge distillation. Extensive experiments with three information-seeking conversation data sets including both open benchmarks and commercial data show that, our methods outperform various baseline methods including several deep text matching models and the state-of-the-art method on response selection in multi-turn conversations.

      • 2) User intent characterization in information-seeking conversations: We introduce a new customer service dialog dataset MSDialog and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns.

      • 3) Transfer learning in information-seeking conversations: The current neural conversational models are generally not efficient for industrial applications, and they rely on a large amount of labeled data, which may not be available. We study transfer learning for multi-turn information seeking conversation systems. We propose an efficient and effective multi-turn conversation model based on convolutional neural networks. We further extend our model to adapt the knowledge learned from a resource-rich domain to further boost our model performance. We have deployed our model in an industrial bot application and observed a significant improvement over the existing online model.

      • 4) Next question prediction in information-seeking conversations: We study the effectiveness of neural matching models for predicting the next question in conversations with the publicly available Ubuntu dialog data.

    • Related Publications

      • Liu Yang, Minghui Qiu, Chen Qu, Cen Chen, Jiafeng Guo, Yongfeng Zhang, Bruce Croft and Haiqing Chen. IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems. In Proceedings of The Web Conference 2020 (WWW 2020), Taipei, China, April 20-24, 2020. Short Oral Paper. Acceptance rate=24.6% (98 out of 397).

      • Liu Yang, Junjie Hu, Minghui Qiu, Chen Qu, Jianfeng Gao, W. Bruce Croft, Xiaodong Liu, Yelong Shen, Jingjing Liu. A Hybrid Retrieval-Generation Neural Conversation Model. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (CIKM 2019), Beijing, China, November 03-07, 2019. Full Oral Paper. Acceptance rate=19.4% (200 out of 1030).

      • Liu Yang, Response Retrieval in Information-seeking Conversations. PhD thesis, University of Massachusetts Amherst, 2019. [PDF][Bibtex][Talk Slides][Talk PDF]

      • Chen Qu, Liu Yang, W. Bruce Croft, Yongfeng Zhang, Johanne R Trippas and Minghui Qiu. User Intent Prediction in Information-seeking Conversations, To appear in Proceedings of the 2019 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2019), Glasgow, Scotland, UK, March 10-14, 2019. Full Paper.

      • Chen Qu, Feng Ji, Minghui Qiu, Liu Yang, Zhiyu Min, Haiqing Chen, Jun Huang and W. Bruce Croft. Learning to Selectively Transfer: Reinforced Transfer Learning for Deep Text Matching. To appear in Proceedings of the 12th ACM International Conference on Web Search and Data Mining (WSDM 2019), Melbourne, Australia, February 11-15, 2019. Full Oral Paper. Acceptance rate=16% (84 out of 511).

      • Minghui Qiu, Liu Yang, Feng Ji, Wei Zhou, Weipeng Zhao, Jun Huang, Haiqing Chen, W. Bruce Croft, Wei Lin. Transfer Learning for Context-Aware Question Matching in Information-seeking Conversation Systems in E-commerce. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), Melbourne, Australia, July 15-20, 2018. Short Paper. Acceptance rate=24% (126 out of 526) (CCF Rank A)

      • Liu Yang, Minghui Qiu, Chen Qu, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, Jun Huang, Haiqing Chen. Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems, In Proceedings of the 41th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, U.S.A. July 8-12, 2018. Full Oral Paper. Acceptance rate=21% (86 out of 409). (CCF Rank A)

      • Chen Qu, Liu Yang, W. Bruce Croft, Johanne R Trippas, Yongfeng Zhang, Minghui Qiu. Analyzing and Characterizing User Intent in Information-seeking Conversations, In Proceedings of the 41th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, U.S.A. July 8-12, 2018. Short Paper. Acceptance rate=30% (98 out of 327). (CCF Rank A)

      • Liu Yang, Hamed Zamani, Yongfeng Zhang, Jiafeng Guo, W. Bruce Croft. Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation, In Neu-IR 2017: The SIGIR 2017 Workshop on Neural Information Retrieval (SIGIR Neu-IR 2017), Tokyo, Japan, August 7-11, 2017. Oral Presentation. [Arxiv Version][Slides][Poster][Bibtex]

  • Deep Learning for Question Answering

    • As an alternative to question answering methods based on feature engineering, deep learning approaches such as CNNs and LSTMs have recently been proposed for semantic matching of questions and answers. To achieve good results, however, these models have been combined with additional features such as word overlap or BM25 scores. Without this combination, these models perform significantly worse than methods based on linguistic feature engineering. In this paper, we propose an attention based neural matching model for ranking short answer text. We adopt value-shared weighting scheme for combining different matching signals and incorporate question term importance learning using question attention network. Using the popular benchmark TREC QA data, we show that the relatively simple aNMM model can significantly outperform other neural network models that have been used for the question answering task, and is competitive with models that are combined with additional features. When aNMM is combined with additional features, it outperforms all baselines.

    • Related Publications

  • Non-Factoid Question Answering / Answer Retrieval

    • Retrieving finer grained text units such as passages or sentences as answers for non-factoid Web queries is becoming increasingly important for applications such as mobile Web search. In this project, we introduce the answer sentence retrieval task for non-factoid Web queries, and investigate how this task can be effectively solved with learning to rank approaches and deep learning approaches. We made the following contributions:

      • Semantic and context features for non-factoid answer retrieval: We designed semantic and context features, beyond traditional text matching features for non-factoid answer retrieval. Results show that features used previously to retrieve topical sentences and factoid answer sentences are not sufficient for retrieving answer sentences for non-factoid queries, but with semantic and context features, we can significantly outperform the baseline methods.

      • A benchmark collection for non-factoid answer retrieval: Currently, there are no comparable collections that address non-factoid question answering within larger documents while simultaneously providing enough examples sufficient to train a deep neural network. We introduce a new Wikipedia based collection specific for non-factoid answer passage retrieval containing thousands of questions with annotated answers and show benchmark results on a variety of state of the art neural architectures and retrieval models.

    • Related Publications

      • Liu Yang, Qingyao Ai, Damiano Spina, Ruey-Cheng Chen, Liang Pang, W. Bruce Croft, Jiafeng Guo and Falk Scholer. Beyond Factoid QA: Effective Methods for Non-factoid Answer Sentence Retrieval. In Proceedings of the 38th European Conference on Information Retrieval (ECIR 2016), Padova, Italy, March 20-23, 2016. Full Oral Paper. Acceptance rate = 21%.[PDF][Data][Code][Slides][Poster][Bibtex]

      • Daniel Cohen, Liu Yang, W. Bruce Croft. WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval, In Proceedings of the 41th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2018), Ann Arbor, Michigan, U.S.A. July 8-12, 2018. Short Paper. Acceptance rate=30% (98 out of 327). (CCF Rank A)

      • Chen Qu, Liu Yang, W. Bruce Croft, Falk Scholer and Yongfeng Zhang. Answer Interaction in Non-factoid Question Answering Systems, To appear in Proceedings of the 2019 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR 2019), Glasgow, Scotland, UK, March 10-14, 2019. Short Paper.

  • Deep Learning for Conversational Recommendation

    • A joint project in progress to enable online chat-bots to recommend products to users with conversation interactions. Typical applications include shopping guidance chat-bots in E-commerce websites.

    • Related References

      • Konstantina Christakopoulou, Filip Radlinski, and Katja Hofmann. 2016. Towards Conversational Recommender Systems. In KDD'16.

      • Yueming Sun, Yi Zhang. 2018. Towards Conversational Recommender System. In SIGIR'18.

      • Yueming Sun, et al. 2016. Conversational Recommendation System with Unsupervised Learning. In RecSys'16.

    • Related Publications

  • Deep Learning for Ad-hoc Retrieval

    • Ad-hoc Retrieval is one of the most fundamental IR tasks. Incorporating topic level estimation into language models has been shown to be beneficial for information retrieval (IR) models such as cluster-based retrieval and LDA-based document representation. Neural embedding models, such as paragraph vector (PV) models, on the other hand have shown their effectiveness and efficiency in learning semantic representations of documents and words in multiple Natural Language Processing (NLP) tasks. However, their effectiveness in information retrieval is mostly unknown. In this project, we study how to effectively use the PV model to improve ad-hoc retrieval. We propose three major improvements over the original PV model to adapt it for the IR scenario: (1) we use a document frequency-based rather than the corpus frequency-based negative sampling strategy so that the importance of frequent words will not be suppressed excessively; (2) we introduce regularization over the document representation to prevent the model overfitting short documents along with the learning iterations; and (3) we employ a joint learning objective which considers both the document-word and word-context associations to produce better word probability estimation. By incorporating this enhanced PV model into the language modeling framework, we show that it can significantly outperform the state of-the-art topic enhanced language models.

    • Related Publications

  • Modeling and Predicting Online User Topical Expertise and Interests

    • In this project, we investigate machine learning models for mining and predicting online user topical expertise and interests. Our work included:

      • Modeling user topical interests and expertise in CQA: Community Question Answering (CQA) websites, where people share expertise on open platforms, have become large repositories of valuable knowledge. To bring the best value out of these knowledge repositories, it is critically important for CQA services to know how to find the right experts, retrieve archived similar questions and recommend best answers to new questions. To tackle this cluster of closely related problems in a principled approach, we proposed a novel probabilistic generative model TEM with GMM hybrid to jointly model topics and expertise by integrating textual content model and link structure analysis. Based on TEM results, we proposed CQARank to measure user interests and expertise score under different topics. Experiments carried out on Stack Overflow data, the largest CQA focused on computer programming, show that our method achieves significant improvement over existing methods on multiple metrics.

      • Modeling user interests in proactive search systems: We study user modeling in proactive search systems such as information cards ranking system in Microsoft Cortana and propose a learning to rank method for proactive ranking. We explore a variety of ways of modeling user interests, ranging from direct modeling of historical interaction with content types to finer-grained entity-level modeling, and user demographic information. To reduce the feature sparsity problem in entity modeling, we propose semantic similarity features using word embedding and an entity taxonomy in knowledge base. Experiments performed with data from a large commercial proactive search system show that our method significantly outperforms a strong baseline method deployed in the production system.

    • Related Publications

  • Modeling and Predicting Online User Behavior and Attribute/Profile

    • In this project, we investigate machine learning models for mining and predicting online user behavior and profile. Our work included:

      • Characterizing and predicting user email reply behavior: Email is still among the most popular online activities. People spend a significant amount of time sending, reading and responding to email in order to communicate with others, manage tasks and archive personal information. We extend previous work on predicting email reply behavior by looking at enterprise settings and considering more than dyadic communications. We characterize the influence of various factors such as email content and metadata, historical interaction features and temporal features on email reply behavior. We also develop models to predict whether a recipient will reply to an email and how long it will take to do so. Experiments with the publicly-available Avocado email collection show that our methods outperform all baselines with large gains. We also analyze the importance of different features on reply behavior predictions. Our findings provide new insights about how people interact with enterprise email and have implications for the design of the next generation of email clients.

      • Predicting user attribute and profile: We study how to model user online debate and discussions to predict user attribute/profile like political affiliations.

    • Related Publications

  • Modeling and Predicting Online User Interactions and Relations

    • As the fast development of Web 2.0, millions of users generated vast amount of posts/messages in online forums (Reddit, Createdebate, Debatepedia, Douban), CQA sites (Yahoo Answers, Baidu Zhidao, Quora, Zhihu) and microblogs (Twitter, Weibo) everyday. In this project, we study how to mine user debate interactions/stances and user relations in online forums with machine learning models. Our work include:

      • Modeling user interactions in online forums: To automatically identify the sides/stances of posts or users from textual content in online discussion forums, it is important to exploit user posts that implicitly contain support and dispute (interaction) information and to mine such interactions from the content of posts. We proposes a two-stage solution based on latent variable models: an interaction feature identification stage to mine interaction features from structured debate posts with known sides and reply intentions; and a clustering stage to incorporate interaction features and model the interplay between interactions and sides for debate side clustering.

      • Modeling user relations in online forums: Advances in sentiment analysis have enabled extraction of user relations implied in online textual exchanges such as forum posts. However, recent studies in this direction only consider direct relation extraction from text. As user interactions can be sparse in online discussions, we propose to apply collaborative filtering through probabilistic matrix factorization to generalize and improve the opinion matrices extracted from forum posts. Experiments with two tasks show that the learned latent representation can give good performance on a relation polarity prediction task and improve the performance of a subgroup detection task.

      • Online forums text mining and summarization: We study how to summarize travel-related information in forum threads to generate supplementary travel guides. We propose to use a latent variable model to align forum threads with the section structure of well-written travel guides. The model also assigns section labels to named entities in forum threads. We then propose to modify an ILP-based summarization method to generate section-specific summaries. Evaluation on threads from Yahoo! Answers shows that our proposed method is able to generate better summaries compared with baselines based on ROUGE scores and coverage of named entities.

    • Related Publications

Professors and Friends