Deep Transfer Learning for Search and Recommendation

-- A WWW 2020 Tutorial

Overview

Training data sparsity is a common problem for many real-world applications in Search and Recommendation domains. Even for applications with a lot of training data, in the cold-start scenario we usually do not get enough labeled data. Transfer Learning is a promising approach for addressing this problem. In addition, features might interact with each other in a complex way that traditional approaches might not be able to represent, Deep Transfer Learning, which leverages Deep Neural Networks for Transfer Learning, might be able to catch such deep patterns hidden in complex feature interactions. Due to these reasons, recently Deep Transfer Learning research has gained a lot of attention and has been successfully applied to many real-world applications. This tutorial offers an overview of Deep Transfer Learning approaches in Search and Recommendation domains from the industry perspective. In this tutorial We first introduce the basic concepts and major categories of Deep Transfer Learning. Then we focus on recent developments of Deep Transfer Learning approaches in the Search and Recommendation domains. After that we will introduce two real-world examples of how to apply Deep Transfer Learning methods to improve Search and Recommendation performance at LinkedIn. Finally we will conclude the tutorial with discussion of future directions.


Contributors

Yang Yang (Senior Staff Software Engineer at LinkedIn Inc., USA),

Yanen Li (Engineering Manager at LinkedIn Inc., USA),

Sen Zhou (Senior Software Engineer at LinkedIn Inc., USA),

Jian Qiao, (Senior Software Engineer at LinkedIn Inc., USA),

Mingyuan Zhong, (Staff Software Engineer at LinkedIn Inc., USA),

Bo Long (Engineering Director at LinkedIn Inc., USA)


Tutorial Material

Slides and other materials are under preparation.

Tutorial Logistics

Venue/Time: 2:00 - 4:30pm on Tuesday, April 21, 2020 in Room 1 [WWW'20 webpage (schedule)]

Tutorial Outline and Description

Introduction

  1. Overview of Search and Recommendation

  2. Transfer Learning in Search and Recommendation

Deep Transfer Learning

  1. Preliminaries

  2. Problem Settings of Deep Transfer Learning
    • Inductive Transfer Learning
    • Transductive Transfer Learning • Unsupervised Transfer Learning

  3. Common Approaches of Deep Transfer Learning

Instance based Deep Transfer Learning
• Pre-training and Model Fine-tuning
• Multi-task Learning
• Model Distillation
• Privacy-preserving Transfer Learning

Deep Transfer Learning in Search and Recommendation

  1. Domain Adaptation
    • Adversarial Discriminative Domain Adaptation
    • Instanced-based Selective Deep Transfer Learning using
    Adversarial Networks

  2. Learning Unified Embeddings by Multi-task Learning
    • Multi-task Learning for Uni
    fied Entity and Text Embed- dings
    • Multi-task Learning for multi-type user behavior

  3. Model Distillation
    • Modeling Compression by Model Distillation for Search and Recommendation

  4. Privacy-preserving transfer learning

Case Study

  1. An End-to-end Example of Learning Unified Embeddings by Deep Transfer Learning at LinkedIn

  2. Adversary Network-based Selective Deep Transfer Learning for Data Augmentation for LinkedIn Applications

Presenters' Bios

Yang Yang is a Senior Staff Software Engineer and Tech Lead at LinkedIn. Before joining LinkedIn, Yang worked at Yahoo! Labs as a Scientist. She obtained her Ph.D. degree at Department of statistics, University of Michigan. She has produced various papers and patents on applying statistical methods and machine learning approaches to real data problem involving large scale data. She has published in conferences and journals including KDD, WWW, PAM, Statistical Analysis and Data Mining, The Canadian Journal of Statistics, IIE Transactions on Healthcare Systems Engineering, and Statistical Analysis for High-Dimensional Data.


Yanen Li leads the AI Features Foundation team at LinkedIn. He has broad interest in Machine Learning/AI and their applications in Search and Recommendations. He received his Ph.D. in Computer Science from the University of Illinois at Urbana-Champaign. He has published 30+ publications in top conferences and journals including KDD, WWW, SIGIR, CIKM, EMNLP etc. with 1000+ of ci- tations. He has been serving committees in multiple top conferences such as KDD, WWW, ACL, EMNLP, WSDM etc.

Sen Zhou is a Senior Software Engineer at LinkedIn. Before joining Linkedin, Sen obtained his Ph.D. degree from Department of EECS, University of California, irvine, working on data fusion and fault-tolerance in wireless sensor networks.


Jian Qiao, is a Senior Software Engineer at LinkedIn. Before joining Linkedin, Sen obtained his Master degree from Department of EECS, University of California, Berkeley.


Bo Long leads LinkedIn’s AI Foundations team. He has 15 years of experience in data mining and machine learning with applications to web search, recommendation, and social network analysis. He holds dozens of innovations and has published peer- reviewed papers in top conferences and journals including ICML, KDD, ICDM, AAAI, SDM, CIKM, and KAIS. He has served as re- viewers, workshops co-organizers, conference organizer committee members, and area chairs for multiple conferences, including KDD, NIPS, SIGIR, ICML, SDM, CIKM, JSM etc.