Counterfactual Learning and Evaluation for Recommender Systems: from Foundations to Emerging Challenges in the Era of LLMs

[slides]

Abstract

Advances in online bandits and reinforcement learning enable us to solve complex decision-making tasks in problems like games, where agents learn through many trials and errors. However, it is often difficult to directly apply these techniques to problems related to recommender and search systems, as trial and error can be costly in terms of effectiveness and implementation. Instead, counterfactual learning and evaluation are often used in such scenarios because they enable batch learning and evaluation of decision-making agents by leveraging logged data. While the lack of direct interaction with the real environment avoids undesirable consequences, it also requires in-depth statistical knowledge for reliable deployment.

This tutorial begins with an overview of the field, highlighting fundamental formulations and methods. The goal of this section is to provide the audience with a solid background to actively engage in the discussion during the workshop. I will then discuss some of the emerging problems in the field that may limit the use and effectiveness of relevant techniques, such as non-stationarity, reward alignment, short- and long-term gaps, the presence of new actions, dealing with languages, and deterministic logging. By exploring these challenges, I aim to outline future research directions in this area, emphasizing the need for novel ideas that can more readily and effectively handle the complexities of real-world data.