DAE 2024 - Session II

Session II (May 15, 10:30am-12:00pm): Online Experimentation, organized by Jonathan Stallrich

Title: Towards Optimal Variance Reduction in Online Controlled Experiments
Speaker: Ying Jin, Stanford University
Abstract: We study optimal variance reduction solutions for count and ratio metrics in online controlled experiments. Our methods leverage flexible machine learning tools to incorporate covariates that are independent from the treatment but have predictive power for the outcomes, and employ the cross-fitting technique to remove the bias in complex machine learning models. We establish CLT-type asymptotic inference based on our estimators under mild convergence conditions. Our procedures are optimal (efficient) for the corresponding targets as long as the machine learning estimators are consistent, without any requirement for their convergence rates. In complement to the general optimal procedure, we also derive a linear adjustment method for ratio metrics as a special case that is computationally efficient and can flexibly incorporate any pre-treatment covariates. We evaluate the proposed variance reduction procedures with comprehensive simulation studies and provide practical suggestions regarding commonly adopted assumptions in computing ratio metrics. When tested on real online experiment data from LinkedIn, the proposed optimal procedure for ratio metrics can reduce up to 80\% of variance compared to the standard difference-in-mean estimator and also further reduce up to 30\% of variance compared to the CUPED approach by going beyond linearity and incorporating a large number of extra covariates.

Title: Novelty and Primacy: A Long-Term Estimator for Online Experiments
Speaker: Sol Sadeghi, Netflix
Abstract: Online experiments are the gold standard for evaluating impact on user experience and accelerating innovation in software. However, since experiments are typically limited in duration, observed treatment effects are not always stable, sometimes revealing increasing or decreasing patterns over time. There are multiple causes for a treatment effect to change over time. In this article, we focus on a particular cause, user-learning, which is primarily associated with novelty or primacy. Novelty describes the desire to use new technology that tends to diminish over time. Primacy describes the growing engagement with technology as a result of adoption of the innovation. Estimating user-learning is critical because it holds experimentation responsible for trustworthiness, empowers organizations to make better decisions by providing a long-term view of expected impact, and prevents user dissatisfaction. In this article, we propose an observational approach, based on difference-in-differences technique to estimate user-learning at scale. We use this approach to test and estimate user-learning in many experiments at Microsoft. We compare our approach with the existing experimental method to show its benefits in terms of ease of use and higher statistical power, and to discuss its limitation in presence of other forms of treatment interaction with time.

Title: A/B tests with Unobserved Network Spillover: Design and Inference
Speaker: Jonathan Stallrich, NC State University
Abstract: A/B testing or online controlled experiment (OCE) is a ubiquitous technique for comparing the effect of two or more treatments in online settings. The classical randomized design approach to implementing controlled experiments relies on the stable unit treatment value assumption (SUTVA), which states that the outcome of an individual is independent of the treatment assigned to any other individual. It is well-known that SUTVA is violated in online platforms due to a phenomenon known as network interference, where individuals are connected via a social network, and the treatment assigned to one individual can influence the outcomes of its neighbors. Over the last decade or so, the classical randomized design has been largely supplanted by network clustering-based designs in A/B tests to account for this phenomenon. We show that there are two problems with network clustering based designs. First, the network itself is often unobserved or challenging/expensive to measure, which means network clustering cannot be implemented to begin with. Second, there are almost always lurking variables, which are unobserved user features that influence both user response and network formation. We demonstrate that the presence of lurking variables makes network clustering-based estimators biased. We propose a two-stage design and estimation technique called HODOR (Hold-Out Design for Online Randomized experiments) to address both problems. Remarkably, HODOR is based on the classical random design, albeit with a correction to deal with network interference. We carry out a theoretical analysis of HODOR to prove its unbiasedness and to establish its optimal configuration. We also develop a statistical inference framework based on HODOR to carry out hypothesis tests and construct confidence intervals. Through simulation studies and real world data examples, we compare the empirical performance of HODOR to the classical randomized as well as the network clustering-based design.

Title: Collaborative Analysis for Paired A/B Tests
Speaker: Qiong Zhang, Clemson University
Abstract: IT companies conduct a large number of A/B tests to evaluate different variations of products, algorithms, and interface designs. The experiments conducted by the same organization are often done to a common set of users. A user's response to different experiments can be highly correlated due to the user's individual effect. In this talk, I will discuss a framework to collaboratively design and analyze paired A/B tests, namely, a pair of A/B testing experiments conducted on the same set of experimental subjects. Through theoretical and numerical results, we demonstrate that the proposed design and analysis approach for paired A/B tests can produce more accurate estimates than the traditionally used single experiments analysis approach for A/B tests. This is a joint work with Xinwei Deng and Lulu Kang.