Session II (May 15, 10:30am-12:00pm): Online Experimentation, organized by Jonathan Stallrich

• Title: Collaborative Analysis for Paired A/B Tests  

• Speaker: Qiong Zhang, Clemson University

• Abstract: IT companies conduct a large number of A/B tests to evaluate different variations of products, algorithms, and interface designs. The experiments conducted by the same organization are often done to a common set of users. A user's response to different experiments can be highly correlated due to the user's individual effect. In this talk, I will discuss a framework to collaboratively design and analyze paired A/B tests, namely, a pair of A/B testing experiments conducted on the same set of experimental subjects. Through theoretical and numerical results, we demonstrate that the proposed design and analysis approach for paired A/B tests can produce more accurate estimates than the traditionally used single experiments analysis approach for A/B tests. This is a joint work with Xinwei Deng and Lulu Kang.