Weitao Duan, LinkedIn

• Title: Experiment Velocity and Trustworthiness at LinkedIn

• Abstract: Controlled experiments, or A/B tests, have been the gold standard for testing a product feature and making launch decisions. Many technology companies, such as Google, Facebook, LinkedIn, and Microsoft, have built large-scale in-house experimentation platforms and fully adopted A/B testing in their decision-making process. Despite the increasing engagement on the site, many of our experiments at LinkedIn suffer from low sample size and low experiment power. To be able to make inference, we adopted a different randomization unit to greatly increase experiment power. In this talk, we will share our lessons and learnings on this. In the second half of the talk, we will discuss the topic of experimentation within an advertising marketplace. We will show that the typical A/B test could lead to severe bias and introduce the budget split design to remove the cannibalization bias.