Julie Beckley, Etsy

• Title: Improving Internet Experiment Validity

• Abstract: Large scale internet experiments are critical for making the best decisions for customers. While randomized controlled experiments are the cleanest way to measure treatment effects, ensuring experiments are behaving as expected is much harder. Based on my experience at Etsy and Netflix, I will go over several case studies of unexpected challenges and the methodological solutions we implemented to resolve them.