Home

Workshop on the Ethics of Online Experimentation

February 22, 2016

San Francisco, California

Experimentation has become standard practice among the developers and operators of online services. Search, news, social media, and ecommerce platforms deploy them regularly and extensively. Engineers have turned to online experimentation systems because they are a source of reliable evidence, because they are quick to deliver actionable insights, and because they are increasingly simple to deploy. Many in industry now find themselves in the position of running large-scale experiments that involve research subjects often numbering in the hundreds of thousands, with wide-ranging, computer-automated variations in experimental treatment. As experimentation platforms and users become easily accessible, however, practitioners may increasingly put the well-being and trust of users at risk.

While the recent controversy involving experiments on Facebook brought these issues to greater attention, the much more wide-ranging role that experimentation plays across online services has yet to generate comparable reflection. The discussion has been rather narrowly focused on cases involving experiments explicitly concerned with human behavior. But even experiments that simply aim to improve the function of technical systems are, increasingly, experiments on humans because they involve systematic variation in users’ experience of these systems. These include manual experiments such as so-called slow-down experiments to study user response to page load times but also extend to automated experiments using explore-exploit algorithms. Difficult ethical issues are not limited to the emerging field of computational social science and pervade many of the common practices of industry.

In light of these concerns, companies often review online experiments before they are actually conducted. In production settings, the review process might vary with respect to formality or standards across companies and even groups within companies. When intended or used for academic publication, experiments or data may have undergone inconsistent review processes, some implementing academic-style institutional review boards and others none at all. Although service providers may be concerned with the wellbeing of users, the community does not yet have a framework for “best practices” when it comes to reviewing online experiments, be they for internal or external consumption.

This workshop therefore aims to draw together researchers from inside and outside of the computer science community to jointly identify and discuss the ethical issues raised by the specific kinds of experiments that are a routine part of running a production online service. The goal is to provide an opportunity for productive dialogue grounded on a much deeper appreciation for the way practitioners devise and conduct experiments and the ethical and legal frameworks that exist to address research involving human subjects.