Experiment Design

Introduction to Experiment Design

An experiment is a systematically designed procedure to test our informed hypotheses. Running an experiment is fundamentally different than observational studies where researchers collect data without intentionally changing the participant's natural behaviors. In an experiment, a researcher carefully designs an intervention to establish cause-and-effect between variables of interest. One may question the utility of experiments as cause-and-effect relationships could be established using pre-existing data. For example, the effect of a father’s education on a child’s athletic performance. One may find a positive correlation between a father’s education and a child’s performance. But can we conclude that highly educated parents cause a more athletic child? Not necessarily. Highly educated fathers may occupy high-paying jobs and could afford housing communities with more sports facilities. So, the effect which seems to be coming from the father’s education level may come from the community. The causal variables, in this case, housing community, are called confounders. We can never guarantee causal relationships with observational data without controlling for the confounders which could be a daunting task. Experiments come to our rescue by providing an easy alternative to hold fair comparison in terms of handling confounders.

In this digital age, digital field experiments provide us best of two worlds (lab and field experiments): a) conducted in less controlled environments, b) opportunity to have more representative sample, and c) much accurate estimates. Notably, digital field experiments are not limited to online experiments. One such example is the automatic personality assessment (APA) system that we designed to understand users’ perceptions of APA systems. Participants’ position within the workplace was detected using Bluetooth sensors and collected through a smartwatch app. Such methods are superior in terms of accuracy and data is collected unobtrusively to capture participants’ natural behaviors. In addition, platforms like Mturk provide researchers an easy venue to conduct lab-like experiments online. Nowadays, researchers are being innovative in designing experiments that answer beyond simple queries. For example, designing treatments based on susceptible different effects on different demographics. Furthermore, designing such experiments only requires attention to the three fundamental elements: 1) validity, 2) heterogeneity of treatment effects, and 3) mechanisms (Salganik, 2019). I will further delve into each of these concepts in the next post.

Resources

Salganik, M. J. (2019). Bit by bit: Social research in the digital age. Princeton University Press.