In this video, I define survey experiments and offer examples of three common types of this research design. To summarize, survey experiments are surveys where respondents are exposed to one of two questions designed to expose them to a treatment. Survey experiments are good for obtaining more honest answers to sensitive questions and for testing the impact of framing policies in different ways. I will begin by clarifying the difference between a survey and a survey experiment, then will describe how to design three types: information experiments, list experiments, and endorsement experiments. I conclude by discussing the strengths and weaknesses of survey experiments as a methodology.
Surveys vs Survey Experiments
As you will recall, surveys can represent the opinion of a population of interest if you have a good sampling frame to draw a random sample of participants from. The representativeness of your survey depends on how well you take that random sample. Survey experiments also require a representative, random sample. They differ from regular surveys in the type of questions you ask. In addition to taking a random sample of your population, you also need to write questions that can serve as a treatment for an experiment. You write two versions of the same question – the difference is your treatment – and expose a random half of the survey respondents to each version of the question. I’ll offer examples of that in a moment, but for now, the key point is that surveys have random samples, while survey experiments have both random samples and randomized questions.
The other thing to remember as we go through the types of experiments is that they don’t all demonstrate causality. Information experiments can show that a frame causes a reaction, but list and endorsement experiments are different ways of measuring an opinion. So the other key difference between surveys and survey experiments is that surveys just measure opinion – they don’t establish what causes opinion. But some survey experiments can actually show what causes opinion.
Three Types of Survey Experiments
To make this more clear, I’ll talk about three types of survey experiments: information experiments, list experiments, and endorsement experiments.
Information experiments involve presenting the respondent with information. You present a random half with one version of the scenario and the other half a different version, and this can help you understand the impact of that information on individual opinion and decision making. For example, this experiment evaluated how college students make decisions about what to major in. The researchers presented the control group general information about the income of college graduates. But they showed the treatment groups information about the average income of graduates with different types of majors. They then asked both groups questions about what they are considering as a major.
What’s important to remember in designing this type of experiment is that your treatment be strong. The information presented needs to be clear enough, and significant enough, to actually impact the reader or viewer’s decision-making. It needs to be concise enough that participants will actually read it – not just skim. And it needs to fully capture whatever you are trying to measure. We’ll go over some examples of how this works in class.
In addition, your control needs to be equivalent to your treatment. So in this example, the researchers didn’t show the control group nothing – they provided related information, but not information that would vary by major.
Information experiments – also called framing or vignette experiments – are true experiments. They test the effect of information on decision-making. List and endorsement experiments are different in that they are primarily about measurement. They both can help you obtain more accurate responses when you think survey participants might falsify their preferences – that they might not give you true answers.
List experiments involve presenting your control and treatment groups two lists of items. These lists are identical except for one thing – the thing you are trying to measure. You then ask survey respondents to select how many things on the list are correct or they agree with. For example, some respondents might not be willing to state outright that they don’t think a woman should be president. This experiment tried to get more honest answers to this question by presenting a list of statements. Half of the respondents saw the first four points, and the other half – the treatment group – saw five points, including “A woman serving as president.” They were then asked how many of those statements upset them. The researchers then subtracted the average number of responses for the control group from the average number of “upset statements” from the treatment group. If the difference is statistically significant, that shows that there are some people upset at the idea of a woman serving as president.
When designing a list experiment, remember to keep your list fairly short. You also need to write other items on the list that some people will agree and disagree with, but that don’t relate to the treatment. For example, it would have been a bad idea to include a statement about female representation on the Supreme Court as another item on the example list because it also deals with gender discrimination.
Finally, an endorsement experiment is designed to obtain more honest opinions about support for individuals or groups. This is usually only done when stating support for a group might be controversial. For example, Lyall, Blair, and Imai conducted a survey experiment in Afghanistan to gauge support for the Taliban. The way this works is to present information about a specific policy or idea. They presented the proposal in a neutral manner to the control group, but told the treatment group that the Taliban supported this proposal. This technique takes advantage of the fact that most people don’t have firm opinions about most policy issues, and take cues from leaders or groups they agree with when deciding what they support. So if an individual agrees with the Taliban generally, they are more likely to say they support the proposal the Taliban support. The key element with designing these questions is to choose a policy that is plausible, but not polarizing – not something people are likely to have firm opinions on. So in the United States, don’t use abortion policy or taxes, but consider something like transportation policy.
Advantages and disadvantages
Survey experiments are a popular research design because they can offer the best of both worlds – the external validity of a survey of a large number of individuals with the internal validity of an experiment. They use randomization to eliminate alternate explanations, and because they can be sent to a large, representative sample of a population, you can be more confident that you can generalize to the wider population. But of course, all the challenges of surveys apply here – you need to have obtained a good sample of the population to guarantee external validity and need to design good questions for internal validity.
Done well, however, survey experiments can be an excellent way to measure public opinion and decision-making.