Surveys

This video discusses the purpose, core methodology, and advantages and disadvantages of using surveys in political research.  I discuss sampling techniques and sources of bias, using examples from the 2016 elections and polling about COVID.  At the end of this video, you should be able to define population, sampling frame, and sample, and understand the challenges of selection bias, measurement error, and sample size.

Donald Trump’s victory in the 2016 presidential election was a surprise for many reasons, but most notably because pre-election polls gave Hillary Clinton a strong lead over Trump, so much so that election forecasts (such as this one from fivethirtyeight.com) uniformly predicted Clinton would win.

There are many reasons why these election predictions were wrong.  Some relate to the US electoral system.  Many polls measured popularity across the nation, and these were actually pretty accurate.  On average, polls anticipated Clinton would will 48.5% of the vote, and Trump 45%.  Clinton ended up winning 48% of the popular vote, and Trump 46% -- a difference well within the margin of error.  But this isn’t how the U.S. elects presidents, and the differences in popularity across states meant a victory for Trump in the electoral college.

Other reasons for the inaccuracy relate to the technical details of polling – how different sources of information are weighted, or given priority.  Or they related to inaccurate interpretation by the media – media elites expected Trump to lose and read the results in a way that confirmed their assumptions.

But it is also true that there were some underlying difficulties in how the polls were conducted – who was included in the surveys – that under-represented Republicans and Trump supporters particularly.  In this video, I will discuss why we use polls and how we try to gather accurate samples of the people we are trying to study.

First off, why do we use surveys?  The short answer is that sometimes the best way to find out what people think is to ask them.  Surveys allow you to ask large groups of people the same questions.  This is great for the external validity of research – taking a large random sample of the population you are trying to study is the best way to ensure you get an accurate response – that the results you get actually reflect what people think.  But there are some disadvantages to survey research.  In order to get a large number of people to complete a survey, you need to ask a limited number of multiple-choice questions.  This leaves a lot of room for measurement error – if you ask a leading question or inappropriately limit the range of responses, you will skew your results.  It can also be hard to tell when people aren’t answering truthfully.  And while a large random sample does give you a good approximation of the views of the overall population, it is really hard to get a truly random sample.  Low survey response rates often generate selection bias in the results.

So surveys have advantages and disadvantages, but let’s discuss how to do a survey well.  First, some vocabulary:

Surveys are all about asking questions of a small group of people in order to understand the views of a larger group of people.  To use the technical terms, we care about four groups when we set up a survey.  First is the population – this is the full set of observations (or people) that you want to understand.  For example, if we are talking about U.S. presidential election polls, the population we are interested in is registered voters. 

The sampling frame is the list of people we use to develop the survey. Sticking with our example, it’s not the list of registered voters – it is the list of registered voters you have contact information for.  Voter registration records are public – it is possible to request voters’ names and addresses.  But if you want to do a phone or internet survey, you have more work to do to figure out how to contact survey participants.  Polling organizations used to develop contact lists using the phone book.  But since many Americans no longer have land lines – and don’t answer their mobile phones, this is no longer an effective way to conduct polls.  Most big pollsters now conduct polls over the internet.  They generate large-ish sampling frames by contacting individuals via mail and telephone.  Polling organizations then send emails to samples from that list of people who agreed to participate in surveys. 

The sample, then, is the group of people that you contact for the survey – the group of people you text or email to participate. 

The last group – not listed here – is the group of people who actually respond to the survey.  I’ll come back to this later, but the differences between who is asked to complete a survey and who actually does are critical.

I’ve mentioned the sample several times now, but how do we actually take an accurate sample of a population?  There are many strategies – I will mention four; two of which are good and two of which are less good.

For surveys, you want to make sure you take a probability sample.  This is when you know the likelihood of each person in the sampling frame being chosen for the sample.  A simple random sample is when everyone is equally likely to be asked to participate in a survey.  Respondents are chosen at random, which ensures they represent the overall population.  Sometimes you don’t actually want an exact representation of the population.  Recall the survey we read about at the beginning of the semester.  The student was researching non-citizen voting, but the number of non-citizens who responded to the survey she analyzed was very small.  If you want to study an underrepresented or small group, you often want to increase the number of participants in your survey from that group.  So, for example, the Pew Research Center now over-samples Republicans and other groups in order to account for past polling errors.  This is called taking a stratified sample.  You divide your sampling frame into groups, and then randomly select participants from each group in order to make sure you have enough respondents.

Let me conclude this section by noting a few less good ways to do polling.  You should never conduct a poll based on a convenience sample – a sample chosen based on who is available.  This is often how you do interview research, but I’ll discuss that in the next video.  The problem with convenience samples is that there is always bias in who is available – you cannot be sure if the people you spoke to reflect the population.

Similarly, opt-in samples also introduce both bias and error.  People who choose to participate in polls are different from people who choose not to.  Sometimes the results of opt-in surveys are entertaining.  The people who opt in to answer Internet polls – like the poll created to choose the name of a British polar research ship in 2016 – tend to choose results based on what they think will be funny.  Which is how Boaty McBoatface won that poll and generated many a meme.  Which is funny, but many of the polls used around the 2016 and 2020 elections were this type of opt-in poll, which easily generates bias and often include a large number of bad actors who choose to respond inaccurately.  Polling agencies are moving away from this method and increasing adopting the probability, stratified sample I described above, but this method is expensive and it’s not yet common.

I would like to conclude by discussing these sources of bias.  There are three main categories of bias: Selection bias, measurement error, and errors related to sample size (called statistical power). 

Selection bias happens in two ways.  One is when the group of people you contact for a survey doesn’t represent the population of interest.  The list of contact information you generated may not have accurately reflected the underlying population.  For example, if you conduct a poll based on land-line phone interviews, you will systematically underrepresent young people, who are more likely to only have a mobile phone. 

Response rates also generate bias.  In national polling today, Republican voters are less likely to respond to surveys than Democrats or Independents.  This gives greater weight to the Republicans who do choose to participate, who may not accurately reflect the Republican base.  Again, polling organizations have begun increasing the number of Republicans in a sample in order to compensate, but when one group of people systematically chooses not to answer questions, it creates uncertainty about the results.

The second category of bias is measurement error.  There are many ways you can generate biased answers – they depend on the questions you ask, who is asking the question, and the honesty of the respondents.  The way you phrase a question makes a huge impact on the type of answers you get.  We’ll discuss further in class, but for now, I will note that these differences are magnified when you are asking questions across languages and cultures.  Ideas like “democracy” have different meanings in different countries and can be translated many different ways.  The results of a polling question about support for democracy, then, will vary significantly based on the specific words you choose.

Even if you ask a good question, you may not get an honest answer.  Even on anonymous surveys, people often feel pressure to give the “right” answer.  For example, if you ask someone whether they voted in the last election, they are likely to say yes even if they didn’t vote.  This is called social desirability bias.  We can also see this in the results of polls about Covid vaccines taken during the pandemic.  This review of survey results from 2021 that asked whether participants had received a Covid vaccine demonstrates two of the problems I have been talking about.  Here, the grey line represents data on the actual number of vaccines given and the colored lines represent poll results for the question about whether someone had received a vaccine.  We can see that the top two polls, which were opt-in Internet polls, were the least accurate.  And we can see that all results were slightly biased upward – people were more likely to give the perceived “right” response. 

The last set of problems relate to the number of people in a sample.  Your sample needs to be large – the smaller the group, the less likely it is to represent the population.  And if you are going to analyze sub-groups (like non-citizens or ethnic groups), then you need to make sure each of the sub-groups is relatively large.  A typical size for a good survey is 1,000 respondents, with individual sub-groups having a few dozen respondents minimum.  So too small is bad, but so is too large.  To return to our Covid surveys, the least accurate surveys had the largest sample sizes.  But sample size can’t compensate for selection bias or measurement error.  As Charles Wheelan said about data, garbage in, garbage out.  In fact, having an extremely large poor sample is a worst case scenario.  The large sample size will drive the standard error of your analysis down – that is just how the underlying math works – so you will have statistically significant results that are wrong.

So to recap, when you take a probability sample of a population, polling can give you a good estimate of the views of that population.  Opt-in samples and interviewer error can generate selection bias, however, as can poorly phrased questions.  More difficult challenges to correct are bias in who responds to a survey and inaccurate responses.  Often the best way to get an accurate assessment is to try multiple methods, and next up we will talk about interview research.