4 Kinds of Survey Error -

Sampling, Measurement, Coverage and Non-Response

Dana Stanley

There are 4 generally-accepted types of survey error. By survey error, I mean factors which reduce the accuracy of a survey estimate.

It’s important to keep each type of survey error in mind when designing, executing and interpreting surveys. However, I suspect some of them are more ingrained in our thinking about research, while others are more often neglected.

Imagine if we interviewed 100 researchers and asked each of them (“Family Feud”-style) to name a type of survey error.

Which type of survey error do you think would be mentioned most frequently? Which type would be most overlooked?

Here is my predicted order of finish in our hypothetical example.

Note for the “Feud”-challenged: Number 1 represents the most commonly named type of error in our hypothetical survey of researchers, while number 4 represents the least commonly named.

1. Sampling Error.

My guess is that sampling error would be the most commonly named type of survey error.

In a recent Research Access post, “How to Plus or Minus: Understand and Calculate the Margin of Error,” I explained the concept of sampling error and gave 3 ways of calculating it.

Sampling error is essentially the degree to which a survey statistic differs from its “true” value due to the fact that the survey was conducted among only one of many possible survey samples. It is a degree of uncertainty that we are willing to live with. Even most non-researchers have a basic understanding, or at least awareness, of sampling error due to the media’s reference to the “margin of error” when reporting public survey results.

2. Measurement Error.

I believe measurement error would be the second most frequently named type of error. Measurement error is the degree to which a survey statistic differs from its “true” value due to imperfections in the way the statistic is collected. The most common type of measurement error is one researchers deal with on a daily basis: poor question wording, with faulty assumptions and imperfect scales. (It can be the most devastating, though)

3. Coverage Error.

Coverage error is another important source of variability in survey statistics; it is the degree to which statistics are off due to the fact that the sample used does not properly represent the underlying population being measured.

There was generally more concern about coverage error in the past; these days, the combination of increasing internet penetration and fast/easy/cheap online survey panels has made it possible to accurately represent many target populations. Concern about coverage error is still an important conversation; however, it is being discussed more in academic and thought-leadership circles than by the average day-t0-day research practitioner.

4. Non-Response Error.

My guess is that non-response error would be the least named type of error in our hypothetical survey. Telephone survey houses historically have routinely made 20 or more call-backs to households that do not answer the telephone. This practice has dwindled due to a combination of the expense of conducting so many call-backs, and the dramatic growth of online surveys, where it is just easier to replace non-responders with fresh sample. It is also not considered acceptable in an online context to conduct scores of follow-up emails; that would get the sender sent to a blacklist post haste.

http://researchaccess.com/2011/11/4-kinds-of-survey-error-sampling-measurement-coverage-nonresponse/