4.9 The Major Types of Survey Errors

All experiments and surveys have errors. In other words, they all contain a degree of uncertainty because at the end of the day, they are based on inferences about the larger population from a sample. These estimates of certainty are reflected in the confidence interval and margin of error. If confidence intervals are too wide, then it does not become meaningful for us to extrapolate meaning from the data. Therefore, understanding survey errors is key to understanding our survey quality.

One type of error is coverage error. This occurs when some groups are not included in the researcher’s initial sample.  Suppose, for example, at the college mentioned in the previous section, nearly 20 percent of the students are commuters who live off-campus.  If the researcher only distributes surveys within the residential dormitories on campus, he will systematically (i.e. non-randomly) exclude a large segment of the population.

Non-response error arises when certain members of a population are less likely to be represented in a survey.  This can occur with satisfaction surveys given by companies to their consumers – such surveys are likely to be ignored by those who are not either completely thrilled or totally dissatisfied.  Companies may mitigate against this by offering vouchers, coupons, or raffle entries to customers who respond.  

Sampling error can impact survey results when a company surveys too few customers and then draws broad conclusions from the results. One telltale sign of a small sample size is a wide confidence interval.  Sampling error can be reduced by using larger sample sizes, and/or surveying more representative segments of the target population.

Measurement error can also impact survey results.  This most commonly occurs when survey respondents are asked to self-report information.  Self-reporting can make people stronger, thinner, more honest, more capable, and smarter than they would be if measurements and results were collected by another person.