4.10 Survey Biases

Instrumentation bias arises when the tools used to collect survey answers change.

Imagine going out to eat at a brand new restaurant in Allston. After you finish your meal, someone from the staff asks if you are willing to answer some survey questions about your experience. You are in a bit of a rush, so you try to politely decline – but then the manager offers you a free meal voucher in exchange for your responses.

Now, close your eyes and picture each of the following scenarios:

The manager hands you a tiny index card, along with a pencil, and asks you to respond to several 1-5 rating scale questions, along with several open-ended questions.
The manager hands you an iPad, and asks you to scroll through a series of a series of questions, with a mix of 1-5 ratings and open-ended opinion prompts.
The manager holds onto the iPad, but asks you the same set of questions.
The manager sends you a text message with the survey questions that you can answer on your own, at any time within the next week (the link will expire in 7 days)

How will each of these four survey delivery modes impact your responses? If you think that your answers would be completely consistent, regardless of instrumentation, you might want to think harder about this question.

With the index card method, you will be able to write anonymously, which should make you feel more “free” with your answers. It is also possible that the act of writing with the pencil could make your hand tired, thereby limiting how much you write for the open-ended questions. Furthermore, the space available on the index card is an important constraint.

Option 2 frees you up from some of the constraints that come with Option 1. You are less likely to get hand cramps, or to run out of space. However, you are aware that you are temporarily “borrowing” someone else’s expensive piece of equipment – it’s possible that such awareness could make you more deferential than you would be with your answers in Format 1.

With option 3, it could be much harder to give candid answers. Of course, the restaurant manager is not your boss, and – objectively speaking – you should not be afraid to hurt his feelings. However, it would be much harder to deliver critical feedback in this way, knowing that your answers could trigger disapproving glances from the manager or even defensive rebuttals.

The major risk with option 4 is that you will simply forget to do the survey. If you do remember, then many of the constraints and problems associated with the other options will be removed. However, as more time goes by, your memory of the dining experience will start to fade, and the likelihood that you will actually complete the survey diminishes, too.