In a previous blog, we went over the basics of crafting effective survey questions. But did you know a second (and surprising) factor can also wreak havoc on your survey results? The order of survey questions can actually “have a greater impact on the result than the particular choice of words used in the questions.”
Typically, a good rule of thumb is to organize questions much like a conversation. In other words, questions “should be grouped by topic and unfold in a logical order." Such structure and flow can “guide your respondents in a way that makes sense to them and delivers results to you." However, it is not foolproof! Be sure to lookout for these common problems:
- Loss of Motivation: When you group together too many difficult or boring questions, survey respondents can quickly lose interest. After all, who wants to answer 10 consecutive questions on their income tax filing? Disinterested respondents, in turn, often abandon surveys prior to completion.
Solution: Start your survey with questions that are relatively simple and engaging to “establish rapport and motivate [respondents] to continue." In the remainder of the survey, sprinkle in the dull or challenging questions—but never more than a few in a row.
- Stereotype threat: Asking sensitive questions (about gender, race, religion, income etc.) early in a survey is a recipe for disaster. It can make some respondents uncomfortable, prompting them to exit the survey. And for the respondents who stay, it can skew their answers to ensuing questions. For instance, research has revealed that “when girls identify their gender before a math test, they perform worse on the test”, as if conforming to expectations.
Solution: Reserve sensitive questions for the end of the survey. Or hire a firm like Op4G, which collects sensitive demographic information during the panel registration process.
- Priming: Sometimes, the information presented in a question “can prime respondents to think about [that information] while answering the subsequent question." This can influence responses in a few ways. First, it can make the response to a later question more similar to that of a former question (known as an “assimilation effect”). For example, when a respondent reports high satisfaction early in the survey, “subsequent responses will probably show a high level of satisfaction too." Second, it can make the response to a later question more divergent from that of a former question (a “contrast effect”). For example, when a respondent provides an extremely positive response early in the survey, he/she may aim to balance it out with a more negative answer later on.
Solution: Randomize the order of survey pages or questions, particularly those that could have a priming effect.
So we’ve covered how to order survey questions…but what about survey answers? As it turns out, the order of answer options can also sway respondents. In self-administered surveys, for example, “respondents tend to prefer the first few options in a list." But for phone or in-person surveys, respondents generally favor later options, perhaps because they are easier to recall. Again, this calls for randomization! Mix up the order of answer options during survey design.
Op4G’s operational staff has over 100 years of combined market research experience and provides survey design, optimization, and consultation upon request. At Op4G, we strongly believe that a well-written survey with clearly outlined objectives leads to error-free and efficient programming. Most importantly, it provides our market research clients value by ensuring their research is fundamentally sound and captures all information necessary to achieve their objectives.
For more help or information on our growing survey consultation services, don’t hesitate to contact your account contact or marketing@op4g.com. Also, stay tuned for future blogs on other elements of effective surveys including data quality, survey length, format, device compatibility, and more.
