Five Survey Question Errors that ruin your data
Data is invaluable for its ability to offer insight into the past and the future. When collecting data, the way you ask survey questions can directly affect results. As a surveyor, you do not want to ruin data quality by asking confusing questions.
Even something as simple as question wording has the power to alter the respondent’s perspective on an issue. Use of improper langauge may also unintentionally force them to answer the question inaccurately or give a false answer by mistake.
Imagine your survey is finding out the most popular pizza among employees in a company.
Which type of pizza do you like best ?
The problem with this question is that it automatically assumes the answer is among the four kinds of pizzas mentioned in the question.
So instead of finding out the most popular pizza, the study measures the popularity among these four types. There might be a New York style pizza a lot of people like, but they can't say so simply because the question does not even mention it.
Questions are the essential part of your survey, and framing them correctly can make or break the survey in terms of its language quality.
Survey tip:
You might want to start by surveying small groups of people with open-ended questions just to understand the right way to ask them what you need to know. This way, you'll know the pattern framework in which respondents are answering your questions and are less likely to overlook options that are important to your respondents.
Five common survey errors
Here are some common errors you should avoid if you want to become a pro surveyor:
Firstly, don’t lead the respondent.
At all costs, avoid phrasing the question in a way that eggs the respondent towards one part of the question. Questions like this are typically called 'biased' or leading questions. A respondent should be completely free to choose their answer.
Questions that are biased are responsible for high survey drop-out rates, because respondents get tired of trying to figure out answers for questions that are either simply confusing to read or blatantly pushing towards a particular choice.
This is why the wording of questions in a survey is especially important—it can influence the people taking the survey, which in turn can affect the overall credibility of the data collected.
How short is the leaning tower of Pisa?
What is your opinion on the height of the leaning Tower of Pisa?
Leading questions also have another problem—they end up becoming unnecessary additions to the question.
Should careful car drivers always insist on using rear seat belts?
Instead of going with a term like “careful car drivers,” which might tend to deviate from the purpose of the survey, it is better to go with a question that is framed like this
Do you think rear seat belts should be made necessary?
Always remember to use neutral wording and easy words to avoid pushing your readers into a certain opinion.
Secondly, don’t load the questions.
The term loaded questions implies asking too many things at once. It's done so that the respondent is forced to answer them in a way that does not necessarily reflect their true opinion.
Here's an example:
Where do you enjoy drinking beer?
The problem with answering a question like this is that it assumes the person enjoys drinking beer. However, many people dislike beer, or won't drink alcohol, and therefore can't answer the question truthfully.
Usually loaded questions are best avoided by pre-testing a survey. You can try a pilot survey (which is basically a test run of your survey) to make sure every question has a response that will prompt the person to answer honestly.
To avoid this, you can simply ask the person if they drink beer or not first, and then insert a logic in the question that says those who drink beer receive a different set of questions relevant to drinking beer. This is a much better option as opposed to wasting time asking the same set of questions to two different audiences.
Do you drink beer?
Where do you prefer drinking beer?
The skip logic feature can help you narrow the questions of a survey, depending on the respondents’ answers.
Thirdly, avoid double-barrelled questions.
Another common mistake in surveying is catpaulting two questions at once towards the respondent. This is called a double-barreled question. Survey questions need to be written so that you should be testing only one parameter per question.
The problem with double-barreled questions is that they are highly unrealistic and the two-birds-with-one-stone approach does not go down well in surveys.
Surveyors sometimes use such questions to reduce the number of questions in their survey and make it less-daunting, but they only end up with flawed questions and inaccurate responses.
If a single question has more than two elements involved, it’s impossible to tell how the respondent is supposed to answer for the different elements involved.
Here’s an example to understand it better:
How satisfied are you with the job benefits and pay at your workplace?
This question throws two things directly at the respondent: The workplace benefits and then again the pay level satisfaction. What will happen in such a case is that some of the respondents will be giving answers regarding pay, while others will give more focus to the workplace benefits. This will result in inaccurate answers from people, and completely ruin your data.
Instead, a better way to frame this question would be to split it into two:
How satisfied or dissatisfied are you with the current pay for your job?
How satisfied or dissatisfied are you with the current benefits of your job?
Fourthly, don’t use absolutes in the questions
Absolutes in questions force respondents into a corner where they can’t give useful feed back. These questions usually have options like Yes/No and wordings such as “always”, “all”, “every” and “ever.”
A good example of this question can be to ask respondents whether or not they eat eggs.
Do you always eat eggs?
The example above would force most respondents to answer no directly. The question is inflexible, which makes it too rigid to answer.
The right way to frame a question like this is letting them choose from a lot of options.
How many days a week do you usually eat eggs?
Lastly, speak your respondents’ language
As a surveyor, you have to use clear, concise, and uncomplicated language and try to avoid too much jargon while framing your questions. Make sure you provide definitions or examples if you need to include tricky terms or concepts. This way, you can be certain that anybody can answer your questions clearly because they understand the questions clearly.
This will also increase the chances of people completing your entire survey.
Do you own a gaming console?
Do you own a gaming console?
(E.g. Xbox, Playstation, Nintendo Wii, etc.)
Here’s one more example to make things clearer:
What is the state of cleanliness of the room?
Instead of putting too many complicated words in the question, you can simply frame it like:
How clean was the room?
The idea is to make sure that the meaning of the question is easily understandable in one reading. The respondent need not put in extra effort. Some exceptions can be allowed in special cases where the respondents are from a group that regularly uses difficult terms.
For example, if your survey sample comprises a group of specialized people, like chartered accountants, then it makes sense to use business and accounting terms in the survey. In these cases, an exception can be made.
The ideal here would be to conduct pilot surveys. You could also have a peer review your survey. A second pair of eyes can always be helpful.
Nevertheless, if you clearly follow the tips given above, your survey will be clear to read while also collecting effective data.
Your respondents will leave the survey feeling like they have given some honest and accurate feedback, which is great for you, because it means you will get the right data.
Always remember: Survey questions are the backbone of any survey-based data collection, so it’s crucial to have well-thought-out survey questions if you want the data to be reliable.
Related Articles
Identifying the right sampling method for your survey
Surveys would be meaningless and incomplete without respondents (...)
Sangeeta Bose Oct 16, 2019
Five response bias to avoid in your survey
Humans are rather interesting creatures (...)
Sangeeta Bose Oct 16, 2019
How do I get more responses for my online survey
Surveying is not only about asking the right questions (...)
Sangeeta Bose Oct 16, 2019
