Writing questionnaires for a survey panel can be a stressful experience. After all, you’re looking to maximize the value of your data -- and the way that you phrase questions will make or break your data set. In this article, we’ll go over some common question-writing mistakes researchers make when creating their survey questionnaires so that you can feel empowered to structure thoughtful questions that yield top-quality data.
To begin, remember that you’ll be communicating with a large pool of human beings. Our survey audiences are vast and each person brings their own lived experience into your build. To best account for the differences that make us unique, write each question in Plain English. Making your survey questions and response options as easy to understand as possible makes your overall project accessible for your respondents.
Continuing in this vein, being thoughtful in how you write and structure your questions will improve the survey-taking experience. Respondents provide the best quality data when they have positive survey experiences.
Keep reading to uncover the three most common question-writing mistakes that we see and how to correct them in order to provide respondents with the best survey experience possible.
1. To be vague is to be uncertain
Each question should be direct, brief, and easy to understand. As such, provide examples and/or definitions to maximize your respondents’ ability to ensure that anyone in your target audience can answer your questions without getting frustrated, confused, or lost.
Let’s take a look at an example of an ill-defined, vague question vs. a clear ask:
Vague Question
- Do you eat pizza regularly?
Clear Question
- How often do you eat pizza?
In the example above, both questions ask about the frequency of eating pizza -- but ‘regularly’ is a vague description. What I consider to be ‘regular consumption’ and each respondent considers to be ‘regular consumption’ may be different. In other words, my data for the vague question will be less actionable because the phrasing is vague enough to create potential misinterpretation or doubt.
Additionally, because we haven’t defined what ‘regularly’ means, respondents may get confused and frustrated by the Vague Question. They want to provide helpful, honest feedback and the phrasing of this question detracts from their ability to do so.
In the Clear Question, we remove doubt and confusion from our data set by allowing respondents to report their pizza consumption plainly. They don’t have to struggle to define ‘regularly’ or worry that their definition doesn’t meet our definition. From the researcher’s perspective, the data collected from the Clear Question will be more actionable and directly reflect respondents’ pizza consumption habits.
Respondents who can clearly understand your survey questions will provide the highest quality of feedback. By eliminating any phrasing that detracts from our overall goal for this question, we can feel more confident in the conclusions drawn from this data.
2. Lose loaded and leading questions
When writing questions, start broadly and get more specific throughout your survey. If you need to ask more personal or intimate questions of your respondents, ask them later in the survey once the respondent has gotten in the groove of answering your questions.
Consider the neutrality of the words you’re using to write each question and aim for complete objectivity. Avoid leading questions that push respondents to one answer or another -- in other words, be mindful of using emotionally charged words, stereotypes, and images. There should be no prompt, subtle or stated, to push a respondent into answering a question a specific way. Below are examples of a leading question vs. a neutral question:
Leading Question
- How likely would you be to purchase our delicious, award-winning pizza product?
Neutral Question
- How likely would you be to purchase this pizza product?
In the Leading Question above, we frame our product as being ‘delicious, award-winning’ pizza. By using emotionally-charged descriptions of our product, we’re leading respondents to answer positively — which leads to poor quality data.
By removing emotionally-charged language and creating the Neutral Question, we enable respondents to provide their unbiased feedback, which leads to actionable data.
A similar trap that leads to poor quality data is loaded questions. Loaded questions ask respondents a question based on an unverified assumption. In environments where each question requires an answer to move forward, loaded questions yield disingenuous and often non-replicable data. Below is an example of a loaded question.
Loaded Question
- At which restaurants do you like eating pizza?
This question is a loaded question because it assumes respondents like eating pizza. Your respondents may not like to eat pizza; even with popular ideas/products, it’s bad survey design to assume how your respondents feel.
Luckily, loaded questions can be easily remedied through preliminary questions that eliminate the need to assume your respondent’s feelings. By removing any assumptions from your questions, you allow respondents to answer honestly.
Below is a revised question set that eliminates our assumption about respondents’ attitude toward pizza, then uses skip/display logic to route qualified respondents to answer our original question:
Preliminary Question
- Do you like to eat pizza?
Respondents who answer that they don’t like to eat pizza skip this question or “leap-frog” over it and proceed onward in the survey. Respondents who answer that they do like to eat pizza have qualified themselves answer the follow up question below.
Follow-up Question
- Where do you like to eat pizza?
As you continue building your survey, ensure that you have adequate response options so that all respondents can answer each question in a manner that reflects their true thoughts, opinions, and values. An “Other” key in option is a good safeguard on multiple choice questions where you feel you may be boxing a respondent in with limited options.
***cta
3. Ask one question at a time
Make your questions direct and don’t ask more than one question at a time. If you ask two questions with one response set, you’re writing ‘double-barreled questions.’ Double-barreled questions are flawed because respondents will inevitably place weight on one portion of the question over the other, which ruins your survey data. See an example of a double-barreled question below:
Double-Barreled Question
- How satisfied or dissatisfied are you with your local pizza chain’s delivery time and topping options?
Some respondents will see the question above and place more weight on their local pizza chain’s delivery time. Others will place more weight on the toppings options. In short, if one question has two subjects, it’s impossible to establish what the respondent is prioritizing and reacting to in their response choice.
Instead, ensure that each question centers on one specific objective, opinion, or piece of feedback. Double-barreled questions can be easily fixed by dividing them into multiple questions:
Revised Question #1
- How satisfied or dissatisfied are you with your local pizza chain’s delivery time?
Revised Question #2
- How satisfied or dissatisfied are you with your local pizza chain’s topping options?
Wrapping Up
In summary, clear and direct questions that cover one subject at a time will create the best survey environment for your respondents. Remember, happy respondents lead to good quality data -- so keep them happy by avoiding the common survey mistakes that we listed in today’s article.
Use accessible language, provide examples, and keep questions brief to ensure that they’re not vague. Watch your language and avoid emotionally-charged questions to prevent bias from leading or loaded questions. Finally, only ask one question at a time for the most accurate data.
Now that you have a better idea of what not to do when writing your questionnaire, feel free to check out our brief overview of quantitative question types and formats to inspire your next survey-writing session.
