Thanks to computer software and the web it is now easier than ever to create a survey. The technical side is so easy that people forget that there is more to it than just adding some questions to a form.
I was fortunate to attend a lecture by Marguerrite Cox, a biostatistician from Duke Clinical Research Institute in the USA, and I thought it might be useful to share her 10 Commandments.
(I have changed the examples to be more relevant to a tech audience)
Before you even start with the questions you need to:
-
Decide on the survey’s goals,
-
Identify the target population
-
Determine the sample size
-
Select a survey mode eg internet, phone, in-person et
Ten Commandments
1. Thou Shalt Not Be Vague
- Know exactly what you want to ask and be specific and add a reference frame.
- "How often do you install x per year?" instead of "how often you install x?"
2. Thou Shalt Not Make Unto Thee Complicated Questions
- Be clear and simple
- Avoid jargon
- Avoid complex sentences
3. Thou Shalt Not Ask Double-barrelled Questions
- Example: "How satisfied are you with the software's support and documentation?"
- Break these questions down into multiple questions:
- "How satisfied are you with your software's support?”
- "How satisfied are you with your software's documentation?”
4. Thou Shalt Avoid Leading Questions
- Example: "Do you believe the documentation is good enough?"
5. Thou Shalt Ask as Few Questions as Possible
- Trade-off between getting all the information you need and reducing the amount of non-responses
6. Honour Thy Order of Thy Questions
- Questions should flow logically
- Questions at the beginning of the survey may influence answers to later questions
- Easy, general questions should be at the beginning
- Sensitive questions should be placed at the end of the survey
- If the questions are similar, random ordering can be helpful
7. Thou Shalt Anticipate All Possible Answer Choices
- Example: "How would you describe your programming skills (check all that apply)?":
- novice, new to this language, still learning, moderate, experienced, practically invented the language, Other (specify), Prefer not to answer.
8. If Thou Uses an Ordinal Scale, Thou Shalt Use a Balanced Scale with 5-7 Choices and a Clear Middle Category
- Ordinal data: categorical data where there is a clear and consistent ordering
- Examples:
- Strongly agree, agree, neither agree nor disagree, disagree, strongly disagree
- Very good, good, fair, poor, very poor
- Never, rarely, sometimes, most of the time, always, No chance
- very little chance, unsure, some chance, very good chance
- Just as the order of the questions is important, the order of the answer choices is important
- Primacy Effect: respondents pick the first choice (paper and internet surveys)
- Recency Effect: respondents pick the choice most recently heard (in-person and phone surveys)
9. Thou Shalt Not Have Overlapping Categories
- How many web sites do you manage?": 1-10, 10-20, 20-50, 50-75, Over 75
- What if they manage 20? Some of your respondents may check 10-20, while some may check 20-50. Some may be confused and not respond.
10. Thou Shalt Do a Trial Run of Thy Survey
- This will help you identify issues with questions and/or responses
A bonus - not so much a commandment but a universal truth!
