Surveys and questionnaires are excellent tools for gathering data, opinions, needs, and observations from large groups of respondents in a relatively short time. They come in many forms, with individual questions being either open-ended, where respondents can provide any type of answer they want, or closed-ended, where respondents must choose from a fixed group of possible responses. The former may require significantly more effort and interpretation to process, while the latter are more amenable to automation.
I’ve incorporated surveys in my own work here and studied specific types of surveys while preparing for my Lean Six Sigma Black Belt exam. I discuss a particularly memorable decision that used survey techniques, at least in part, here. The treatment of this subject in the BABOK is outstanding, so I’m mostly going to paraphrase its descriptions.
The process for conducting surveys is outlined below.
- Prepare the survey or questionnaire and plan for how the data is going to be collected and processed.
- Define the objective: Determine what information you hope to gather to support the decision(s) you are typing to make.
- Define the target survey group: Identify the relevant audience to be queried. This may involve anything from a broad customer base to a narrow job function.
- Choose the correct format: Determine the type of survey questions to ask to get the types of responses you need. You should consider the level of engagement of the audience you are polling. Highly engaged customers and employees may be willing to spend a lot of time and effort providing detailed and voluminous responses, while you may design a lighter and more streamlined process in hopes of gaining sufficient responses from less engaged participants.
- Select a sample group if appropriate: You may want to or have to choose a subset of a larger group to survey. This may require statistical structuring across many demographics to keep the results from being skewed.
- Select distribution and collection methods: Determine how the surveys will be sent and how the answers will be returned and processed.
- Set the target response rate and response end time: Determine the minimum number of responses you need for the effort to be considered valid, and the time window within which responses will be accepted.
- Determine if additional activities are needed to support the effort: Additional work may need to be done to design the questions or interpret the responses. This work may involve interviews and other techniques.
- Write the questions: These will be based on the information you need and the decision processes you hope to support. The size, format, and complexity of the survey will be determined by the audience you plan to query and the information you hope to obtain.
- Test the survey or questionnaire: This involves testing of the mechanics of distribution of questions and collection and processing of responses (verification), and also the methods of assessing the responses for correctness and applicability (validation).
- Distribute the survey or questionnaire while considering:
- the urgency to obtaining the responses,
- the level of security required, and
- the location of the respondents.
- Gather the responses and document the results.
- Collate the responses.
- Summarize the results.
- Review the details and identify any emerging themes.
- Formulate categories for breaking down the data.
- Arrange the results into actionable segments.
When writing about BABOK techniques on this blog I don’t generally go into their strengths and weaknesses. I figure if you understand the techniques well enough you should be able to reason through most working situations or potential questions on certification exams. However, there are some well known issues with this process that aren’t mentioned in the BABOK.
The first issue is selection bias in the respondents, which can take many forms. One example is that people may be more likely to offer responses (or reviews) when they are angry with a product or service. That might yield useful information about complaints, but it may not give an accurate reading of the overall level of satisfaction. Another example is that the nature of the survey may tend to elicit responses from people in excess of their proportion in the overall population. Many requests for responses published in magazines yielded notoriously skewed figures for the prevalence of certain behaviors. In addition, respondents may be inclined to stretch the truth by telling a tall tale or two. The method of polling may skew the results as well. Several studies over recent decades have identified potential issues with telephone canvassing, especially in advance of political elections. If certain demographics are less likely to be at home at certain hours, do not have landlines, or tend to screen calls or hang up on pollsters, the validity of such polls can be severely compromised.
Another issue is that people may simply lie. This can happen when the results aren’t sufficiently confidential or when respondents don’t wish to seem mean, prejudiced, or otherwise unpleasant. It’s also possible when describing affinities for things people have no talent for. Respondents may enjoy the idea of singing popular music in a band, but it’s not going to matter much if they can’t carry a tune in a bucket. (I experienced a couple of these in a career assessment survey in high school. I don’t know if my responses skewed any wider results since the exercise was intended to illuminate our own interests and abilities, but if they were trying to do anything else it couldn’t have been good.)
A major problem with polling, especially in certain subjects, is that the questions may tend to lead subjects toward certain responses. This may not be purposeful, in which case testing, review, and revision should be applied to correct any deficiencies, but we are probably all aware of polls that are structured to drive public opinion rather than accurately reflect it.