Reasons for piloting questionnaires

Daniel Gile

March 30, 2006

In a previous text (Do respondents to surveys tell the truth?) explanations were given as to why bias from respondents in surveys might distort the image of reality generated by collected data. Such bias is only one obstacle in the way of researchers who carry out survey investigations. Another one (inter alia) is potential misunderstandings of questions. Very often, single terms or even full sentences turn out to be understood by respondents in a way different from the idea they were supposed to convey. This may be due to clumsy wording, but also to interpersonal variability in language use or to different mindsets and expectations in survey drafters and respondents. While it is obviously important to be aware of such risks and to do one’s utmost to draft survey questionnaires very clearly from the start, how clear the wording really is can only be ascertained through experience, that is, with actual responses. A questionnaire which has been validated in the past may be used without further experimenting, but any new questionnaire should be piloted, that is, submitted to a small number of respondents, improved on the basis of the responses collected, then sent out to a further small group of respondents and improved again etc., until no further problems which require attention are found.

Another problem with questionnaires is low response rate. This may be due to lack of clarity in the wording of questions, but also to excessive complexity and/or length of the questions and/or questionnaires themselves, which require more time and attention from respondents than they are willing to devote to the exercise. Again, piloting may help investigators find the best wording and the best balance between the amount of information requested from individual respondents and the proportion of respondents who actually complete the questionnaires.

Low response rate may also be due to under-optimized delivery of questionnaires to potential respondents: sub-optimal sampling, wrong timing, wrong package (hard copy or electronic), wrong persons, etc. Again, piloting may avoid waste of resources.

Leaving aside weaknesses in questionnaire design, piloting can also help improve questionnaires which were germane to the needs initially: sometimes, respondents make comments and suggestions that inspire new ideas, new points to explore, a different way of looking at things which the investigator did not think of initially and which can be integrated into the questionnaire. Thus, much may be gained by doing a survey in several steps, each serving as a pilot phase for the next. In such a way, more useful information may be gained in a four stage survey with 50 respondents in each than in one survey with 200 respondents.

These are only a few of the reasons that make piloting an important part of questionnaire surveys. Conducting one without a piloting phase is hazardous to one’s health as a researcher.