Validating a Survey

A survey should be thought of as a two-way communication. If the person taking the survey misunderstands what you are asking, or if you misunderstand what their rating means, the communication breaks down and your conclusions will be based on misunderstandings and therefore will likely be invalid.

Perhaps the most critical question that needs to be asked in creating a survey is “can I accurately interpret the information once I collect and analyze it?” In other words, do the responses mean what I think they mean? There are several ways to assess the likely validity of the data collected with a survey. The most common approach is called Face Validity. It simply means that the items seem to be asking the right things and seem to be clear, at least from the author’s point of view. Often a survey instrument is passed around to various colleagues to solicit their opinion as to whether the survey is clear and seems to be asking the right questions. This is the most common, and perhaps least useful, approach to testing the validity of an instrument. Often colleagues’ suggestions are valuable, and this process should be used, but only as a starting point.

Verbal Protocols

One of the quickest and easiest ways to assess whether people taking your survey understand what you are asking is to recruit several people from the population you are interested in to take the survey out loud. To conduct a verbal protocol for this purpose, simply have a respondent talk out loud, making their thinking explicit as they complete the survey. They should be encouraged to think out loud, but you should not influence their thinking by explaining the survey, or answering questions. Simply ask them to tell you what they are thinking as they read the questions and respond. A remarkable amount of good information can be found this way. The goal is to have them tell you what they think the questions mean, and to briefly explain why they are responding to the question the way they did. Use the data from several verbal protocols to improve the clarity of the survey items.

Pilot Testing

Often times a pilot test with approximately 30 respondents (a number that allows valid correlation analysis) can provide very valuable information about both the validity of the instrument and the usefulness of the items. Ask respondents to complete the instrument and then provide written commentary about items that they felt were vague, or had trouble answering accurately. Use correlation analysis to test whether items that are theoretically related show strong correlations. If they do not, the items are probably measuring something other that what you intended. Look at the distribution of scores for each item. If all of the scores pile up at one end of the scale or the other, then the question is not sensitive to differences, and really does not provide any valuable basis for analysis. Try to rewrite the question to make it more sensitive to differences in the population. Where scores are not related in expected ways, or where there are confusing results, either rewrite items or add items to clarify the results.

Random Qualitative Validation

The best way to validate an instrument would be to ask each respondent to explain why they rated each question the way they did. Of course, that would make completing the survey a big, time consuming pain in the neck and likely result in an unacceptably low response rate. One solution is to collect qualitative explanations from a random sample or respondents for each of the items on the survey. In this way, each respondent only has to explain a few ratings, but collectively you can gather information about all of the items. As an example, Random Qualitative Validation (RQV) on a 50 item survey could be structured so that there are ten versions of the survey. Each version asks the respondent to “briefly describe why you rated this item the way you did” for five items.

Example:

The five items are different for each version, so that collectively they cover all 50 items on the survey. In this way each respondent only has to explain five ratings and it does not become a big or daunting task. The benefits to this approach are substantial and greatly increase the accuracy of the analysis and improves the ability to identify survey questions that are confusing or not understood the way the author intended. The ten versions of the survey are randomized and distributed so that there is no bias in who gets which version.