Ranking questions

Often we want to know respondents relative preferences among a list of options. There may be five different menus we could purchase for a conference lunch and we want to know what people would prefer. A common way to ask this question is simply to list all of the options and ask the respondent to rank them from 1-5 in order of preference. We then examine the rankings and select the most popular choices.

There are three problems with this approach, two practical and the other theoretical. First, ranking more than a few items creates problems with validity. If you have someone rank 10 items from least favorite to most desirable, what is the difference between a 6 and seven? Respondents have difficulty organizing many items into a coherent system of favorites. The general advice is to keep the list short; three items are ideal, but never more than seven.

The second problem is that no matter how strongly you emphasize that ‘1’ should be assigned to the most preferred option, some people want to use the highest number ‘10’ and vice versa. So even though you may indeed get everyone to rank 10 items, you cannot be sure that everyone has used the ranking scheme in the same way.

The third problem is theoretical, it is actually impossible from a pure measurement standpoint to pool a set of rankings and produce a meaningful conclusion for a population. You can certainly find out which item was ranked #1 most often, but it is not possible to conclude that that is the favorite given the cumulative rankings for everything else.

Consider the following table of results in a survey of 50 graduate students:

If one just looks at the relative frequency of choices, “Creating study areas for students” is the most popular #1 choice. However, if you look at combined first and second rankings, “Hiring additional faculty” leads 30 to 18 in terms of the number of respondents who choose that option as their first or second choice. This becomes increasingly difficult to interpret, and some have argued impossible to establish mathematically what the overall ranking of the options is on the basis of the collective results. This topic goes beyond the scope of this site, but it is important to realize that it is not a straightforward analysis of preference when rankings are used.

If ranking is important, the best approach is to use sets of three items asking the respondent to choose their most and least favorite. Another way to get an indication of what respondents value most or least is to use a standard likert scale asking the respondents to rate each item on a scale of 1-7 in terms of importance, and then, after they have rated each item, asking them to go back and circle the most important one in the list.

Now, go back and circle the option that you feel is the most important use of budget money. When this technique is used, one can report how often each option is selected as the most important, and still not require ranking. The use of the Likert scale allows one to measure importance, without forcing hierarchical choices leading to more valid conclusions.