Web Questionnaires: Tested Approaches from Knowledge Networks for the Online World
By Mario Callegaro

Superior data collection is a cornerstone to capturing representative information about the population of interest that informs decisions in business and public policy, and in advancing the social sciences. In all modes of data capture, the questionnaire is important, yet it can be argued that in online-based research, a self-administered Web survey relies even more heavily on the quality of the questionnaire. Although some findings can be transferred from the literature of paper and pencil questionnaires, they need to be retested for Internet research. At Knowledge Networks, we have always addressed this issue by doing research ourselves and keeping up-to-date about new research and findings from the published literature and conference presentations.

After a carefully designed questionnaire is developed, there are still many decisions to make in order to have it programmed, then accurately administered to respondents. In this issue of Accuracy's Impact on Research and those that follow, we discuss several aspects of Web questionnaire design that can impact response distributions:

• Visual presentation of the question on the screen – important, as there is no interviewer present to help the respondent clarify the meaning of questions; probe to avoid non-response to individual questions; or objectively spell out response categories
• How the respondent can select the appropriate response option (e.g. radio button vs. pull-down menus)
• Use of visuals

Data collection with online questionnaires is still a relatively new methodology, and more research is still to be done with Knowledge Networks in the forefront.

Visual presentation of the question on the screen

The key concept in evaluating and designing a Web survey is that the respondent is inferring the meaning of the question not only from the text itself, but also from its visual aspects. Because no interviewer is present, everything is left to the respondent. One example is the size of text boxes for open-ended answers.

Open-ended questions appear to be a promising question format for Web surveys. Some researchers believe that the relative ease of typing longer answers compared to handwriting will provide richer responses [1]. A key factor in open-ended answers is the size of the text box. In an experiment conducted by Knowledge Networks, respondents were randomly assigned to either small (n=172) or large (n=164) text boxes for three open-ended questions [2]. The text of the questions appears below:

Q0: Your WebTV can be used to surf the Web, chat with other viewers, program your VCR, email your friends and family, and more. How do you use your WebTV?
Q4: In your opinion, what are the principal benefits, if any, of ordering goods on the Internet?
Q10: In your opinion, what role, if any, should be U.S. government play in overseeing Internet security and privacy?

As we can see from Figure 1, large text boxes elicited longer open-ended answers; all results of the comparisons are statistically significant.

Figure 1. Average number of characters typed in small and large text boxes for three questions.

This experiment provides support for the hypothesis that respondents infer a meaning about the question response from its visual aspects: a large text box is communicating that a long answer is desired. This meaning is inferred without the need for any special instruction. Based on this experiment, it is now standard practice for Knowledge Networks surveys to give clients a choice of three sizes of text boxes, small (e.g. one line), medium, and large.

How the respondent can select the appropriate response option

Another feature to keep in mind when designing Web questionnaires is to think about how the respondent will choose the appropriate response alternative. Unlike paper and pencil or telephone questionnaires where the range of possibilities is limited, there are many features available for entering data in a Web survey.

One example is the usage of pull-down or drop-down menus, a feature common in Web pages – such as payment pages, where one enters the month and year of a credit card's expiration date. Although frequently used in other applications, pull-down menus can be problematic for a Web survey. One problem is what to show as the first option. There is a general agreement among researchers [3-5] that if pull-down menus are used, the first option to be shown on the bar should be something like "select one" or "click here" and not the first response category. Experiments done by displaying the first response category showed a disproportional number of answers for that category.

A second problem has to do with the fact that if the respondent is using a mouse with a scroll wheel, it is more likely that the original answer is changed. This is because once inside a drop down menu, moving the wheel changes the answers. In a randomized experiment, Healey [6] found that respondents who used a scroll wheel mouse were going back to the pull-down menus to change their answers more often than respondents who used a mouse with no scroll wheel.

A third problem is having long lists of items, which forces the respondent to do a lot of scrolling in order to find the right category to select.

Given the above issues, we advise clients to use pull-down menus sparingly, and when the list of options is not too long. The first option shown should be "select one."

Use of visuals: pictures

Images that are supplemental to the question text help the respondent to contextualize the question and to clarify the meaning. From our point of view, the problem is the selection of the image by the person designing the questionnaire. An interesting experiment done at Knowledge Networks sheds some light on this.

Respondents were randomized into different groups and completed a travel, leisure and shopping questionnaire [7]. For each topic, four versions of the questions were provided showing:

1) no picture;
2) a picture of low-frequency behavior;
3) a picture of high-frequency behavior;
4) pictures of both low- and high-frequency behavior.

For example, one question inquired about the frequency of eating out. In the high-frequency condition, a picture of somebody eating fast food was shown. In the low-frequency condition, a picture of a sit down restaurant with a waiter was shown. Another question asked how many sporting events were attended in the past year. In the low-frequency behavior condition, the text of the question was accompanied by a picture showing a huge baseball stadium full of people; in the high-frequency condition, a picture of two young children in a small baseball park was shown.

As expected, the average number of visits to restaurants or trips to sporting events was significantly higher when the high-frequency picture was shown in comparison to the low-frequency behavior picture. Moreover, the condition in which a picture was shown, no matter its content, elicited a significantly higher frequency of the behavior in comparison to the condition in which no picture was shown.

From this experiment we can infer that, in the case of a picture supplementing text, the pictured content can affect the response. This can skew responses in directions that are not always easy to predict. If pictures are to be used, they should be selected carefully and ideally pretested. Pictures can be a great memory aid, but again the content of the picture should not be left as a last minute decision, but treated as integral to the question wording.

The preceding experiments and many others illustrate Knowledge Networks' ongoing commitment to quality. We continue to test our own research methods, procedures and assumptions as frequently as possible through scientific evaluation. Since our founding in 1998, this endeavor has been a basic tenet in our corporate philosophy; it extends from online survey design, to statistical sampling and other areas. In our next issue, we will feature an experiment that measures respondents' compliance with a task requested via questionnaire.

We welcome feedback from you, our readers, with regard to the experiments covered in this article, as well as Web questionnaire design issues about which you have a question, or you feel warrant further exploration.

See Footnotes

Mario Callegaro is Knowledge Networks' Survey Research Scientist.

He has published nationally and internationally in the areas of telephone and cell phone surveys; polling and exit polls; longitudinal surveys; event history calendar; interviewer effect; web surveys and survey quality. He holds a B.A. in Sociology from the University of Trento, Italy, and a M.S. and a Ph.D. in Survey Research and Methodology from the University of Nebraska, Lincoln.