[ Summer 2010 ]
New Report from AAPOR on Online Surveys
By J. Michael Dennis, Ph.D.
The American Association for Public Opinion Research released in March a significant report of its Online Task Force. Comprised of researchers from diverse academic, government, and commercial survey professionals from both sides of the probability versus nonprobability dialogue, the Task Force attempted to bring together in one document a framework for describing the wide variety of online survey methodologies, a review of the published and publicly available methodological literature that is emerging around online surveys, and recommendations about appropriate uses of probability-based versus nonprobability-based web panels.
The majority of the content of the report is devoted to online methodologies based on nonprobability sampling because, as the Task Force explains in the report, less is known about this type of online surveys even though they are the most common form of online research and because "they represent a substantial departure from traditional methods."
A significant strength of the report is that there was an attempt to bring forward methodological research to inform the recommendations of the task force. The evaluation research by Yeager et al, among others, is used to inform an evaluation of the online panels. Below are excerpts from the conclusions and recommendations (pp. 5-6). The reader is encouraged to read the entire report.
Excerpts from the AAPOR Online Task Report
- Researchers should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values. There currently is no generally accepted theoretical basis from which to claim that survey results using samples from nonprobability online panels are projectable to the general population. Thus, claims of "representativeness" should be avoided when using these sample sources.
- The majority of studies comparing results from surveys using nonprobability online panels with those using probability-based methods (most often RDD telephone) often report significantly different results on a wide array of behaviors and attitudes. The degree to which those differences might be due to mode effects versus the nonprobability character of online panels is a matter of ongoing debate. The few studies that have disentangled mode of administration from sample source indicate that nonprobability samples are generally less accurate than probability samples.
- There are times when a nonprobability online panel is an appropriate choice. Not all research is intended to produce precise estimates of population values and so there may be survey purposes and topics where the generally lower cost and unique properties of Web data collection is an acceptable alternative to traditional probability-based methods.
- Research aimed at evaluating and testing techniques used in other disciplines to make population inferences from nonprobability samples is interesting and valuable. It should continue.
- Users of online panels should understand that there are significant differences in the composition and practices of individual panels that can affect survey results. Researchers should choose the panels they use carefully.
- Panel companies can inform the public debate considerably by sharing more about their methods and data describing outcomes at the recruitment, enrollment, and survey-specific stages.
- Full and complete disclosure of how results were obtained is essential. It is the only means by which the quality of research can be judged and results replicated.
|
Opt-In Online Panel Task Force Members:
Reg Baker, Market Strategies International and Task Force Chair
Stephen Blumberg, U.S. Centers for Disease Control and Prevention
J. Michael Brick, Westat
Mick P. Couper, University of Michigan
Melanie Courtright, DMS Insights
Mike Dennis, Knowledge Networks
Don Dillman, Washington State University
Martin R. Frankel, Baruch College, CUNY
Philip Garland, Survey Sampling International
Robert M. Groves, University of Michigan
Courtney Kennedy, University of Michigan
Jon Krosnick, Stanford University
Sunghee Lee, UCLA
Paul J. Lavrakas, Independent Consultant
Michael Link, The Nielsen Company
Linda Piekarski, Survey Sampling International
Kumar Rao, Gallup
Douglas Rivers, Stanford University
Randall K. Thomas, ICF International
Dan Zahs, Market Strategies International |
Dr. J. Michael Dennis leads Government & Academic Research and KnowledgePanel® Operations for Knowledge Networks.
He is a methodological thought leader in the design of KnowledgePanel-based research. Dr. Dennis has managed numerous surveys for academic, government and foundation-based customers and is a frequent presenter at the annual meeting of the American Association for Public Opinion Research. His current areas of methodological inquiry are non-response bias, panel conditioning, and data collection mode effects.
