Making Quality Real: Delivering on a Promise of the Best Service and Online Survey Sample

Knowledge Networks Profile: Mike Dennis, SVP General Manager, Government & Academic Research

As Senior Vice President, Mike Dennis has led Knowledge Networks' Government and Academic Research team for the last eight years. He and the staff of Government & Academic Research have overseen hundreds of studies, including those that have borne scientific scrutiny of the Knowledge Networks (KN) web panel methodology. Through a recent promotion to SVP General Manager, Mike also now guides KN panel operations. In this role, he brings his understanding of the statistical and data quality requirements of the Federal survey system to managing the evolution of KnowledgePanel® – Knowledge Networks' probability-selected, nationally representative online panel.

Redoubled panel representation efforts

When first joining Knowledge Networks in early 2000, Mike recognized the value of methodological research for helping customers evaluate the KN panel approach. Mike and his team conducted a number of methodological studies, often in collaboration with clients, on topics such as data collection mode effects; panel conditioning effects; non-response bias; effects of questionnaire design on survey results; and benchmarking.

More recently, Mike and his KN colleagues have been addressing response and attrition propensities of younger adults and non-whites. According to Mike, "KN currently has a significantly redoubled effort to recruit and retain young adults and minorities, in particular members of the growing U.S. Latino population – both unassimilated and Spanish-language dominant households. Our recent efforts in this area reflect both the fact that these groups are both more difficult and costly to recruit and retain on the panel, as well as clients' demand for conducting public policy and other research studies with these groups."

Because these groups are heavily sought-after for survey research, there is a risk that members of these groups could be "over-surveyed" on the KN Panel, resulting in panel fatigue and unacceptable attrition from the panel. As a preventative measure, KN developed a system for load balancing the survey invitation in order to prevent over-surveying of high-demand groups on the panels, and to produce representative panel samples for our customers. Since 2000, Knowledge Networks has provided quality assurance in this area through this proprietary capability. Recently patented, our approach to selecting online survey sample provides a safeguard by scientifically correcting for biases created as a result of previous sample draws from KnowledgePanel during the course of normal operations. Without such a methodology, research panels are liable to place too much respondent burden on high-demand segments, as well as produce unrepresentative panel samples.

Research on Research to benefit our clients and the industry

As head of Government & Academic Research, Mike has led the ongoing transparent investigation of KnowledgePanel, running many parallel studies – where the same survey is conducted via an opt-in panel and KnowledgePanel simultaneously. He and his KN colleagues have noticed a number of substantive differences in results, as well as in the underlying demographic representativeness of the two sample types.

"At face value, the idea of opt-in web panel surveys sounds promising: respondents volunteer for research, and the costs are low. But sample from opt-in panels based on self-selection raises the potential for bias caused by clustering – such as clustering of respondents who are more aware and knowledgeable of politics, society, or new products. In the opt-in panels, there is no way to control for or meaningfully measure the bias caused by respondents' self-selecting themselves into research." An illustration is below.

U.S. Population by Sampling Groups

Opt-in or river panels are based on volunteers who choose the surveys in which they participate. KN research has proven that opt-in panel respondents have characteristics that skew toward affluence and higher education; they also have more knowledge of public policy and typically report higher levels of purchase intention for new products. In addition, opt-in panels generally have a high percentage of female respondents and an under-representation of non-white populations. In contrast, KnowledgePanel has the lowest absolute error rate in the industry.¹

Service and expertise

For principle investigators attempting to measure real phenomena in society, a solid foundation in probability-based survey sampling is critical. Mike and his team help investigators think through initial sampling plans, questionnaire creation, weighting and other aspects of survey design that contribute to their ability to publish the final results and/or lead to Federal policy making. Mike's client experience serves him well in his new broader role in panel operations. Last year, for example, he worked with Boston University on a study that examined the epidemiological issues surrounding the consumption of alcohol – particularly the age of onset of alcohol drinking, including the risk factors for binge drinking. Before turning to Knowledge Networks, the researchers had begun collecting data by random-digital telephone survey, which they stopped due to the extremely high costs. The study required a specific population – relatively younger adults and within that group, individuals with some history of drinking alcohol (the study included a relatively large screening aspect). Boston University investigators needed a high response rate to assure that their results could be published in a peer-reviewed journal in the epidemiological field.

Mike and his team helped Boston University design a sample plan that ultimately met their objectives. As part of the experiment, the investigators conducted the same survey with individuals who had opted not to join KnowledgePanel. This approach enabled the investigators to determine whether KN panelists were representative of the entire population, when it came to the epidemiology of the age of onset of alcohol consumption. The experiment compared individuals who did not join KnowledgePanel with those who did; the results did not reveal significant differences for the age of onset drinking or the associated risk factors. This validation helped assure the Boston University scholars that results from this study would be accepted by a peer reviewed medical journal. Alcoholism: Clinical and Experimental Research published an article on this experiment in their February 2008 issue.

Longitudinal research

An online panel company's capacity to conduct longitudinal research hinges on the tenure and representativeness of its sample. Knowledge Networks conducts a large number of longitudinal studies in which the same respondent is invited to complete successive waves of surveys. KnowledgePanel is an ideal platform for longitudinal studies due to the high panel retention rate across the waves of data collection. In addition, one can validate survey results in terms of their conformance to statistical benchmarks provided by U.S. government benchmarks (e.g., the National Health Interview Survey or Current Population Survey).

In a longitudinal study that has appeared in numerous academic journals – The University of California Irvine Stress and Trauma Study funded by the National Science Foundation – Mike and his team surveyed respondents over a three and a half year period, with seven waves of data collection. Between data collection waves, KN maintained contact with research subjects to remind them of their role in the longitudinal study. During the last wave (Wave 7) of data collection, conducted between September and October 2004 – three years after 9/11 – over half of the original respondents from Wave 1 attrited from the KN panel. Despite this sample issue, the Wave 7 data collection still achieved a 79% survey completion rate as a result of extensive locating efforts and refusal conversion of respondents that had been dropped from KnowledgePanel.

Summary

The strength of the KN approach lies in the sample representativeness of KnowledgePanel®. Through methodological research and survey innovations, KN will continue to improve upon the probability-based panel model. As Mike indicates: "We make continual strides to maintain our panel's high-quality composition. We have approximately 400 papers and articles – and even a few books – that have been published, with many more in process, which feature data from KnowledgePanel."

Dr. J. Michael Dennis leads Government & Academic Research and KnowledgePanelSM Operations for Knowledge Networks.

He has managed numerous surveys for academic, government and foundation-based customers. A frequent presenter at the annual meeting of the American Association for Public Opinion Research, his current areas of methodological inquiry are non-response bias, panel conditioning, and data collection mode effects.

 

¹Study Sponsored by the Stanford Institute for the Quantitative Study of Society, Comparing the Results of Probability and Non-probability Sample Surveys, presented at 2005 AAPOR Conference