Comparison Study: Early Adopter Attitudes and Online Behavior in Probability and Non-Probability Web Panels

[ Spring 2009 ]

Latest Methodological Research Finds Early Adopter Bias in Non-Probability Web Panels

By J. Michael Dennis, SVP, General Manager, Government & Academic Research; Larry Osborn, Sr. Project Director; and Karen Semans, Research Analyst

It can be argued that consumers have a simpler choice in shopping for a new car than researchers have in selecting a Web panel provider to use in support of new product analysis, concept screening, social science research or public policy making. While there are clearly more makes and models of cars than there are Web panels, there is a paucity of information to help the researcher determine which panels are the most accurate and reliable. To change this asymmetry of information, we need to know about the "respondent inside" a panel – a look beyond demographics – to know what the differences are in opt-in non- probability panels compared to probability-based KnowledgePanel®.

The reason is straightforward. Panel members can and do affect the outcome of our results and the certainty of our interpretation and recommendations. Since 2000, through our methodological research, we have been sharing this look inside each panel type for variations in attitudes and online behavior. Our most recent effort shows an early adopter bias in non-probability panels; the bias we measured could affect how you interpret new product or concept testing research, or even public policies and programs. For example, 41% to 44% of non-probability panelists say "I try new products before others," compared to 24% to 26% of those in probability samples – a difference of nearly 100%. (Click here to jump to other results from the new study).

Continuing with the auto industry analogy, that industry over time has had to compete on features, quality, and price, with necessary information being disclosed by manufacturers that allows independent third parties to rate cars on all these dimensions. Consumers can even test drive cars before purchasing them; it's pretty hard to test drive a Web panel free of charge. So please consider our methods research your free test drive!

Evaluating Web panels at first appears deceptively simple, because there are only two types of Web panels. First, there are the Non-Volunteer Access Panels (NVAP), in which potential panel members are chosen by the research company using a statistically valid sampling method. In addition, there is a known published sampling frame for that recruitment to establish panel representativeness. Examples of statistically valid sample frames are list-assisted random digit dialing samples and the USPS Delivery Sequence File of residential addresses. KnowledgePanel® typifies the NVAP model. In contrast, the second type of Web panel, the Volunteer Access Panel (VAP), is made up of respondents who can join at will from many venues, including emails, email spam, Web or print ads, or by word of mouth. VAPs are sometimes called "opt-in" or "non-probability" Web panels.

To date, there hasn't been much research published comparing the accuracy of these two panel types. As a result, it is common for researchers to emphasize differences in the theoretical underpinnings of panels' methodological approaches. On these grounds, the probability panels are often regarded as superior, because they stand on the shoulders of the giants of the scientific method based on random selection. The probability Web panel is an extension of the randomizing method based on area-probability samples and RDD sampling that, respectively, have dominated federally sponsored survey research since at least the 1960s through the 1990s. The advent of the non-probability Web panel is a reaction to the higher costs of basing Web panel recruitment on a statistically valid sample frame, and to the high costs of building Web panels large enough to support small area estimates or subpopulation surveys.

Research questions

Consumer research often attempts to identify individuals who will be early adopters of a new product or service. They may have bought a Betamax video recorder before VHS became the standard, or stood first in line when the iPhone came on the market. If a survey sample consists of too many early adopters, the survey might provide inflated and erroneous measures of willingness to purchase a new product or service, leading to bad business decisions.

So, our research question emerged as follows: Does using a representative online probability panel versus a non-probability panel make a difference in the survey measurement of attitudes and behaviors related to "early adoption?"

Specifically, how do the two types of panels compare in terms of:

Self-images of early adoption of a new product
Number of online interviews completed recently
Use of online social networks
Duration of home Internet use

Sample sources

The study design involves a comparison of four U.S. online panels – two that are based on probability sampling and two that are not.

The two probability-based online panels are:

KnowledgePanel®
The 2007-2009 American National Elections Studies Web Panel

These two panels were selected for our study because they are the only two NVAP Web panels that meet these criteria:

probability-based sampling
inclusion of non-Internet households, enabled for Web surveys
general population coverage of age 18 and over for national representative surveys of at least 1,200 interviews

KnowledgePanel is the only online panel representative of the U.S. adult population that uses probability-based sampling and provides laptops and an Internet Service Provider (ISP) to those who did not have them at the point of recruitment.

Funded by the National Science Foundation, the American National Election Studies (ANES) Web panel was built and maintained by Knowledge Networks under a contract with Stanford University. Knowledge Networks provided MSN TV 2® devices and ISP service to the non-Internet households. For the first time, the traditional pre- and post-assessment of the 2008 election was conducted online. Since its inception in 1977, the ANES survey data have become the gold standard for science research on the elections. ¹

We also selected two non-probability panels for the study. The panels are unnamed in our research and were selected randomly from a list of well-known opt-in panel firms. The goal of the blind random selection is to assure fairness in the selection process; in addition, we sought to have the results, within reason, be projectable to the VAP type of Web panels. The identities of the two opt-in panel firms were not disclosed to the researchers conducting the study, and the results are not reported by the identities of the two opt-in panel companies. Knowledge Networks purchased the samples from the firms providing a simple sample specification. ²

Questionnaire & survey administration

The survey items in our methods research were designed to measure early adopter attitudes and behaviors, including online behavior. All respondents from the four panels were administered the same questions using the same questionnaire format and the same survey system used by Knowledge Networks. The data collection occurred simultaneously for all four online survey samples in September - October 2008, providing a comparison database on which all the analyses for this paper were made.

The actual questions used for this research are in Attachment A.

The sample sizes, within-survey completion rates, and field period dates are reported below. As you can see, the non-volunteer probability based panels had considerably higher completion rates (65.8% and 63.7% for the ANES Web Panel and KnowledgePanel, respectively), than the two opt-in panels' at just under 5%.

Number of Interviews, Survey Completion Rates, and Field Period Dates

	ANES Web Panel	KnowledgePanel®	Opt-In Panel A	Opt-In Panel B
N Interviews	1,397	1,210	1,221	1,223
Survey Completion Rate	65.8%	63.7%	4.6%	4.7%
Field Period	8/11/08 - 9/2/08	8/22/08 - 9/12/08	8/21/08 - 9/11/08	8/21/08 - 9/11/08

Statistical weighting & analysis

Each data set corresponding to each Web panel was weighted to the same U.S. Census population benchmarks for the key demographics of age; gender; race-ethnicity; educational obtainment; and Census Region. The two probability samples from KnowledgePanel and the ANES were also weighted to take into account their probability of selection in the U.S. population; an additional post-stratification weighting variable – Internet/non-Internet household – was also applied, as these two panels do have sample coverage of the non-Internet household population. All cross-tabulations presented in this study are based on these weighted data.

In general, the results from the two probability samples tend to be close to each other, while the same is true for the two non-probability samples. The fundamental difference in sampling methodology, when controlled in the experiment by the survey instrumentation and method of survey administration, produced very similar results within panel types but very different results between panel types.

Below are selected findings. All of the differences reported below, between the probability and non-probability samples, are statistically significant (p < 0.05).

Non-probability panels have low survey completion rates compared to probability samples, leading to increased risk of non-response bias. A difference of 13 times is notable.
Non-probability panelists are more likely to have taken a large number of surveys recently. A reported one in four opt-in panelists had participated in 20 or more surveys in the past four weeks.
About two and half times as many opt-in panelists said they spend 10 or more hours a week online at home than the probability-recruited respondents.
Non-probability panelists are more likely to report attitudes and behaviors in ways that indicate that they are early adopters of new products and concepts. In one opt-in panel, a majority (54%) agreed with the statement: "I often try new brands because I like variety," compared to only 34% from a probability-based panel.
The probability-based panels reported the correct population estimate of adult Facebook users – 13% in summer 2008 – while the opt-in panels reported usage at almost twice that level. ³
The above differences in the survey results occurred despite the use of statistical weights that helped improve the sample representativeness of the opt-in panel survey data for the purpose of this analysis. Therefore, the differences that are occurring between probability and non-probability panels are not correctable by post-stratification weighting and reflect real differences in attitudes and behaviors. Attachment B displays the weighted frequencies for the sample demographics for the four sets of interviews.

Additional cross-tabulations are displayed in Attachment C.

Some conclusions

The results from the opt-in panels are inconsistent with the conclusions reached in Francis S. Bourne's seminal essay, "The Adoption Process," where he indicates five new product adoption groups; the opt-in panels' results run contrary to common sense about the prevalence of early adopters in society. ⁴ Following on Bourne's logic, let's assume that there is a 1:1 correlation between attitudes and behavior with regard to "first to try new products"; that would yield 16% top two box using his model and published in his book.

Hot New Products We know that this is never the case. In fact, most literature suggests that interest in and likelihood of trying new products is somewhat overstated in surveys. Because that is the case, then the 24 - 26% from probability samples is very supportable; the 41 - 44% from opt-in non-probability samples is not supported by any known adoption curve metrics.

The factors responsible for the differences in results are in part found in the survey data themselves. First, there is the high percentage of the interviews completed by a small share of the opt-in panel respondent pool; one in four opt-in panel respondents has participated in 20 or more online surveys in the past four weeks. Excessive survey participation could lead to changes in attitudes, awareness, and behaviors related to new products and concepts (i.e., panel conditioning). Second, the survey completion rates are very low for opt-in panels. It is possible that the one in twenty opt-in invited cases that chose to participate in the survey are disproportionately early adopters. Finally, the opt-in panelists tend to use the Internet a lot more than the panelists from the probability samples, plausibly increasing their awareness of and taste for new products and services.

With each additional comparison study, the evidence is mounting that there are linkages between the design of the opt-in panels and the lower accuracy of their survey data. Similarly, the accuracy of the probability samples is a reflection of a commitment to the theory of random selection and full-population sample coverage.

Finally, the decreased accuracy of survey data from non-probability Web panels offsets their lower cost and ability to survey subpopulations with the precision needed for complex research. Clearly, their advent is a reaction to the slightly higher investment of basing Web panel recruitment on a statistically valid sample frame, and to the high costs of building Web panels large enough to support small area estimates or subpopulation surveys.

Dennis Dr. J. Michael Dennis leads Government and Academic Research for Knowledge Networks. Dr. Dennis has managed numerous surveys for academic and Foundation-based customers and for the Research Triangle Institute. A frequent presenter at the annual meeting of the American Association for Public Opinion Research, his current areas of methodological inquiry are nonresponse bias, panel conditioning, and data collection mode effects.

Osborne Larry Osborn is KN's Sr. Project Director for the American National Election (ANES) Web panel, sponsored by the National Science Foundation (built and maintained by Knowledge Networks under a contract with Stanford University). His responsibilities include management of telephone recruitment of panel members; development and oversight of panel survey processes; data delivery; and report production. Prior to joining KN, Mr. Osborn was a Senior Survey Director at Abt Associates.

Semans Karen Semans is a Research Analyst, responsible for questionnaire development, quality control of survey instruments, monitoring data collection, and preparation of data sets.

Footnotes