[ Spring 2010 ]

NEW PRODUCT SPOTLIGHT

KnowledgePanel CalibrationSM: Using KnowledgePanel® to Improve the Sample Representativeness and Accuracy of Opt-in Panel Data

Creative-Crop-Digital-Vision-Getty-Images At Knowledge Networks we recognize that your research needs often require local geographies and/or extremely large sample sizes which at first blush would not suit the use of KnowledgePanel®. Yet, KnowledgePanel provides the most accurate sample platform for conducting online surveys in the U.S., due to its foundation in probability sampling and its national representativeness. As a result you may have wanted to use KnowledgePanel, but the research sample fit seemed out of reach.

To solve this, we have developed KnowledgePanel CalibrationSM. This new KN product meets your needs for higher accuracy than opt-in only sample would provide and can be used when we simply cannot deliver the entire sample you need on KnowledgePanel.

To read more about the KnowledgePanel Calibration methodology click here.

We have seen the advantage KnowledgePanel Calibration brings in terms accuracy of results versus using only an opt-in sample for a study.

As shown below, Knowledge Networks examined the responses to five early adopter questions* on both KnowledgePanel and an opt-in panel. We then applied our KnowledgePanel Calibration technique to the sample using the 200 KnowledgePanel cases which represented one sixth of the total study "n." As you can see, the top 2 box agreement results found in opt-in only sample changed between 21% and 42% as a result of applying the KnowledgePanel calibration weights. Please note that we are not talking about percentage points; but instead the metric is percent change between the opt-in results compared to the calibrated result.

Agreement Rates for "Early Adopter" Measures (Top 2 box)
by Sample Source and with Calibration

chart

Note: Sample sizes n=1,000 Opt-in Panel, 200 KN Panel, 1,200 Total

The point is that if KnowledgePanel Calibration were not used, there would have been overestimated agreement rates by as much as 42%, and a business or policy decision would have been affected.

As a result, we believe that KnowledgePanel Calibration provides an advantage to your research findings, both in terms of accuracy of results versus using only an opt-in sample for a study.

Advantages of KnowledgePanel Calibration

For some studies involving either very large sample sizes or the targeting of very rare or small subpopulations, a blended-web sample approach is an effective methodology, because it increases the accuracy of results.

Compared to conducting the survey exclusively with non-probability opt-in panels, the advantages in using KnowledgePanel Calibration are:

Improved accuracy and sample representativeness. The calibration weights are based on the nationally representative KnowledgePanel, providing a statistical means for projecting the opt-in panel data to the entire national population (for the surveyed group).
Operational and analytic efficiencies. Knowledge Networks manages the coordination and data collection from both samples, producing a single analytic deliverable having a single calibration weight.
Cost effectiveness. In most studies, the cost of the approach is only modestly different than conducting the study exclusively with opt-in web panel samples, yet the accuracy pay-off is high.

Importantly, KN has proven the technique in past KN studies. We have employed the calibration approach in dozens of studies during the past year for Federal and State government agencies, industry associations, and consumer research firms, as well as for our custom market research projects.

Here is how KnowledgePanel CalibrationSM Works:

In the KnowledgePanel Calibration approach, we conduct the same survey with KnowledgePanel respondents as well as with a companion sample of respondents from an opt-in web panel. The same screening criteria are used for both sample sources to identify the eligible sample for the interview (e.g., customers of an energy company in a certain metro area, sufferers of certain medical conditions, consumers of certain products, etc.). The KnowledgePanel interviews serve an important function by providing the statistical information needed to calibrate the interviews from the non-probability sample source. We are, simply put, substituting KnowledgePanel benchmarks for the CPS and applying conventional post-stratification weighting using the control totals from KnowledgePanel.

The calibration is useful in correcting for sampling error and self-selection bias in the non-probability web panels: for instance, exclusion of non-Internet households and over-representation of hyper Internet users and of early adopters of new products and services, to name just a few. While the calibration approach cannot correct for all the error present in the opt-in panel interviews, the calibration will improve accuracy of study findings and insights, giving researchers more confidence in the data investments they have made.

Please visit http://www.knowledgenetworks.com/ganp/ for a list of Knowledge Networks' Government & Academic Research representatives and more information about KnowledgePanel®.