2007-2008 AP-Yahoo News Election Panel Study
Response Rates
Description of KnowledgePanel®
The Knowledge Networks (KN) Web panel is a probability-based panel. By definition, all members of the KN Web panel have a known probability of selection. As a result, it is mathematically possible to calculate a proper response rate that takes into account all sources of nonresponse. In contrast, opt-in Web panels do not permit the calculation of a response rate since the probabilities of selection are unknown. Consequently, opt-in panels are mathematically capable of computing only the survey completion rate representing the final stage of gaining cooperation of survey research subjects, excluding the nonresponse resulting from panel recruitment, connection, and panel retention.
The panel sample selection methodology used for this study was developed by KN in recognition of the practical issue that different surveys target different subpopulations. The methodology was also developed to attempt to correct for nonresponse and noncoverage error in the panel sample that could be introduced at the panel recruitment, connection, and panel retention stages of building and maintaining the panel.1 The panel sample selection methodology, which has been used by KN since 2000, provides statistical control on the representativeness of KN panel survey samples as measured by their proximity to population benchmarks.
Response Rate Calculations
Below are the components of the response rate calculation and the calculations themselves. An extended description of how to compute response metrics for online panels is found in Callegaro & DiSogra (2008) [ http://www.oxfordjournals.org/our_journals/poq/special.html.
Household Recruitment Rate (RECR)
Panel recruitment is done using RDD telephone methods and address based sampling (ABS). The recruitment rate is computed using the AAPOR Response Rate 3 (RR3). If at least one member of the household is recruited, the household as a unit is counted in the household recruitment rate.
Household Profile Rate (PROR)
The study profile rate is computed as an average of the cohort profile rates for all households in the study sample. Although the average number of profiled panel members per household is usually greater than one, a household is considered "profiled" when at least one member completes a profile survey.
Study Completion Rate (COMR)
For this particular study only one panel member per household was selected at random to be part of the study sample. At the end of the fielding period the number of cases that completed the study out of the number of assigned cases is the study completion rate.
Cumulative response rate (CUMRR) = RECR x PROR x COMR
Because for this study one member per household was selected in computing the cumulative response rate, we use the household recruitment rate multiplied by the household profile rate and the survey completion rate.
Comparison of response rates
It is important to note the differences between an RDD telephone or mail sample and KnowledgePanel®, which are very different in nature. RDD telephone and mail samples can be compared because they are one-time surveys. However, an online panel such as KnowledgePanel® is composed of people recruited at different times and, more importantly, committed to answer many surveys for a period of time and not just that single survey. Further, with the KnowledgePanel®, Panelists must also complete Profiling surveys in order to become members of the Panel. These differences are reflected in the recruitment and profile rates reported above. These differences make directly comparing response rates between one-time surveys and Panel surveys difficult and perhaps not illuminating.
References
Callegaro, Mario & DiSogra, Charles (2008). Computing response metrics for online panels. Public Opinion Quarterly 72(5). pp. 1008-1032.
The Full text of the paper is available on the Public Opinion Quarterly – Special issue webpage: http://www.oxfordjournals.org/our_journals/poq/special.html
AP Election Survey 2007-2008: Response Rate Report - pdf
1 In our patented solution (U.S. Patent No. 7,269,570), a survey assignment method uses a weighting factor to compensate for members which are temporarily removed from a panel because of an earlier draw of sample. This weighting factor adjusts the selection probabilities of the remaining panel members. The sample is drawn using systematic PPS sampling where the panel poststratification weights will be the Measures of Size (MOS). If the user requirements call for independent selection by stratum, the panel weights (MOS) will are adjusted in the following procedure: Sum the MOS for each stratum, call this sum Sh for stratum h. Consider the user-specified or system-derived target sample size for stratum h to be nh. Then multiply each MOS for Members in stratum h by nh/Sh. Then use an interval of k=1 and apply systematic PPS sampling to achieve the desired yield per stratum.
