[ Summer 2009 ]
New Methodological Research on Web Panel Conditioning and Attrition
Confirms previous findings of weak effects on KnowledgePanel® survey data
Highlights from a paper by Yelena Kruse, Mario Callegaro, J. Michael Dennis, Stefan Subias, Mike Lawrence, and Charles DiSogra (of KN) and Trevor Tompson of The Associated Press
Knowledge Networks recently presented six papers and poster sessions at AAPOR's 64th Annual Conference. This body of work extends KN's tradition of "research on research," through which we offer transparency to researchers interested in issues related to Web panels. Several of our AAPOR presentations focused on the 2008 Associated Press-Yahoo! News Poll conducted on KnowledgePanel®, with contributions from political scientists at Harvard and Stanford Universities. Here, we present highlights from a paper on this topic, given by Yelena Kruse, a Senior Research Analyst on our Government & Academic team.1, 2
The study is one in a series conducted by Knowledge Networks focusing on two areas of concern in survey research – Internet panel conditioning and attrition – phenomena that could affect data accuracy and reliability. Panel conditioning causes bias and error introduced into surveys when respondents have been influenced by participation in previous surveys, such that their answers differ from the answers of individuals who are interviewed for the first time. Also known as "time-in-sample bias," conditioning may be attributable both to the learning process and to increased experience in surveys. Through our research on research conducted and published since 2001, KN has found evidence of limited and, in some cases, statistically insignificant KnowledgePanel conditioning effects. Panel attrition is another area that we carefully have tracked over time, presenting methodological research, and operationally addressing this with diligent panel management.
The 2008 Associated Press-Yahoo! News Poll includes eleven waves of surveys with general population U.S. adults from KnowledgePanel – the only available probability-selected, nationally representative Internet panel. In terms of baseline, wave one obtained 2,735 respondents with a completion rate of 77%. On a compressed schedule with about four weeks between each survey, 1,086 respondents completed all eleven waves. We employed an incentive system based on demographics (rare vs. non-rare3) and speed of response (early vs. late respondents4). The data collection timeframe spans the period prior to the presidential primaries in November 2007, until just after the general election – a decidedly fast-paced schedule. The design is unique, in that it includes both a longitudinal component as well as three separate fresh cross-sectional samples.
In terms of conditioning, the study compares responses of longitudinal and cross-sectional samples to several survey questions, while controlling for other variables. It also tests for an increase in the stability of attitudes of the same respondents over time. In studying panel attrition, as predictors we measure demographic characteristics, such as age; ethnicity; gender; marital status; number of children in the household; and self-reported health status of respondents. In addition, we analyze how candidate preferences affect our panel drop-out rate as well as incentives' effect on attrition.
KnowledgePanel is an ideal platform for longitudinal studies due to high panel retention rate across waves of data collection; one can validate survey results in terms of their conformance to U.S. government benchmarks (e.g., the National Health Interview Survey or Current Population Survey). In addition, KnowledgePanel is managed to the highest industry standards). We create a sense of community, where voicing opinion takes precedent; in addition, we support our panelists through multiple channels of communication, to reflect varied levels of respondent sophistication. In terms of our recruitment methodology, we recently introduced Address-based Sampling (ABS) as part of our solution, to deliver the highest level of U.S. population coverage for probability based Web surveys. KN now employs a dual-frame sampling approach based on both ABS and Random Digit Dial (RDD). This combination provides accurate, statistically valid representation and coverage of many difficult-to-recruit populations, such as cell phone-only households; young adults; and racial and ethnic minority subpopulations.
Panel conditioning: Reassuring outcomes
While findings from the literature are mixed, our analysis found only three significant differences out of 14 comparisons (see Table 1):
- In wave six, we asked respondents to name Obama's religion and found a significant difference in the number of correct responses
- Longitudinal respondents were 28% more likely to answer correctly than cross-sectional sample respondents.
- This finding confirms previous results from the literature on panel conditioning with regard to knowledge questions.
- We also asked about feelings toward the election—excitement, interest, hope, boredom and frustration.
- We found significant differences on these attitudes, both in wave three ("hopeful") and in wave nine ("bored").
- We found significant differences on these attitudes, both in wave three ("hopeful") and in wave nine ("bored").
Table 1: Comparing Longitudinal vs. Cross-Sectional Samples
|
W3 |
W6 |
W9 |
Certainty to vote in the presidential election |
– |
– |
|
Voting early in the presidential election |
|
– |
– |
Correct answer for Obama's religion |
|
0.72 |
|
Feeling excited about the presidential election |
– |
|
– |
Feeling interested in the presidential election |
– |
|
–; |
Feeling hopeful about the presidential election |
0.83 |
|
– |
Feeling bored with the presidential election |
– |
|
0.44 |
Feeling frustrated with the presidential election |
– |
|
– |
Red Numbers: Significant odds ratios for the dichotomous (0 – 1) variable longitudinal/cross sectional sample. A significant odds ratio indicates a difference between the two samples on the item in question. The model is controlling for the following variables: demographics; party ID; religion and number of previous KN surveys taken.
In addition, we compared responses of the same respondents to attitudinal questions over time, to see if opinions had crystallized—a potential sign of panel conditioning, according to new research. In Chart 1, the Y axis shows respondent correlations between two waves; the X axis shows waves of comparisons. While in waves six and eight, these items are not asked, correlates in later waves appear among waves five and seven as well as waves seven and nine (an upward trend shows an increase over time). When we apply the Haan (1997) t-test for linear trends in a time series, the environment variable shows a significant increase over time—but only by about five percentage points.
Chart 1: Inter-Wave Stabilities
Two explorations of panel attrition
We examine attrition by applying two different analyses. The first uses a technique called survival analysis (widely used in medical research); in this case, survival means remaining on KnowledgePanel, i.e. not dropping out. For this, we use a Kaplan-Meier Method which does not control for covariates, and we compare rare vs. non-rare groups as well as survival rates by party ID. An important finding is that while the drop out rate for Republicans and Democrats is similar, more respondents attrite from Independent and other parties.
Our second attrition analysis employs a Cox Regression model, in which the effects of other variables are controlled. In waves two through four, the rare group receives a $5 incentive; the non-rare group receives nothing. In waves five through nine, the incentive system changes to include incentives for late respondents. Note that we analyzed waves one through four separately (in wave ten, everyone received the same incentive, and in wave eleven, no one received anything; therefore, we do not include these in our analysis).
Overall, rare respondents have a similar survival rate as non-rare. Only non-white respondents are slightly more likely to drop out than white respondents, but there are no age- or education-based effects. Not surprisingly, respondents with children in the household are more likely to leave the panel than those without. When running separate models for Republicans and Democrats, including candidate preferences for primaries/caucuses, we establish that Republicans who didn't know what candidate to vote for in the primaries were more likely to attrite than those who indicated their preference for John McCain. But candidate preference does not have an effect on Democrats' attrition.
When we include other demographic and attitudinal variables in the model for waves five through nine, we create two separate models—rare respondents vs. non-rare respondents. Notably, in both groups, late respondents are more than twice as likely to leave KnowledgePanel than early respondents. Internet households are more likely to remain, but there are no candidate preference effects on survival rates.
Chart 2: Survival Curves for Early vs. Late Respondents, Waves 5 - 9
In wave five, 984 people were classified as late respondents; 1,071 people were classified as early respondents.
Some evidence of late respondent attrition effect
In terms of panel conditioning, both analyses find weak evidence to support this potential effect. Only three out of 14 logistic regression models show a statistically significant effect of sample origin on survey responses. This outcome is reassuring, because a fast-paced longitudinal study design, conducted on a Web panel, might be considered a "worst case scenario" for potential conditioning effects.
As far as attrition, Republicans and Democrats have similar attrition rates, and an incentive seemed to help keep rare group respondents on KnowledgePanel. But late respondents are at least two and one-half times more likely to attrite than early respondents—a finding that we look forward to studying further. In addition, incentive seems to have some positive effect on survival rates.
A unique opportunity for testing longitudinal panel effects
The results described in this study apply only to KnowledgePanel, which provides a unique finding ground for longitudinal effects over time – from the same respondent – due to the panel's probability-based recruitment. Potential areas for future research include
- Panel conditioning: Investigate online panels that use recruitment methodologies other than probability sampling, so are comprised of different samples
- Panel attrition: Probe underlying reasons for higher attrition rate among late respondents
We look forward to bringing you additional highlights from our continual efforts in the area of research on research. In the next issue of A:I/R, we'll examine results from a study conducted via KnowledgePanel, to determine the extent to which demographic differences might explain the disparity in attitudes and behaviors when comparing Internet and non-Internet households. Please stay tuned!
Photo: © Amaxim-Dreamstime.com.
