2011 AAPOR Presentations
Knowledge Networks adheres to the AAPOR Code of Professional Ethics and Practices.
EXAMINATION OF PANEL CONDITIONING EFFECTS IN A WEB-BASED 2007-2008 ELECTION STUDY
J. Michael Dennis, Knowledge Networks; Email
Yelena Kruse; Knowledge Networks; Email
Trevor Tompson; Associated Press
Survey panels provide a useful mechanism for longitudinal measurement of within-person change and also can be cost effective compared to conducting another cross-sectional survey.
However, because of sample losses that predictably occur, the panel sample can deteriorate in terms of representativeness or else the respondents themselves might change in their self-reports to survey questions as a direct consequence of their participation in the panel. This latter set of consequences constitutes panel conditioning.
Not all panel conditioning leads to more error. Respondents might become better “survey takers” in being able to report their true attitudes and opinions in answering survey questions (i.e., improved predictive reliability). Other consequences are less helpful, such as actual changes in attitudes, opinions, behavior, and knowledge that can be attributed to panel participation.
In this paper, we’ll accomplish two things. First, we will present a typology of panel conditioning. Second, we will show the results of multivariate regression tests that isolate the impact of prior survey taking on survey responses. We have identified certain political knowledge questions for which there is evidence of limited panel conditioning effects (i.e., more panel experience, more knowledge), while a host of opinion and attitude items did not have such effects.
The data source will be the 2008 Associated Press-Yahoo! News Poll conducted by Knowledge Networks with contributions from political scientists at Harvard University and Stanford University. The study involved an eleven-wave web panel election survey of general population U.S. adults. The baseline data collection occurred prior to the onset of the political primaries (November 2007) and the final data collection took place after the November 2008 general election.
More Info – PowerPoint
SEASONAL YIELD VARIATION AND RELATED RESPONSE PATTERNS IN ADDRESS-BASED MAIL SAMPLES
Charles A. DiSogra, Knowledge Networks; Email
Erlina Hendarwan; Knowledge Networks; Email
The probability-based online KnowledgePanel® run by Knowledge Networks maintains size and representative diversity through ongoing mail recruitment. In 2010, four national address-based samples, drawn from the USPS Delivery Sequence File frame, were fielded in two replicate waves per sample from end of January through November. Each sample, drawn without replacement, consisted of two strata with 40% of the sample targeting Hispanic census blocks in one stratum and 60% allocated to a second stratum of all remaining blocks.
All mailings were virtually identical in size, strategy, timing and sample design. Every wave or mail drop had 22,500 residential addresses covering all 50 states. Each had an initial packet sent first class to Current Resident with a $2 incentive enclosed, a reminder postcard mailed one week later to the entire sample, and a reminder letter mailed to non-responders two weeks after the postcard. All materials are in English and Spanish.
Three response modes were in effect at all times: postage-paid reply envelope, online Web site portal, and toll-free telephone number. Total yield is measured daily as response from all modes. Mode-specific yields are also tracked daily. Each response is linked with its respective mail sample cohort through a unique PIN.
Mailings were at the ends of January and February, early April and May, end of June, mid-July, and ends of August and September. Total respondent yields show variation across mailings with lowest yield occurring with the May mailing and highest with the July mailing. The low May mailing yield is consistent for reply-mail returns and telephone call-ins. Online respondents, however, demonstrate a very different pattern showing progressively higher yields from January right through the August mailing (September mailing results are in progress at this writing). Statistics on yield variation and response patterns plus plotted seasonal response curves are presented.
More Info – Full Paper
EXPLORING HEALTH-RELATED EXPERIENCES AND ACCESS TO CARE: DIFFERENCES BETWEEN ONLINE AND TELEPHONE SURVEY ADMINISTRATION
Michelle M. Doty; The Commonwealth Fund
Jordon Peugh, Knowledge Networks; Email
Jeffery Shand-Lubbers, Knowledge Networks; Email
In an era of health reform where decisions concerning health care delivery are changing, researchers and policy makers need to rapidly gauge public opinions and needs. Online survey administration using a probability-based panel of respondents offers the possibility of getting results quickly without sacrificing integrity of data. This paper compares results from nationally representative surveys of adults using two different data collection modes: online and telephone.
To explore the comparability of data collected online versus by telephone, we fielded identical questions on access to care, experiences with health care delivery, satisfaction with quality of care, and demographics to a national landline RDD sample of 2,501 U.S. adults and a national probability-based web panel sample of 1,039 U.S. adults, using Knowledge Network’s KnowledgePanel®. The purpose of this paper is to highlight where results from the two modes align well; to assess where responses do not converge; and explore potential reasons for discrepancies.
Using these two methodologies, we found considerable similarities on responses to behavior/experience related variables, such as cost-related access problems and ease in getting care after-hours. There were 15 to 20 percentage point differences, however, on attitude and opinion questions. We compare how responses to opinion questions vary by income, race-ethnicity, insurance status, and other key demographics.
In addition to an overall comparison of these modes we explore the impact of cell phone only households (which are included in the online panel and not in the RDD sample) and the probability that certain sub-groups such as those with a chronic condition, low-income respondents, and racial/ethnic minorities may be excluded with RDD samples.
The paper will discuss implications for using the different methodologies for conducting health policy research and consider how these methodologies differ on other items such as design effects, response rates, time in field, and cost ratios.
More Info – PowerPoint
CONDUCTING SURVEYS ON SENSITIVE TOPICS; SEXUAL BEHAVIOR AND HEALTH
William McCready , Knowledge Networks; Email
Debra L. Herbenick; Indiana University
Michael Reece; Indiana University
Rick Li; Knowledge Networks; Email
KnowledgePanel provided the survey data for one of the largest nationally representative study of sexual-health behaviors ever fielded, the National Survey of Sexual Health and Behavior (NSSHB), conducted by Indiana University sexual health researchers from the Center for Sexual Health Promotion. These data provide an updated snapshot of contemporary Americans’ sexual behaviors. It includes the sexual experiences and sexual-health-related behaviors of 5,865 adolescents and adults ages 14 to 94. This presentation will focus on the basic age-related findings and the various methodological approaches to panelists that resulted in the consent, cooperation and trust required to generate these important health-related data about a sensitive topic. Initial findings from the survey, presented in nine separate research articles, were published on Oct. 1 in a special issue of The Journal of Sexual Medicine, a leading journal in the area of sexual health. According to the study’s findings, 1 of 4 acts of vaginal intercourse are condom protected in the U.S. (1 in 3 among singles). These data, when compared to other studies in the recent past, suggest that although condom use has increased among some groups, efforts to promote the use of condoms to sexually active individuals should remain a public health priority. The study helps both the public and professionals to understand how condom use patterns vary across these varying stages in people’s relationships and across ages. A unique feature of the study was the inclusion of adolescent men and women. Many surveys of adolescent sexual behavior create an impression that adolescents are becoming sexually active at younger ages, and that most teens are sexually active, these data show that partnered sexual behaviors are important but by no means pervasive aspects of adolescents’ lives. In fact, many contemporary adolescents are being responsible by abstaining or by using condoms when having sex.
More Info – PowerPoint
DOCUMENTING & TRACKING HOW COUPLES MEET
William McCready , Knowledge Networks; Email
Michael Rosenfeld; Stanford University
Larry Osborn; Knowledge Networks; Email
The National Science Foundation supported a large-sample survey using KnowledgePanel to generate information about how couples meet, with a special emphasis on GLBT couples. This presentation will discuss the findings from the baseline study and propose plans for tracking partners as these relationships change over the coming years. Of particular importance are the different meeting patterns of GLBT and hetero couples. These pattern-differentials are hypothesized to influence the persistence or disintegration of the relationships. This presentation focuses on three aspects of KN's panel that were crucial in making this study work: 1) the ability to efficiently oversample self identified GLB respondents, 2) the ability to follow respondents over time at modest cost, and thereby to study couple dissolution rates and 3)the ability to capture respondents' own stories, in their own words. The purpose of the project was to provide a national perspective on how couples in various social contexts meet and stay together or dissolve their relationships. This research is an initial foray into the longitudinal study of partnering in the contemporary era. This project has relevance for both health and sociological topics and is groundbreaking. The panelists in KnowledgePanel provide self-identification as to their sexual identity and more than 1,000 say they are GLBT. These panelists provided a national perspective on how a rare population partners and it could only be achieved with a large national probability sample that incorporated a pre-screening for sexual identity. The sample can be tracked inexpensively because they are in regular contact with Knowledge Networks while doing surveys for other projects. A crucial, and valuable, component to this project is a qualitative aspect of the work consisting of the ‘meeting-up-stories’ of the couples, both straight and gay. The nature of these stories becomes one of the predictors of relational permanence or dissolution.
More Info – PowerPoint
CHOICE OF CONTENT PRESENTATION MODE IN WEB-BASED SURVEY ADMINISTRATION
Larry Osborn; Knowledge Networks; Email
Wendy Mansfield; Knowledge Networks; Email
Carl Ramirez, U.S. GAO
Jill Lacey, U.S. GAO
Virginia Chanley, U.S. GAO
Gregory Wilmoth, U.S. GAO
Ana Ivelisse Aviles, U.S. GAO
Survey research often involves information provision to respondents in questionnaires. The information provision allows for measurement of research subjects’ reactions, attitudes, and opinions regarding the supplied information. Web-based survey administration, since it supports multimedia, allows for multiple modes to provide such information, including video, audio, or on-screen text. An experiment testing the optimal mode(s) of information provision would be useful for researchers in the development of web-based survey content. Recently, KnowledgePanel® provided data for a nationally representative study on peoples’ opinions and attitudes around global climate change and geoengineering technologies, conducted for researchers from the U.S. Government Accountability Office (GAO). The Internet-based survey was conducted with a sample of 1,006 U.S. residents aged 18 and older and included the presentation of content outlining two examples of geoengineering technologies. Respondents were initially offered the choice of reviewing this content in one of three modes—video/audio, audio-only, or on-screen text. Respondents were subsequently allowed to review the content in one or both of the remaining alternative modes, if desired. Of the total sample, 175 respondents chose to review the content in a second presentation mode and approximately 50 respondents chose to review in all three modes. Our paper will examine three key questions. First, are there presentation modes that appear to be more or less effective based on a statistical analysis of combined initial and subsequent content reviews by respondents? Second, are there respondent characteristics related to single-mode choice versus multiple mode choices? Third, are there respondent characteristics related to initial mode preference and/or switching to other modes? Based on these data, implications and recommendations for single and multiple information presentation modes will be discussed. We expect the study results to be useful for researchers who design online survey experiments and discrete choice, referendum, and willingness to pay studies.
More Info – Full Paper
USING A PROBABILITY-BASED ONLINE PANEL TO SURVEY AMERICAN JEWS
Graham Wright; Brandeis University
Benjamin Phillips; Abt SRBI
Josh Tobias; Brandeis University
Jordon Peugh; Knowledge Networks; Email
Karen Semans; Knowledge Networks; Email
Americans Jews represent less than 2% of all adults in the U.S. This low prevalence makes creating a probability-based sample of American Jews cost-prohibitive. We explore the use of a probability-based online panel to reach a representative national sample of respondents who are Jewish by religion or background, the latter being identified by screening panel members with no religious affiliation.
In 2010, we conducted three surveys of Jewish respondents using Knowledge Networks’ (KN) KnowledgePanel®. This online panel of approximately 50,000 U.S. residents aged 13 and older is built through random-digit dialing and address-based sampling and includes participants who would not otherwise have online access. The panel had three distinct advantages. The large panel size and prior information on religious affiliation allowed a larger sample than would be possible in a sample specifically drawn for this purpose. Panel cooperation rates were high and demographic information available on panel nonrespondents allowed for adjustment for nonresponse. Issue saliency bias was reduced by the generic nature of panel recruitment.
We examine the coverage properties of the KnowledgePanel® sample with respect to demographic and religious factors in comparison to surveys of American Jews from nonprobability online panels, traditional mail panels, RDD designs, and meta-analyses of surveys with items on religion. The panel appeared to be representative with respect to sex, region, and education. However, young adults aged 29 and below were under-represented. The panel initially appeared to be biased with respect to Jewish religious denominations. However, estimates of the denominational composition of American Jews were strongly influenced by question wording and response options as well as rapid denomination change among American Jews.
We also conduct sensitivity analyses of the impact of weighting assumptions on outcomes of interests. In general, estimates were quite robust to weighting assumptions, strengthening confidence in the sample.
More Info – Full Paper
SAME FAMILY, DIFFERENT PLANET: COMPARING THE VIEWS OF TEENS AND PARENTS ON THEIR RELATIONSHIP AND POTENTIAL SUBSTANCE USE
Roger Vaughan, The National Center on Addiction and Substance Abuse at Columbia University (CASA)
Emily Feinstein; The National Center on Addiction and Substance Abuse at Columbia University (CASA)
Jordon Peugh; Knowledge Networks; Email
Jeffery Shand-Lubbers; Knowledge Networks; Email
Research with dyads of parents and teens provides a way to compare differences and similarities in attitudes, opinions, and experiences within the same households. Capturing and comparing these two perspectives can be helpful for planning policy interventions and communications related to helping both parents and teens deal with substance use issues.
As part of a research series on teens and parents, the National Center on Addiction and Substance Abuse at Columbia University (CASA) conducted surveys of parents and teens using KnowledgePanel, Knowledge Networks’ (KN) national probability-based web panel that is recruited through RDD and ABS. Data were collected on 456 parent/teen dyads.
Examining these dyads, we uncover important differences between teen and parent perceptions about their relationship and the availability to and use of drugs and alcohol by teens. We find significant within pair differences on each of eight variables asked of both teens and parents. Looking at these data in total can in some cases obscure the degree of difference between parents and teens in the same household. While 7% of parents and 11% of teens overall report that half or more of the teen’s friends drink alcohol, looking within pairs 30% disagree on how many of the teen’s friends drink. One of the largest differences between parents and teens was perceptions of teen’s likelihood to try drugs in the future, with 61% of parent/teen pairs in disagreement. Generally, parents are more likely than teens to expect that the teen will try drugs in the future.
The full paper will present information on KN methodology, survey development, sampling methodology, and comparison to a parallel RDD survey. Presented results will identify demographic factors associated with greater or lesser differences among pairs and explore associations between parent/teen disconnect and risk factors for substance use.
More Info – Full Paper
HOW THE ORDER OF RESPONSE OPTIONS IN A RUNNING TALLY CAN AFFECT ONLINE SURVEY ESTIMATES
Tom Wells; The Nielsen Company
Mario Callegaro; Google
Charles A. DiSogra, Knowledge Networks; Email
In the design of online surveys, running tallies or constant sums are used to help respondents sum up the allocation of amounts so that the total sums to 100%.
We hypothesized that for time allocation, the order of the presentation of the time categories could make a difference in the distribution of reported time spent. We expected primacy effects, with the first-presented time category having a higher allocation of time than the later-presented options.
Two independent experiments were conducted, one on a sample of Google online advertisers, and one on a general population adult sample from KnowledgePanel®.
In the Google experiment, advertisers were asked to estimate percent of time to complete an online task consisting of 3 subtasks. The order of the subtasks was randomized. Consistent with our hypothesis, we found primacy effects — the subtask presented in position 1 obtained a higher mean allocated time than when presented in later positions.
In the KnoweldgePanel experiment, respondents were asked to provide running tallies of the percentage of television they typically watch during the morning, afternoon, and evening (separately for weekdays and weekends). As before, the order of the subtasks was rotated. Primacy effects were detected again, however differences by position were small and not statistically significant. Because time spent watching TV is a regular activity, viewing patterns are more likely to be encoded or ingrained in memory, and more likely to be reported reliably, with responses less susceptible to order effects. In the second experiment, we also tested the restriction that running tallies must sum to 100%. Roughly 25% of respondents were prompted for out-of-range sums on the first tally (weekday) and 5% were prompted on the second tally (weekend). This drop-off in prompting indicates that respondents quickly learned what is required to complete a given task.
More Info – PowerPoint
CAN A NON-PROBABILITY SAMPLE EVER BE USEFUL FOR REPRESENTING A POPULATION?: COMPARING PROBABILITY AND NON-PROBABILITY SAMPLES OF RECENT COLLEGE GRADUATES
Cliff Zukin, Rutgers University
Jessica Godofsky, Rutgers University
Carl Van Horn, Rutgers University
Wendy Mansfield, Knowledge Networks; Email
J. Michael Dennis, Knowledge Networks; Email
In spring 2011 the John J. Heldrich Center for Workforce Development at Rutgers University will survey recent college graduates to examine the extent to which their education prepared them for jobs in the US labor market. Contacting this sample—defined as those 22 to 29 years of age and having graduated with a bachelors degree between 2006 and 2010—is like finding a very expensive needle in a haystack if a typical RDD design is used. The costs are cut considerably by using Knowledge Networks' panel, a nationwide probability sample of some 50,000 members. KN has some 700 such members of their panel who fit the desired profile of respondents. However, even if we managed to reach and interview two-thirds, we would have fewer than 500 respondents. To augment this number, surveys will be completed with a like number of similarly profiled respondents selected through an opt-in non-probability sample. While we are not sanguine about employing a non-probability design, neither are we sanguine about probability surveys where the response rate may be 20% or so, as calculated by AAPOR standard definitions. Yet, this is where we find ourselves at the beginning of a new decade. Response rates for RDD surveys are threatening to reach single digits, and traditional RDD landline sampling frames miss close to 40 percent if Cell Phone Mainly households are added to those who are Cell Phone Only. Clearly it is time for experimentation. This paper compares and contrasts respondents selected through a KN probability sample with those obtained through an opt-in non-probability sample. We’ll test the null hypothesis that there is no difference between the samples on economic, workforce, political, social and psychological variables. We’ll also examine the extent to which we can weight our way out of this problem as survey research enters the 21st century.
More Info – PowerPoint
