2009 AAPOR Presentations
Knowledge Networks adheres to the AAPOR Code of Professional Ethics and Practices.
PRESENTATION OF A SINGLE ITEM VERSUS A GRID: EFFECTS ON THE VITALITY AND MENTAL HEALTH SUBSCALES OF THE SF-36V2 HEALTH SURVEY
Mario Callegaro, Knowledge Networks; Email
Jeffrey Shand-Lubbers, Knowledge Networks; Email
J. Michael Dennis, Knowledge Networks; Email
There is a lot of debate whether questions should be presented on a grid or in a single item per screen. From just an operational point of view, grids take less time to complete, which should decrease response burden, although new research shows that respondents seem to prefer a single item per screen. From a measurement point of view, grids pose numerous issues: higher item non-response, higher Cronbach's alpha, higher item non-differentiation, sometimes higher measurement error but not always.
In this experiment we are testing the Vitality (4 items) and Mental Health (5 items) subscales of the SF-36v2®. The SF-36v2Health Survey asks 36 questions to measure functional health and well-being from the patient's point of view. It is called a generic health survey, because it can be used across age (18 and older), disease, and treatment group, as opposed to a disease-specific health survey which focuses on a particular condition or disease. Two of the four items of the vitality scale and two out of five items of the mental health scale are reversed in meaning.
A sample of 2,500 KnowledgePanel® respondents was randomly assigned to one of five experimental conditions: Group 1: Standard grid; Group 2: Shaded grid; Group 3: One item per screen with horizontal response options; Group 4: One item per screen with vertical response options; Group 5: One item per screen with vertical shaded response options. Approximately 360 respondents completed the survey per condition for a completion rate of 73.4%.
The survey was optimized to be seen on a screen with minimum resolution of 800 by 600 pixels. During the study we collected the browser type for each respondent. This allowed us to exclude cases in which the survey was taken either on a MSNTV or on an iPhone/pda. The final sample used for the analysis, after exclusions, was of 1,449 cases for an average group size of 290.
We hypothesized that items presented on a grid would lead to more measurement error indicated by a lower Cronbach's alpha for and higher "inconsistencies" for the grid presentation. The main reason is that when items are presented on a single screen they allow the respondent to focus more on each question. When items are on a grid it is easier to get confused especially when the meaning of some of them is reversed. The index of consistency was in fact computed by correlating the total sum of scores for the reversed item with the total sum of scores for the non reversed items. If respondents are consistent in their answers the correlation should be higher.
Results are going in the expected directions (lower alpha level for the grid presentation and higher correlation for the single item presentation), although the differences among groups do not reach statistical significance.
More Info – Poster | Full Paper
WEB PANEL STUDIES OF THE 2008 ELECTION: NEW OPPORTUNITIES FOR CAUSAL ANALYSIS OF DYNAMIC CHANGE IN THE ELECTORATE
J. Michael Dennis, Knowledge Networks; Email
Trevor Tompson, The Associated Press
In regard to election survey methodology, 2008 might well be remembered as the year that Web panel surveys came of age. There were four major academic longitudinal surveys that were conducted by Web whose data will become publicly available: the ANES (Stanford University & University of Michigan); the NAES (University of Pennsylvania); the Cooperative Campaign Analysis Project (Stanford University, UCLA, et al); and the AP-Yahoo election study (with contributions from Harvard University & Stanford University). Our goal in the presentation will be to inform AAPOR members of innovations in survey design, principally as a result of innovative uses of Web-based panel surveys, giving election scholar new options for the study of the causes of dynamic change in the electorate during the 2008 presidential campaign. The paper will accomplish this by doing two things. First, we will summarize the similarities and differences of the major academic election surveys, including the traditional RDD and in-person surveys. Second, we will situate the AP-Yahoo election study in the context of the other major surveys, providing details on the survey methodology employed for the 2008 Associated Press-Yahoo News Election Poll. The paper will provide an in-depth description of the sample design, data collection methodology, respondent incentive plan, and panel retention efforts. By these means, the paper will provide a resource for election scholars making decisions about which public use data files to use and for what purposes.
More Info – PowerPoint
THE CHALLENGE AND IMPORTANCE OF INCLUDING SPANISH-DOMINANT LATINOS IN ONLINE PANEL STUDIES ADDRESSING THE U.S. HISPANIC POPULATION: LESSONS FROM KNOWLEDGEPANEL LATINOSM
Charles A. DiSogra, Knowledge Networks; Email
Tom Wells, Knowledge Networks; Email
Jannet Torres, Knowledge Networks; Email
KnowledgePanel LatinoSM (KPL) is a new Knowledge Networks product designed for the conduct of statistically balanced online surveys of the U.S. Hispanic population inclusive of Spanish speakers and non-Internet households. Recruited non-Internet Spanish-dominant households are provided with a laptop computer (Spanish configured) and given free dial-up access in order to take assigned Spanish-language surveys. As with all English-speaking non-Internet KnowledgePanel households, a free device and access to the Internet is likewise made available to the English-dominant Latinos to take assigned surveys in English. This paper presents the design features and challenges of building a two-language online panel from a multi-frame telephone recruitment approach combining both random-digit dial (RDD) methods and geographically targeted listed-surname samples. The limitation of using only these frames is the exclusion of no-phone and cell-phone-only households, a problem that is to be addressed in a second phase of KPL development in 2009. This paper will demonstrate that, in addition to the usual CPS demographic characteristics, a key language-related dimension for balancing the Latino population is a six-level language-spoken-at-home measure. National benchmarks for this measure come from the Pew Hispanic Center surveys. Key findings will show that there are notable differences among Latinos between level of language spoken at home and the language used to take online surveys. These differences also extend across multiple levels of acculturation when compared to a published acculturation scale. Language, although a strong proxy for acculturation, is by no means 100% reliable when dichotomizing and selecting Latino samples on language alone. Finally, using data from Knowledge Networks' public affairs survey which is part of the battery of proprietary surveys to profile panel members, comparison results will be shown when national Latino samples include and exclude Spanish speakers demonstrating the importance of language used in survey samples to achieve representative study estimates.
More Info – PowerPoint
PANEL CONDITIONING AND ATTRITION IN THE AP-YAHOO NEWS ELECTION PANEL STUDY
Yelena Kruse, Knowledge Networks; Email
Mario Callegaro, Knowledge Networks; Email
J. Michael Dennis, Knowledge Networks; Email
Stefan Subias, Knowledge Networks; Email
Michael Lawrence, Knowledge Networks; Email
Charles A. DiSogra, Knowledge Networks; Email
Trevor Tompson, The Associated Press
This paper examines the potential for attrition bias and panel conditioning in a longitudinal online election panel survey. The 2008 Associated Press-Yahoo! News Poll was conducted by Knowledge Networks with contributions from political scientists at Harvard University and Stanford University. The study involved an eleven-wave Web panel election survey of general population U.S. adults. The baseline data collection occurred prior to the onset of the political primaries (November 2007) and the final data collection took place after the November 2008 general election. All of the interviews were conducted with KnowledgePanel® respondents. Our assessment of panel conditioning is made possible by the study's sample design, which includes both the longitudinal sample component as well as three separate fresh cross-sectional samples. We examined the potential for the impact of panel conditioning on self-reports of certain attitudes, preferences, and behaviors such as the propensity to remain undecided and to report being certain about voting or having voted early. We employ Extended Cox hazard modeling to estimate risk factors responsible for attrition from the panel study. We found some evidence of panel conditioning for one political knowledge question. Not surprisingly, longitudinal respondents were more likely to correctly name Obama's religion than cross-sectional respondents, confirming previous results from the literature for panel conditioning on knowledge questions. For the other seven items about the presidential election, only two showed some evidence of panel conditioning. Regarding panel attrition, the rare (i.e., nonwhites, or adults age 18 to 29, or less than high school education) and non-rare respondent groups were attriting from the panel at the same rate during the first four waves, probably due to a specific incentive system put in place for the rare respondents. In subsequent waves, late-participating respondents were more likely to drop from the study, than early respondents in both rare and non-rare groups. Undecided Republicans were more likely to quit the panel during the first four waves than Republicans who chose John McCain. Finally, respondents who heard or read about election polls and remembered the sponsor were more likely to stay on the panel than those who did not remember the sponsor of the polls. These results are discussed in light of the current literature of panel conditioning and attrition in both traditional longitudinal household panels and online panels.
More Info – PowerPoint | Full Paper
THE EFFECT OF EMAIL INVITATION CUSTOMIZATION ON SURVEY COMPLETION RATES IN AN INTERNET PANEL: A META-ANALYSIS OF 10 PUBLIC AFFAIRS SURVEYS
Yelena Kruse, Knowledge Networks; Email
Mario Callegaro, Knowledge Networks; Email
Melanie Thomas, Knowledge Networks; Email
Poom Nukulkij, Knowledge Networks; Email
In online survey panels, panelists are accustomed to receiving requests to complete a survey and a standard (generic) invitation is often used. The purpose of this research is to examine how the subject and body of email invitations to complete online surveys affect the completion rates in samples from a nationally representative Internet survey panel. Specifically, the purpose is to investigate the effect of a customized (study-specific) versus a generic email invitation on survey completion rates, and how the topic of the survey may change this effect.
Ten experiments, in which respondents were randomly assigned to a customized or generic email invitation group, were conducted between 2008 and early 2009 using KnowledgePanel® respondents. Each survey had a different topic, and the results show that generic and custom email invitations elicit similar completion and break-off rates. To summarize the experiments we used meta-analysis statistics, which best fits our research question. Having access to the raw data also allowed us to exclude cases in which the email invitation bounced back or in which the members completed the survey from their member's page instead of clicking on the link in the email invitation. This feature allowed us to get a cleaner picture of the effect of the email subject and content on survey completion rates.
More Info – PowerPoint | Full Paper
INCORPORATING PUBLIC CHOICES IN POLICY PLANNING USING ONLINE TOWN HALL MEETINGS INTEGRATED WITH A STATEWIDE SURVEY
Bill McCready , Knowledge Networks; Email
Rita Brogan, PRR; Email
Poom Nukulkij, Knowledge Networks; Email
This paper will focus on the methodology from an innovative public policy project integrating true probability sampling with qualitative and quantitative data collection as part of the Long-Term Air Transportation Study (LATS), a Washington State Department of Transportation (WSDOT) project designed to incorporate public opinion in aviation policy planning.
Knowledge Networks and PRR designed an online town hall approach combining probability sampling, qualitative interactive dialogue and quantitative surveys into an integrated method that enriches the planning process. This paper will describe this methodology in detail, including two online town hall meetings in August and November 2008 and a statewide online survey in March 2009.
All participants in the online town hall meetings and survey are members of KnowledgePanel®. Participants were asked to attend two online town hall meetings consisting of a presentation on aviation policy in Washington state, polling questions to gauge opinions, and real-time dialogues with an expert in aviation policy, with approximately 100 people per meeting. The resulting interactions provide policymakers with access to public opinions during the planning process, while the public has a unique opportunity to contribute to important policy planning. This research paper will describe the outcomes of the two online town hall meetings in terms of their impact on the planning process and in terms of their contribution to the design of the statewide survey.
Furthermore, this paper will describe the results of the statewide survey both in terms of it being derived from the online town hall meetings and in terms of its own quantitative contribution to the overall planning process. The key research question for this paper is "how do qualitative dialogues contribute to, and integrate with, quantitative surveys to produce an enriched source of public opinion within the policy planning process"?
More Info – PowerPoint
DO WE HEAR DIFFERENT VOICES?: INVESTIGATING THE DIFFERENCES BETWEEN INTERNET AND NON-INTERNET USERS ON ATTITUDES AND BEHAVIORS
Chan Zhang, University of Michigan
Mario Callegaro, Knowledge Networks; Email
Melanie Thomas, Knowledge Networks; Email
Charles DiSogra, Knowledge Networks; Email
Although the digital divide is closing, the gap is still wide. According to the most recent Current Population Survey supplement, in 71% of American households at least one member uses the Internet at any location (including at home), and only 62% of households have an Internet connection from home as of October of 2007. Moreover, some subgroups of the population are more likely to be offline/non-Internet than others. Recent literature is exploring the difference between Internet and non-Internet households on a variety of survey topics. Some papers are also exploring how weighting would compensate for the absence of non-Internet households in a sample. The studies done so far have used either telephone surveys or mixed modes (Web and mail) to collect the data. In our study, we compare Internet and non-Internet households answering survey topics on the same platform (i.e. a Web survey). The analysis uses data coming from KnowledgePanel®, a probability-based online panel recruited by Knowledge Networks via RDD where non-Internet households are given a WebTV, or in some cases a laptop computer and Internet connection enabling them to complete online questionnaires. In this paper, demographic characteristics of the offline population are first explored and compared to CPS October 2007 supplement. Then, responses to a series of survey questions are analyzed using multinomial logistic regressions, to identify whether Internet access status still makes a difference when controlled by relevant demographic variables. Three different models are presented controlling for 4, 7 and 12 demographics variables highly correlated with Internet status. This simulation is measuring what would have happened if non-Internet households are absent from the analysis. Initial results reveal differences between the two groups that do not disappear after controlling for the demographics correlates of Internet status.
More Info – PowerPoint | Full Paper
