2008 AAPOR PRESENTATIONS

COMPUTING METRICS FOR ONLINE PANELS
Mario Callegaro, Knowledge Networks; Email
Charles DiSogra, Knowledge Networks; Email

There are many ways and different terminology used to compute metrics such as response rates for online panels. In a recent effort, the Advertising Research Foundation is promoting an online quality research council to work on different quality topics; one such topic addresses metrics for online panels. This paper starts by reviewing the different metrics proposed by survey and market research associations such as the European Society for Opinion and Marketing Research (ESOMAR), the European Federation of Associations of Market Research Organizations (EFAMRO), the Interactive Marketing Research Organization (IMRO), and the International Organization for Standardization (ISO) for online data collection. We ground the discussion on the distinction between probability-based online panels and opt-in or access panels. After describing the data collection stages for a prerecruited probability-based web panel and for an opt-in panel, we propose some measures for response, refusal, attrition, and re-interview rates and the relevant formulas. We use concepts derived from existing AAPOR standards for web survey response rates. Complex design and weighted response rates are also addressed. We conclude with a discussion of which metrics can be used with a specific online panel (probability-based or opt-in) and debate the meaning of the different metrics and the relationship between response rates and survey quality.

More Info – Full Paper

THE USE OF LOTTERY-TYPE-GAME ITEMS TO EXAMINE CHARITABLE GIVING AFTER KATRINA: A FIRST LOOK AT USING GAMES IN ONLINE PANEL RESEARCH
Christina Fong, Carnegie Mellon University, National Bureau of Economics
Poom Nukulkij, Knowledge Networks; Email

The use of online panel research creates unique opportunities in many different areas of research. Surveys administered using online panels are less susceptible to respondent biases (e.g., social desirability) that may be present in telephone or in-person surveys, potentially allowing for more accurate depiction of attitudes and behaviors. In this research, we present a study that takes advantage of the unique capabilities of online research. In June 2006, researchers affiliated with the National Bureau of Economic Research, Carnegie Mellon University and Knowledge Networks conducted a survey investigating peoples' attitudes and behaviors towards charitable donations to victims of Hurricane Katrina. Participants consisted of members of KnowledgePanel®, the RDD-recruited Internet panel managed by Knowledge Networks. Panelists were asked to allocate any portion of $100 to a charity in a state affected by Katrina or keep it for themselves; they could divide the amount in any way, using any criteria they saw fit. One critique of studies of this nature is that hypothetical behavior may not reflect actual behavior. To increase the salience of the experiment, participants were informed that Knowledge Networks would pay the dollar amounts indicated for one out of every ten participants. The "winners" who would have the money donated to charity or given to them were those whose randomly-selected number matched the first digit chosen in a pre-specified drawing from the Louisiana State Lottery Pick 3 lottery game. First, we will describe the study design, detailing the lottery game and context that was presented to participants. Information about the rules and regulations governing the use of lottery type games will also be presented. Next, we will provide a brief overview of demographic and panel characteristics related to donating behavior. Finally, we will discuss the fulfillment of these incentives with the selected charities and reactions from KnowledgePanel members.

More Info – PowerPoint | Full Paper

A NATIONAL EXPERIMENT EXAMINING THE EFFECTS OF POLITICAL ORIENTATION AND THE SALIENCE OF RACE AND LOCATION ON SELF-REPORTED CHARITABLE GIVING AFTER HURRICANE KATRINA
Linda J. Skitka, University of Illinois-Chicago
Elizabeth Mullen, University of Illinois-Chicago
Bill McCready, Knowledge Networks; Email

The purpose of this study was to experimentally examine the relationship between politically motivated reasoning, and the relative salience of race and location on self-reported charitable giving after Hurricane Katrina in a national random sample (N = 942). The experiment was a 2 (Locale: Gulfport, New Orleans) X 3 (Race Prime [using imbedded photographs]: None, White, Black) X 3 (Political Orientation: Democrat, Moderate, Republican) between-subjects experimental design with the dependent measure being self-reported recall of participants' charitable giving after Hurricane Katrina. Demographics and "personal experience with Katrina" were included as control variables. Results indicated that Republicans' self-reported charitable giving was affected by racial primes when they believed the images they saw represented damage to Gulfport, MS, but not when they believed the damage was to New Orleans. Republicans reported giving significantly more money when primed with white compared to black or no victims associated with Gulfport, but did not vary as a function of race primes associated with New Orleans. Democrats self-reported charitable giving was higher when photographic primes included pictures of people believed to be New Orleans hurricane victims than when not-primed with pictures of people, regardless of whether those picture primes were of Black or White victims of the catastrophe, but were unaffected by race in the Gulfport condition. Political moderates reported similar levels of charitable giving regardless of locale or race priming conditions. These results suggest that ideologues' memory or willingness to report money donated to charity are affected by subtle cues about race or context: Republicans reported giving more to whites when primed with a more stereotypically white and conservative context (Gulfport, MS), whereas Democrats reported giving more when primed with people rather than property damage in the more stereotypically black and Democrat context of New Orleans.

More Info – PowerPoint

LONGITUDINAL TRACKING OF VOTER PERCEPTIONS DURING THE 2007-2008 ELECTION CYCLE USING A PROBABILITY-BASED INTERNET PANEL
Trevor Tompson, The Associated Press
Mike Lawrence, Knowledge Networks; Email
Stefan Subias, Knowledge Networks; Email

The Associated Press and Yahoo! News are conducting a longitudinal study tracking the attitudes and perceptions of a national sample of more than 2,000 U.S. adults during the 2007-2008 election cycle. Between November 2007 and October 2008, the survey participants will be administered eight (8) online surveys, measuring the perceptions of these individuals from before the first primary through just before the general election in November, 2008. The survey participants are members of the probability-based internet panel created by Knowledge Networks. The study presents an opportunity to investigate sample and data quality for an online longitudinal sample. Traditional longitudinal studies use costly means for limiting attrition, such as substantial respondent incentives and interviewer visits and telephone calls to respondents. While the study utilizes relatively small panelist incentives, email reminders, and limited telephone followup, it emphasizes the opportunity for participants to share their evolving perceptions from well before the first primary or caucus up to the general election along with a limited opportunity to share "man on the street" perceptions with AP reporters. It will also include planned fresh cross-sections to test for panel conditioning. We will present analytic findings regarding methodological issues using survey response data and metadata from the first few waves of online data collection. The areas of inquiry are: (i) predictors of unit-level non-response (survival analysis across the waves); (ii) item non-response patterns and the impact of item non-response on survey results; (iii) panel conditioning effects on the survey response data. Substantive findings will also be presented bearing on the relationship between media consumption related to the election and survey results.