2005 AAPOR PRESENTATIONS

COMPARING THE KNOWLEDGE NETWORKS WEB-ENABLED PANEL AND THE IN-PERSON 2002 GENERAL SOCIAL SURVEY: EXPERIMENTS WITH MODE, FORMAT, AND QUESTION WORDINGS.
Tom W. Smith, NORC, University of Chicago; Email
Rick Li, Knowledge Networks; Email
Mike Dennis, Knowledge Networks; Email

This paper is an examination of the interplay of mode of data collection and item nonresponse caused by 'Don't Know' responses. A series of experiments were carried out to replicate and extend earlier comparisons between an in-person survey (the General Social Survey) and Web-enabled Knowledge Networks surveys. Results indicate that levels of Don't Know on e-survey are highly contingent on format and conditioning effects and that a Web-based format for Don't Knows can be designed to match DK levels found in in-person surveys. Moreover, differences in distributions across modes are notably reduced with DK responses are excluded, but do not disappear. Support for public spending was generally greater on the in-person GSS than on the KN e-surveys and the differences were notable on items concerning the underclass. Finally, wording experiments produced results that were consistent in direction across modes, but not always comparable in magnitude. Thus, while e-surveys produce similar findings to in-person surveys in many circumstances, notable differences occur in other situations.

More Info - PowerPoint | Full Paper

DATA COLLECTION MODE EFFECTS CONTROLLING FOR SAMPLE ORIGINS IN A PANEL SURVEY: TELEPHONE VERSUS INTERNET.
Cindy Chatt, Gallup; Email
Mike Dennis, Knowledge Networks; Email
Rick Li, Knowledge Networks; Email
Paul Pulliam, RTI; Email

We evaluate telephone and Internet-based modes of survey data collection by controlling for sample origin. Previous research has focused on sample effects only. Our main result is that substantive response differences are primarily associated with mode of data collection and not with sample origin. Sample origin is controlled by conducting both Internet and telephone interviews with members of the Knowledge Networks (KN) web-enabled panel. The survey, which was sponsored by RTI International, measures policy and civic attitudes regarding 9/11 in early 2002, and was designed by RTI International and the Odum Institute at the University of North Carolina. The survey analysis is based on 2,979 web interviews with KN panelists, 300 telephone interviews with KN panelists, and 600 telephone interviews with persons that refused to join the KN panel or else take the web panel survey. The differences caused by mode in this Internet versus telephone study were strikingly similar to the telephone versus mail mode effects found in civic attitude studies by Tarnai and Dillman and in telephone versus face-to-face mode effects by Krysan. These studies found a tendency (which we confirm) for telephone respondents to answer on the extreme positive end of the scale. The Internet respondents are more likely than both telephone sample groups to use the full range of scales.

More Info - PowerPoint | Full Paper

Statistical Tests of Data Quality in a Contingent Valuation Survey Using Knowledge Networks Data.
Joel Huber, Duke University; Email
Jason Bell, Duke University; Email
W. Kip Viscusi, Harvard University Law School; Email

This paper uses statistical tests on Knowledge Networks data to determine the extent of survey bias in a contingent valuation study about water quality with randomization of survey versions. The validity tests include examining the data against theoretical predictions and an across-person test requiring respondents to be sensitive to the scope of differences in cost of living and water quality. There are also validity tests determining whether panel membership influenced the valuation results. There are four variables in the regression of the determinants of the value of water quality benefits. The first variable is whether the respondent stopped and then continued the survey at a later time. The second variable is the time the respondent has been a member of the Knowledge Network panel. The third variable is the number of days the respondent took to complete the survey. The final survey methodology variable tested is whether the respondent subsequently quit the panel at any time until May 2004. Although the sample is nationally representative it is useful to test for possible selection biases arising from panel members who were invited to participate but did not successfully complete the survey, and these results are also included. Overall, there is no indication that any of these key aspects of the panel methodology bias the survey responses. There were no significant effects of any of the Knowledge Networks panel variables so that there is no evidence that national performance of the survey task is importantly influenced by any of these variables.

More Info - PowerPoint | Full Paper

STATISTICAL TESTS OF DATA QUALITY IN A CONTINGENT VALUATION SURVEY USING KNOWLEDGE NETWORKS DATA.
Trudy Cameron, University of Oregon; Email
George M (J.R.) Deshazo, UCLA; Email
Mike Dennis, Knowledge Networks; Email

Researchers frequently acknowledge several reasons for possible non-representativeness in surveys using panel samples. We model the selection process for the Knowledge Networks panel, starting with a random-digit-dialed set of initial contacts and following these cases through a number of distinct attrition opportunities, ending with one sample drawn for an actual survey and the individuals who chose to respond to it. Using GIS methods, we match over 525,000 RDD addresses or telephone exchanges to the corresponding county and the most appropriate census tract. We use a set of fifteen orthogonal factors based on census tract characteristics, plus county voting percentages for candidates Gore and Nader in the 2000 Presidential election. We find many statistically significant determinants of attrition at our different attrition opportunities. To illustrate the effects of selection, we consider a second subsample where survey respondents expressed their opinions about the proper role of government in terms of environmental, health and safety regulations .In a formal maximum likelihood selection model, we find some evidence of a slight liberal (pro-regulation) bias that may stem from non-random selection, but the effect is not statistically significant and the hypothesis of no liberal/conservative bias cannot be summarily rejected. Less sophisticated models in the class of propensity score corrections show minimal significant effects on selection propensity in the regulatory preference outcome models, but the distortions are quantitatively very tiny.

More Info - PowerPoint | Full Paper