Study Finds Probability-Based Surveys To Be Most Accurate
The research industry has raised doubts about the accuracy of non-probability based online surveys since their inception, even while their use has grown. A new peer-reviewed study by Stanford University researchers, Professor Jon Krosnick, David Yeager, and their co-authors, provides an extensive comparison of seven non-probability Internet survey panels with two other survey platforms based instead on probability sampling. One is based on KnowledgePanel® from Knowledge Networks, and the other probability-based solution uses random-digit-dialing telephone methodology. The authors concluded that the non-probability Internet surveys were less accurate, and customary weighting adjustments did not uniformly improve them.
Knowledge Networks provides you access to the study and the supplement via the links below.
- Report on the Study - .pdf
- Online Supplement - .pdf.
Prof. Krosnick presented these findings at a KN breakfast forum in the Boston area; below is the full video and selected clips.
Jon Krosnick Forum Presentation – Full Video (39:11)
Windows Streaming Video
Presentation Clips - Windows Streaming Video
- Moving from Telephone to Internet (00:41)
- Indicator of validity #1: LinChiat Chang's experiment: Mode comparison with sample held constant/latent variable analysis (02:29)
- Indicator of validity #2: Systematic measurement error: Response order effects (01:53)
- RDD recruitment with Internet survey administration: Description of new analysis (03:19)
- Weigh constructs and why they matter/unweighted comparisons among panel types (03:17)
- Internet survey accuracy/response rates (01:51)
- Assessing validity (Telephone vs. Internet results) (00:43)
- Estimating reliability via latent variable structural equation modeling (01:39)
- Internet data collection: Probability versus opt-in sample (results) (07:15)
- Has anything improved since 2004? (01:02)
Krosnick/Yeager/Javitz respond to questions about their study – ABCNews.com
Gary Langer, ABC News polling expert, has given over his most recent blog for Krosnick, Yeager and Javitz to answer questions about their much-discussed new paper. They give clear answers that shed additional light on the shortcomings and proper uses of opt-in polling. Click here to read what they had to say.
