[ Fall/Winter 2009 ]

New Study from Yeager, Krosnick and Co-authors Advances Understanding of Accuracy, Best Uses of Online Panels

Plus, new Decision Maker's Guide from KN summarizes key issues, findings about online research quality

c-Jannyjus-Dreamstime.com Using online research in a way that best suits your goals means understanding clearly the accuracy of probability versus non-probability research methods and their recommended uses. At KN, we consider it our job to keep you up to date on the latest research on research to inform sound decisions – and we want to refer you to two important new documents on the topic.

One is a much-talked-about new paper from Jon Krosnick, David Yeager, and their co-authors that directly compares probability and non-probability approaches to research – and reaches significant findings which are:

The probability sample surveys done by telephone or the Internet were consistently highly accurate.
The opt-in sample surveys done via the Internet were always less accurate and were sometimes strikingly inaccurate.
Best practices weighting of the opt-in samples sometimes improved their accuracy and sometimes reduced their accuracy but never made them as accurate as the RDD telephone and probability sample Internet surveys.

The basis for these findings is: In 2004-05 and again in 2009, the Krosnick/Yeager team compared results from one probability-based online panel (KnowledgePanel®), one probability-based phone sample and seven opt-in (or volunteer) panels, and found that probability-based approaches to research were consistently more accurate.

You can read the complete paper, and a supplement, at these links:

Report on the Study (.pdf)
Online Supplement (.pdf)

You can also read the authors response to the many questions raised in the industry about the study here.

We are also happy to offer you free access to the 2009-10 edition of The Decision Maker's Guide to Online Research™, which not only incorporates findings from the new Krosnick et al paper, but also provides you with insights and research on research about:

the effect of cell phone-only households on research accuracy
using Address-based Sampling to achieve probabilistic-based research
the special problems of recruiting and maintaining young adults and Spanish-dominant Latinos for online panels
how to calculate a true response rate
the need for transparency when evaluating any form of research
questions you need to ask your online research vendor

To view The Decision Maker's Guide to Online Research™ online, just click here. From that page, you can also download the PDF version – look for the link in the far right column at the top of the page.