Opportunity Calling – But Not By Telephone

by Jordon Peugh

Throughout my career, most of my clients have been health-care related academic, foundation, or non-profit organizations - for whom statistical representation and academic rigor are of paramount importance. For my quality-conscious customers, the cornerstones of excellent survey research were clear: representative survey samples, clearly worded survey questions, quality control of the survey process, and timely and appropriate consulting on all aspects of the survey design. A good example is the health policy surveys I conducted as a grantee of the Commonwealth Fund, such as their annual International Health Policy survey series.1 Similarly, rigorous methodology was required for federal agencies in, for example, their evaluation surveys on the health impact of social marketing campaigns These clients started with an assumption that without a clear understanding of who is being surveyed, what questions are being asked, and how respondents are being interviewed, one cannot be confident that the results are accurate and meaningful. 

Until recently, Random Digit Dialing (RDD) telephone interviewing was considered the gold standard among these clients (and my go-to recommendation) whenever they wanted to interview the general population. However, today, the shine has started to wear off the RDD promise.

According to the CDC, approximately 23% of households do not have a landline telephone (20% are cell phone only and 3% have no phone of any kind). So with more than one in five households missing from the sample entirely, it is hard (if not impossible) to say that RDD samples today represent America - especially since cell-phone-only users tend to be younger, are more often male, more likely to be living in or near poverty, and more likely to smoke and binge drink, to name just a few known differences.2 To leave out the cell-phone-only group must by definition skew the results in some way that we can't know or correct for in our analysis.

And if you imagine the unflattering reaction you would get at a cocktail party by discussing market research via telephone, then you know that response rates to RDD surveys are declining (if not plummeting off a cliff) as more and more people refuse to take calls. In my experience, the typical response rate for a commercial market research project is now well under 20%. When was the last time you consented to take a telephone survey? As response rates continue to decline, the costs we pass on to our clients increase (as we have to dial more and more households to reach a single willing respondent).

So what is the alternative to RDD?

• Mail surveys using Addressed Based Samples? I do admit to a soft spot for postal mail studies. They are tangible, personal, non-intrusive, and relatively affordable. But, and this is a big "but", they take ages to complete and do not allow for much if any customization like sub-group screening, interesting skip patterns, experimental questionnaire design, or display of graphics/video. (Try using a mail survey – as I once did – to interview married individuals about their prioritization of 12 different health plan options; the skip patterns alone made our respondents crazy!)
• Online surveys using opt-in panels? For my clients with a focus on representation and publication goals, the non-probability nature of the typical opt-in online panel makes this option unacceptable. I only recommend these kinds of online samples in circumstances where the population of interest is very low incidence or the topic is particularly sensitive (it is indeed challenging to talk to people over the phone about their investable assets, sexual orientation, or potentially illegal activities). But even then, we could often come up with a better alternative than using opt-in online samples (albeit at much greater cost).

But what if you want research that is scientific and representative but also interactive, speedy, and affordable?

From my experience so far, the answer is KnowledgePanel. Knowledge Networks has created a probability based panel that accurately represents the U.S. So, the panel offers the best of all worlds; we have the scientific rigor of proper sampling combined with the speed and nimbleness of online surveys. I have become a quick and enthusiastic convert to this methodology, having lived in the other worlds for some time. And I am excited to have just joined Knowledge Networks as VP of Business Development for the Academic and Government group so I can begin to offer KnowledgePanel to my clients.

One last editorial note: Do I think that RDD-based telephone surveying is dead? Well, probably not yet. But I do think anyone considering such an approach should be prepared to take a hard look at who is being reached (and not reached) and to dig deep into their pockets to improve coverage and effectiveness. Or, instead of trying to twist this "old fashioned" methodology to fit our changed society, consider an alternative that is not ahead of its time, but right on time.

Jordon Peugh is Vice President for Business Development in Government and Academic Research. For more than a decade, Ms. Peugh has conducted specialized research for numerous foundations, academic institutions and corporations in a variety of areas. These include projects on domestic and international health policy, needs and experiences of patients and physicians, adoption of health information technology, impact of social marketing campaigns, and consumer and constituent loyalty. A frequent grantee of the Commonwealth Fund, Ms. Peugh directed the annual International Health Policy (IHP) survey series from 2003 to 2009. She holds a Master’s degree in Sociology from New York University and a Bachelor’s degree in Sociology from Purchase College, State University of New York.

1 Schoen, C., Osborn, R., Doty, M., Bishop, M., Peugh, J. (2008). In Chronic Condition: Experiences of Patients with Complex Healthcare Needs in Eight Countries, 2008.   Health Affairs Web Exclusive, November 13, 2008.

Schoen, C., Osborn, R., Doty, M., Bishop, M., Peugh, J., Murukutla, N. (2007). Toward Higher-Performance Health Systems: Adults' Health Care Experiences In Seven Countries.   Health Affairs Web Exclusive, October 31, 2007.

Schoen, C., Osborn, R., Huynh, P.T., Doty, M., Zapert, K., Peugh, J., Davis, K.  (2005). Taking The Pulse Of Health Care Systems: Experiences Of Patients With Health Problems In Six Countries.  Health Affairs Web Exclusive, November 3, 2005.

2  Blumberg SJ, Luke JV. Wireless substitution: Early release of estimates from the National Health Interview Survey, July-December 2008. National Center for Health Statistics. May 2009. Available from: http://www.cdc.gov/nchs/nhis.htm.