[ Spring 2009 ]

Meeting the Challenge of Cell Phone-Only Households, Young Adults and Minorities

Introducing Address-Based Sampling to KnowledgePanel®

By J. Michael Dennis, SVP, Managing Director, Government & Academic Research; Charles DiSogra, Chief Statistician; and Patricia Graham, CMO

Society is changing in ways that affect online survey research. This manifests itself in an urgent question in survey research about what is called the "sample frame"; that is, who is in a survey and who is not. Generally speaking, your sample can only be as good as the frame is complete. Most specifically, whom can I reach and survey online?

We recognize the challenge and have done something about it, as we did back in 1999 by including non-Internet households in our online sample. Knowledge Networks is once again a pioneer in Web panels by introducing Address-Based Sampling (ABS) to KnowledgePanel®. To meet today's challenge of more comprehensively representing the U.S. population in our online panel, we are adding cell phone-only households and recruiting more young adults, and minorities, including those who are Spanish-speaking-dominant at home.

Through our efforts in successfully conducting an ABS pilot recruitment in 2008, KnowledgePanel will now be based on a dual frame sample recruitment (ABS and RDD landline telephone frames). This dual frame delivers at least 34% more sample coverage of the U.S. adult population than any other online panel available today.

KnowledgePanel® Population Coverage in 2009

1 Occasional internet access from any location. December 2008 RDD survey, Pew Internet and American Life Project.
2 Blumberg SJ, Luke JV. December 17, 2008. Wireless substitution: Early release of estimates from the National Health Interview Survey, January-June 2008. U.S. National Center for Health Statistics.

By this initiative, Knowledge Networks innovates while maintaining our scientific roots in probability-based sampling, further enhancing the accuracy and projectability of KnowledgePanel surveys to the entire national population, as well as subpopulations.

More about why we are introducing Address-Based Sampling
and its advantages to you

When KN started KnowledgePanel® recruitment in 1999, the state of the art in the industry was that probability-based sampling could be cost effectively carried out using a national random-digit dial (RDD) sample frame. RDD at the time allowed access to 96% of the U.S. population. This is no longer the case. We introduce the ABS sample frame to rise to the well-chronicled changes in society and telephony in recent years that have reduced the long-term scientific viability of the RDD sampling methodology: declining respondent cooperation to telephone surveys; do not call lists; call screening, caller-ID devices, and answering machines, dilution of the RDD sample frames as measured by the working telephone number rate; and finally, the emergence of households that no longer can be sampled by RDD – the cell phone-only households (CPOHH).

According to the Center for Disease Control, approximately 21% of U.S. households cannot be contacted through RDD sampling: 18% as a result of CPOHH status and 3% because they have no phone service whatsoever. Among some segments of society, the sample noncoverage is substantial: almost one-third of young adults age 18-24 reside in CPOHHs.

CPOHHs have particular demographic characteristics

After conducting an extensive pilot project in 2008, we made the decision to add an Address-Based Sampling (ABS) frame in response to the growing number of cell phone-only households that are outside of the RDD frame. Before conducting the ABS pilot, we also experimented with supplementing our RDD samples with cell phone samples. However, this approach was not cost effective for you, our clients, and raised a number of other operational, data quality, and liability issues (e.g., calling people's cell phones while they were driving, for example).

Source: Mediamark Research Inc. Survey of the American Consumer (Wave 56)

The key advantage of the ABS sample frame is that it allows sampling of almost all U.S. households – an estimated 97% of U.S. households are "covered" in sampling nomenclature. Regardless of households' telephone status, they can be reached and contacted. Second, our ABS pilot project revealed some other advantages beyond the expected improvement in recruiting adults from CPOHHs as well:

• improved sample representativeness for minority racial and ethnic groups
• improved inclusion of lower educated and low income households
• exclusive inclusion of CPOHHs that have neither a landline telephone nor Internet access (approximately 4% to 6% of U.S. households).

Address-Based Sampling methodology

In terms of methodology, ABS involves probability-based sampling of addresses from the U.S. Postal Service's Delivery Sequence File. Randomly sampled addresses are invited to join KnowledgePanel through a series of mailings and in some cases telephone follow-up calls to non-responders when a telephone number can be matched to the sampled address. Invited households can join the panel by one of several means:

• completing and mailing back a paper form in a postage-paid envelope;
• calling a toll-free hotline maintained by Knowledge Networks; or
• going to a designated KN website and completing the recruitment form.

After initially accepting the invitation to join the panel, respondents are then profiled online answering demographic questions and maintained on the panel using the same procedures established for the RDD-recruited research subjects. Respondents not having an Internet connection are provided a laptop computer and free Internet service. Respondents sampled from the RDD and ABS frames are provided the same privacy terms and confidentiality protections that we have developed over the years and have been reviewed by dozens of Institutional Review Boards.

Large-scale ABS sampling for our KnowledgePanel recruitment will start in April and continue throughout the year. During this time, KnowledgePanel will begin to see marked improvements in sample coverage of CPOHHs and young adults.

Because we will have recruited panelists from two different sample frames – RDD and ABS – we are taking several technical steps to merge samples sourced from these frames. Our approach preserves the representative structure of the overall panel for the selection of individual client study samples. An advantage of mixing ABS frame panel members in any KnowledgePanel sample is a reduction in the variance of the weights. ABS-sourced sample tends to align more truly to the overall population demographic distributions, and thus the associated adjustment weights are somewhat more uniform and less varied. This variance reduction efficaciously attenuates the sample's design effect and confirms a real advantage for study samples drawn from KnowledgePanel with its dual frame construction.

Comparison of Unweighted ABS Pilot Recruitment to National Benchmarks Grouping ABS Pilot (unweighted) U.S. Population (a)
18 to 29	18.6	21.9
30 to 44	30.7	26.9
45 to 59	32.1	27.8
60+	19.2	23.4
White	76.5	81.2
African American	14.6	11.8
Other races	8.9	7.0
Hispanic	9.7	13.7
Not Hispanic	90.3	86.3
Married	54.7	55.6
Not married	45.3	44.4
Home owner	64.4	70.1
Rent/Other	35.6	29.9
Less than high school	12.3	13.6
High school or equivalent	32.8	31.4
Some college	27.6	28.0
Bachelor's degree or beyond	27.3	26.9
Under $24,999	20.5	21.6
$25,000 to $49,999	26.6	26.6
$50,000 to $74,999	17.8	20.1
$75,000 or more	35.1	31.8
Cell phone-only households	14.4	17.7 (b)

a. U.S. Census Bureau, CPS, December 2008
b. NCHS, National Health Interview Survey, 2008

The initial evidence from our 2008 ABS pilot is compelling that the methodology will further improve sample representativeness

Our pilot results have proven that ABS can fulfill the researcher's need for more comprehensive coverage of the U.S. population than RDD can on its own.
Table 1 shows how well the unweighted recruitment corresponds to national benchmarks. In RDD telephone surveys it is more typical to have young adults, lower educated and lower income persons underrepresented. As seen in the table, this was found not to be so with our ABS pilot recruitment. And of course, we recruited CPOHHs as 14.4% of all recruited households (vs. the national benchmark of 17.7%), something not possible to do in an RDD sample.

By definition, a high-quality sample will represent the population frame from which it is drawn. However, if one or more segments of the population are not
available in that frame before you even draw your sample, you end up with something less than representative, and analysis of survey results will suffer. From a quality perspective, this is a problem. Knowledge Networks' innovative solution has been to operationalize a dual-frame sample to provide a higher coverage of the population and meet the challenges of today's societal and technological changes – providing the representative research you, our clients, need.

So for your KnowledgePanel sample, the answer to "Who's there to reach?" will be "Everyone!"

Dr. Charles DiSogra is Knowledge Networks' Chief Statistician and heads the statistics unit at the Menlo Park office. He brings over 20 years experience in survey research, sample design, data analysis, and administration. Dr. DiSogra has a masters degree in public health and a doctorate in nutritional epidemiology with an emphasis in biostatistics and policy analysis from the University of California at Berkeley.

 

Dr. J. Michael Dennis leads Government and Academic Research for Knowledge Networks. Dr. Dennis has managed numerous surveys for academic and Foundation-based customers and for the Research Triangle Institute. A frequent presenter at the annual meeting of the American Association for Public Opinion Research, his current areas of methodological inquiry are nonresponse bias, panel conditioning, and data collection mode effects.

 

Patricia Graham is Chief Marketing Officer and Executive Vice President of Knowledge Networks. Over the past 32 years she has evaluated the marketing plan performance, advertising and communication plan effectiveness of hundreds of new and established brands.

Push pin Photo: © Davidagall I Dreamstime.com