[ Spring 2007 ]

Of Trains, Panel Quality, and Sample Coverage

By Charles DiSogra

The claim of high-quality online research is frequently cited, but upon closer examination rarely earned. This is particularly true for popular opt-in Internet panels, as I will explain in a moment. Knowledge Networks' KnowledgePanel®, which is not an opt-in Internet panel, fulfills that promise of high quality by combining sound sampling methods with the rapid interactivity of Web data-collection techniques. Widely recognized as one-of-a-kind, the online KnowledgePanel® is unique because it uses statistical sampling to select potential panel members from the general population. The important point is that these methods utilize known sampling probabilities. What does that mean? Basically, all U.S. households with landline telephones and the members of those households have a known calculated chance for being selected to join KnowledgePanel®. Knowing this probability information allows KnowledgePanel® samples to be made statistically representative across multiple socio-demographic population characteristics—the same characteristics published and updated monthly by the U.S. Census Bureau. This scientific approach is the same principle used in the highest quality random-digit dial (RDD) telephone surveys. So far so good.

The KnowledgePanel® sampling approach is in sharp contrast to the unscientific method behind the ever-proliferating opt-in Internet panels. The so-called opt-in panels boast impressive numbers of members (in the millions!) made up of volunteers coming from perpetually open or pop-up invitations displayed on the Web. They are not derived from a definable population (loosely perhaps "people on the Internet"?). The opt-in members are not selected using probability methods, and thus their selection probabilities are simply unknowable and cannot be calculated. These opt-in panel members, by definition, self-select to join. In survey parlance, these are aptly referred to as convenience samples, since they are convenient to get. The point is that we don't really know who or what these volunteers represent or how to project any data from them to a larger population. This assessment is widely supported by professional survey researchers. But, some clients still find the opt-in panel option irresistible!

The Great Opt-in Train Robbery

The attractiveness of cheap access to large numbers of online volunteers successfully lures many budget-conscious clients. Most vulnerable are those unable to penetrate the dense statistical garments draped over the opt-in panel methods, and those not sufficiently informed to discern the serious drawbacks of the pseudo-representative nature of opt-in respondents. Maybe this will make it clearer. Let's say you want to measure something in the U.S. population, so you step aboard a crowded cross-country passenger train somewhere in the country. Now go up and down the aisles flashing a sign asking who would like to join a survey panel. Even though many passengers see neither you nor your sign, you are successful in getting some people to say yes. The more train cars you walk through the more and more volunteers you get. Not bad. Now you go back to all these volunteers, get some demographic information and ask them your critical study questions. You gather all the data, and while the train passes through a dark tunnel, you weight and adjust these respondents' characteristics to look like, well, the population of the United States! (Just ignore the possibility that some passengers may be tourists from other countries.) And the answers to those critical study questions are more than suspect, at least. So is this really a research bargain or a proverbial train robbery? Well, you get the picture. Sadly, many who use these online opt-in panels to make important decisions seem to relish the large number of respondents they have and how well their weighted demographic data line up regardless of the sampling problems.

Coverage and Quality Trump Quantity

To paraphrase the kung-fu master of a classic television series, "Which would you choose, grasshopper?" The opt-in "panels of millions" built on convenience or the more compact KnowledgePanel® built on solid scientific methods? Here are some facts. KnowledgePanel® members are recruited using state-of-the-science RDD sampling methods. The panel is designed not just to enlist households for online surveys, but to also have online the households that do not have Internet capability. This is somewhat of a conundrum uniquely resolved by supplying the recruited non-Internet households with online access. Knowledge Networks provides the non-Internet households with MSNTV units and pays their ISP fees. These enabled panel members can now complete online surveys through a television screen at the same time as Internet capable households complete the same survey via their home computers. No other panel does this.

This "broader coverage" solution continues to be the panel's hallmark and an important selling point to clients who are interested in including the "non-Internet households" in online surveys. Although the proportion of non-internet households has been declining over the years, for a variety of reasons it may probably never drop to zero. A recent National Technology Scan study conducted by Dallas-based Park Associates found that 29% of U.S. households are currently without Internet access either by choice or by circumstance. Cost, no service, no computer skills, or just disinterest are some of the main reasons cited. KnowledgePanel® samples include and approximately represent this otherwise inaccessible group for online surveys.

No Shows Can Be a Challenge

Getting those non-Internet households online is one way to get wider coverage of the population and make KnowledgePanel® more inclusive and thus more representative. However, time and "coverage" does not stand still. Technological and social changes have accelerated other trends producing new industry-wide threats to both the sampling approach and the representative picture of RDD samples. Since KnowledgePanel® is dependent on an RDD telephone recruitment survey, these trends are relatively important. Some segments of the population are just not participating in telephone surveys by becoming harder to reach and less cooperative. Young adults (mostly male), urbanites, renters, and disproportionately more ethnic minorities are not showing up in telephone surveys as they did 10 or 15 years ago. Industry-wide, their response rates have been declining over the past decade relative to their proportion in the larger population. In a typical RDD survey, fewer than half the number we would expect are being successfully interviewed. To add insult to this injury, the overall response rates in the larger general population have also been going down although not as rapidly as that for young adults.

On our panel, are we able to recruit and retain young people, African Americans, persons of Hispanic origin, etc.? Absolutely yes! However, the effort to do this is greater and the numbers today are just short of ideal given the high demand for studies to include or even over-sample these groups. KnowledgePanel is currently working to increase the number of members from these groups to anticipate the growing research demands of our clients in 2007. Using appropriate communications, enhanced support, and relevant incentives, we believe the demographic profile of KnowledgePanel® can more than satisfy our clients' research needs.

Staying Ahead of the Curve

Challenges lie ahead in keeping KnowledgePanel® ahead of the curve. One of those challenges is the rapidly growing number of U.S. households that are abandoning landline telephones for cell phone only service. Approximately 12% of households no longer have a traditional landline telephone. Since landline telephone numbers are the basis of KnowledgePanel® sampling frame, a solution that will capture more of these cell phone only households is in the works. Because our panel is a representative picture of the U.S. population, this trend is also mirrored among our panel members. A small but growing percentage has likewise transitioned from landline telephone service to cell phone only status since their initial recruitment on the panel. Wider and growing representation of these households on KnowledgePanel® will once again keep us ahead of the curve for excellent population coverage in survey research.

Dr. Charles DiSogra is Knowledge Networks' Chief Statistician and heads the statistics unit at the Menlo Park office.

He brings over 20 years experience in survey research, sample design, data analysis, and administration. Dr. DiSogra has a masters degree in public health and a doctorate in nutritional epidemiology with an emphasis in biostatistics and policy analysis from the University of California at Berkeley. He can be reached at cdisogra@knowledgenetworks.com.