The reliability of online research is a business issue that marketers must confront

When marketers discuss the metrics of ROI, their focus is often on how—how can new measurements establish a more direct link between marketing efforts and sales? What methodologies and services will help make those metrics a regular part of the information at decision makers' fingertips?

Just as imortant as how ROI may be measured, however, is how well those measurements are executed, from data source to analytics. Decisions of every kind are only as good as the information on which they are based; any focus on finding new ways to measure must be tempered by attention to the validity of the information source, which will affect the accuracy of the input you use in decision making.

The Internet has put a wealth of information—and a powerful tool for custom research—within easy reach of marketers. It can provide quick feedback at a low price, with the ability to target narrow interest groups at will. Online research has quickly been adopted for decisions large and small, from product opportunity assessment to advertising testing; it figures prominently as t he data source in many of t he ROI measurements that hold promise for the future.

But Internet research brings with it a variety of concerns that the MR community must address now that the "sense and respond" model of marketing has overtaken the old "make and sell" approach. This sense and respond model is about anticipating customer needs and swiftly responding with the appropriate capabilities and customer offerings. To execute this leap to the information age's use of consumer-centric marketing, we must cast an eye toward knowing if the public being queried and relied on to inform product offerings and marketing actions is at all representative of market behavior.

Building large online panels effectively requires asking questions about the engagement of those respondents and the representativeness of their opinions relative to known benchmarks. As marketers and policymakers seek to apply this data judiciously, the connection between research accuracy and return on investment comes into clearer focus; an insight cannot be acted on wisely unless its accuracy is understood.

A recent Stanford University study is among the first to compare top online consumer panels to accepted benchmarks— data from the census, the Centers for Disease Control (CDC), and other reliable sources, as well as a high-quality telephone survey. The same questions were administered across all vendors, creating an apples-to-apples comparison. Results show that the variability of some Internet research panels is at times large enough to alter outcomes—a nd therefore decisions based on the findings. There also seems to be a direct connection between the research methodology applied and the accuracy of data produced.

Volunteer samples lead to overstatement and variability

The highest-quality data generally came from those who began with a statistically valid sample, typically created via randomdigit dial (RDD) sampling. The greatest variability of results from identical surveys was associated with a "volunteer" recruitment approach, in which panelists "select themselves" rather than being selected.

In terms of absolute error, the Knowledge Networks Panel—the only online panel based on an RDD sample—showed the greatest reliabilit y overall. By contrast, the volunteer panels posted wide-ranging error rates, more than double that of KN. (See Chart 1.)

This pattern held true for a variety of specific metrics that could be essential to understanding market size and the impact of alternative marketing variables or public policies. Examples of the differences include

• Membership in a frequent flyer program: The volunteer online vendors averaged 38 percent saying they are a member—nearly double the benchmark of 18 percent.
• Smoke every day or occasionally: The average for volunteer panels was 30 percent—nearly 50 percent above the CDC benchmark of 22 percent.
• Have current driver's license: The valid KN panel was within a percentage point of the U.S. Census/Statistical Abstract benchmark (89 percent benchmark vs. 89 percent KN), while all other online firms averaged 94 percent.

In addition, the study looked at many data points for which no benchmark exists; if we use the high-quality telephone data as a benchmark in these cases, we see additional proof of the difference between Knowledge Networks and online firms that rely on "volunteer" respondents. For example:

• Volunteer panels overrepresent use of coupons, by a factor of 100 percent in some cases. While the KN (50 percent) and telephone (54 percent) data show that about half of respondents use coupons in a typical week; the volunteer firms came in with a much higher average estimate of 73 percent. For those attempting to measure such a promotion variable as part of a plan to introduce a new product or line extension, the implications are stark. (See Chart 2.)
• Volunteer panels overrepresent those consumers who consider themselves very comfortable with computer technology; while the KN and telephone surveys arrived at statistically identical estimates of 48 percent, the average for volunteer firms was 79 percent who consider themselves at least very comfortable using a computer.

VOLUNTEER PANELS OVERREPRESENT USE OF COUPONS BY A FACTOR OF 100 PERCENT IN SOME CASES.

The Stanford research also shows how low in-panel completion rates dramatically alter the effective panel size of all-volunteer panels. (See Chart 3.) One major vendor posted a 2 percent in-panel rate—meaning that 98 percent of those who received the survey did not complete it. Since higher completion rates are essential to lowering nonresponse bias in results, these findings have important implications for users of online data. Levels of nonresponse are crucial, since those not responding to a survey could reflect a population that the client wants to reach to best understand what to do in the marketplace.

Some of the key findings of the Stanford study have also been aff irmed by other, less extensive, comparisons of valid versus volunteer online panels. For a test of reactions to a new line of facial products, the same survey was fielded on the representative KN panel and a volunteer panel, with dramatic results.

The volunteer estimates of purchase incidence, impressions of product superiority, and product purchase with a coupon were all double or nearly double those from the KN panel. The manufacturer was confident in the KN data that showed consumer acceptance and product performance problems that the volunteer sample apparently masked.

A core business issue for marketers

Over representation of some consumers and bias introduced by low cooperation rates make online research accuracy more than an academic concern. Because it can distort consumer interest in or acceptance.

Patricia Graham is Knowledge Networks' Executive Vice President, Client Service and Business Development. You can reach her at pgraham@knowledgenetworks.com.

Send this article