Finding Words for the New Research Reality
By Simon Kooyman

"Better, cheaper, faster" is the slogan that has been almost unanimously adopted by the large international market research groups to describe online research. Online panels have become the new Wild West, with different companies digging and shining up the same "golden" nuggets over and over – in this case, prolific respondents.

But in recent months, more and more clients and researchers have discovered that online research may be "better" in terms of interactivity, video components and convenience, but not necessarily in terms of quality. The variability in outcomes has led to confusion and wrong decisions in more cases than clients care for – and their frustrations are growing. Client confidence has been sapped by counterintuitive data, non-replicable results, and in some cases information that clearly contradicts data from the same survey fielded in other modes.

"We're having tremendous issues moving from concept to launch," P&G's Kim Dedeker was quoted as saying in Ad Age, referring to research studies that have offered contradictory guidance on which concepts hold the best potential for success. Ad Age wrote, "While she was careful not to blame online research or specific vendors, [Dedeker] said the problems boil down to ‘the integrity and methodology.' "

Aligning quality to the decision at hand

What happened to the scientific underpinning of market research, which took more than 30 years to develop and strengthen? With the introduction of the oxymoronic volunteer research panels, representativeness and probability sampling were demoted to the minor league, at least for commercial research – to be replaced with low pricing and "smoke and mirrors". By contrast, in government and academic research, the transition has hardly occurred; that is why I am proud that policy makers and academicians have embraced Knowledge Networks' KnowledgePanel®, which is uniquely bringing scientifically valid research to the online space, through representative sampling and inclusion of non-Internet households.

As the online research industry has matured, we have begun to see a trend toward aligning the value of the commercial business decision with the quality of research. Less expensive data sources are used for research to support lower-end business decisions, and high-quality data are used for the high-value, often strategic decisions. This is consistent with what we know to be the case in other industries, such as oil, steel or paper—the grade of the raw material directly determines the quality of the end-product.

As a result, the "Q" word has been re-introduced into the marketing research dialogue. If you were to scan all the brochures of market research companies circa 2007, I doubt you would find a word (other than research) used more often than quality. It is hard to remember a time when research companies were rushing so intently to wrap themselves in the mantle of "quality"...which, of course, means that that the term has less and less meaning every day.

Quality is just one casualty of the new research marketing environment; a few other words and phrases that we should put on the Endangered Meaning List would be:

• representative: a truly representative sample of consumers is a matter of beforehand sampling; it cannot be approximated or devised after-the-fact – but that is what some researchers have in mind when they say "representative" today
• RDD: When we talk about "random-digit-dial" (RDD) sampling, we usually assume that dialing is based on a computer-generated sample frame; but these days some people use it to mean simply calling random numbers.
• Scientific samples: A sample should refer to a random selection of respondents from a pre-defined population, such that the probability of selecting an individual from that population is known and non-zero; only then is it possible to project results from that sample to the larger group/population.

Part of what we are lacking, it seems to me, is a vocabulary not of words, but of numbers. After all, numbers are supposed to be the stock and trade of research and statistics. But when people claim to offer "high quality" or "unmatched reliability," we rarely see a number value attached to those claims.

Restoring the true metrics for quality

In addition, we need to embrace a complete set of concepts and metrics that addresses research reliability; when we cherry-pick a "metric of the moment" or soundbite of the month, we do a disservice to the complete spectrum of concerns that should be on every researcher's agenda.

Here is the vocabulary that I would like to see included in every research company ad or brochure, with numbers to back up their claims:

• Number of invites per panelist sent per month
• Number of surveys completed by panelists per month (too many or too few can be equally problematic)
• Percent of panelists that self-report belonging to three or more other panels (speaks to the professional respondent problem)
• Panel size defined as "number of unique panelists who completed one or more surveys in the prior 30 days"
• Number of new panelists defined as "number of unique panelists who completed their first survey in the prior 30 days"
• Reliability (a measure of the consistency of results between two identical studies fielded on the same panel but with different respondents)
• Representativeness of the population (a measure of probability sampling)
• The quality of the survey experience--i.e., insuring that interviews are well developed, that branching and filtering is done correctly, and that the respondent is not repeatedly interviewed on similar topics is also critical to survey quality.

This complete set of metrics tells much more of what we need to know about quality – which is why, of course, they are so rarely cited by those who bring volunteer on line samples to market. People who volunteer to take surveys aren't representative, and they introduce bias that can never be reconstructed.

Of course, the underlying virtue that all of the above addresses is transparency. All research is subject to limitations, and responsible research requires complete transparency and disclosure to insure that results are interpreted correctly and conclusions are well-founded. At Knowledge Networks, we consider transparency to be part of the extraordinary quality and service (EQS) that informs everything we do.

It is up to buyers and users of research to demand more than vague superlatives that quickly become devoid of meaning. Consider ESOMAR's "25 Questions to Help Research Buyers"; this document offers an excellent framework for replacing words with concrete information that speaks to quality. (You can view Knowledge Networks' answers to these questions at: www.knowledgenetworks.com/25ques.)

If answering just these questions was a requirement for every research company, words like "quality" could be evaluated with greater certainty. In fact, they might even start to mean something again. They do to me.

Simon Kooyman is CEO of Knowledge Networks.

From 2001 to 2005, Kooyman served as Chairman and Chief Executive Officer of Ipsos North America and Chairman of Ipsos-ASI worldwide. While at Ipsos, Kooyman led the company's worldwide efforts to move market research to the Internet, including the development of a leading North-American Internet panel.