Opt-in Panels and Decision Making: Learning to Tell a Cat From a Pig
By Simon Kooyman

The expression "the cat is out of the bag" refers to a trick attempted by desperate livestock farmers in marketplaces. It is related to another term, "a pig in a poke," which describes a pig in a (usually burlap) bag which was sold at market. In unfortunate times, some farmers would attempt to put a cat in a bag and sell it as a pig in a poke, hoping that they could get away before the contents of the poke were discovered.

So "the cat is out of the bag" indicates that someone has been caught knowingly doing something they shouldn't have. And, in my view, this assessment applies to much online research today.

The "cat" is opt-in panel research, which too many MR suppliers and their commercial clients have come to think of as pretty much equivalent to other types of research – plus cheaper and quicker. But gradually accumulating experience and some head-to-head comparisons have shown that today's opt-in data could well be contradicted by tomorrow's – and that is not a good thing for people who rely on it to draw conclusions and make decisions.

This new or growing awareness has led to a variety of efforts in the past six to twelve months – committees formed by almost all of the major research organizations; conferences from Research Business Report and IIR (KN has had a presence at both); and articles in the trade press. In many cases, it seems as if my colleagues are getting their first glimpse of the issues of online panels.

As you probably know, nearly all online research panels are "opt-in" – also known as "volunteer" and "self-selected." This means that consumers can sign themselves up at will, often under false pretenses, and there is no defined "universe" of respondents. No true response rates can be calculated, and the data is not representative of any group but the people who answered. Opt-in also is an invitation to people who would scam the system: "cheaters," "professional respondents," and others.

There is an online alternative – one represented by KnowledgePanel® from Knowledge Networks; in this case, an RDD sample is used to recruit households by phone. The result is a sample representative of the full U.S. population and projectable to other groups. People claim representative sampling is not important anymore; but on the 7th of November next year, no major TV network will make election predictions based on a volunteer panel. I can guarantee you that.

What it's like to be an opt-in panelist

Here is a basic fact of any research – not just online: Faster, cheaper and better is not a realistic goal. You can have faster... possibly faster and cheaper. But will it also be better under those circumstances? Many research firms and their clients have few procedures in place for monitoring data quality; the focus is on "procuring" at the right price. But do the marketers in the C-suite know that they may be making decisions based on dollar-store data of limited reliability?

Here is another basic fact, disguised as an acronym; and this one goes way beyond the confines of research: "GIGO." So what does that mean? No, a gigo is not one thousand mego. Neither is it abbreviated slang for a male prostitute. And, no, it's not a French cut of lamb. GIGO stands for "garbage in, garbage out." DuPont, Ross Labs, Burberry, and Boeing – they all know this simple truism, and they buy very specific, high-quality raw materials for their high-quality and critical products. If those companies were in the research business, and if they knew enough to recognize the differences (that is a big "if," as we will see), they would not be using opt-in research for their most important products.

A number of people I know belong to some of the bigger opt-in panels; and while the experience can be quite different from company to company, there are some themes, such as:

• an almost relentless focus on earning money and rewards; this tacitly endorses the idea of becoming a "professional respondent." One research firm proudly announced, in order to attract respondents, that they paid out more than $4 million in incentives last year
• the tendency to send multiple surveys in a week, and even two a day
• too few controls on the lengths of surveys – 45 minutes is not uncommon, and that is with no special incentive
• questionnaires that tip off the topic of the survey, encouraging people to artificially screen themselves in or out, and
• the willingness to have people take surveys for as long as they will sit still – one person I know of took 27 surveys during a two-hour period, and the research company would have let her keep going if she had not ended it herself.

There are also "consolidator" Web sites that charge you a fee to sign you up for dozens of panels all at once, with the clear intent that you will get the money back (and then some) by taking surveys. This is a long way from what I would consider the research basics.

To me, the metaphor of fishing is apt when it comes to opt-in online research. The universe of those who will volunteer for online surveys is a fraction of the total online audience; and the demand for bodies – for fish – is growing. So the fishing grounds are getting depleted. People constructing questionnaires are not doing as good a job – so panel companies are using lousy nets, throwing back and re-catching the same tired fish all the time. Throw in global warming, and we will need Al Gore to rescue us from extinction.

Guidelines for a true quality system

Throughout my career I have been involved in quality improvement programs -- the so-called QUIPS, qual management systems, the QUAMs, zero defects, "do it right the first time," and so on. Right now, for our industry, there is ISO 20252 – a residue of that whole movement, specifically applied for market research. It is not really a quality management or improvement system or methodology; it is more like a good housekeeping guide.

Let me give you my simple summary of what a quality system is built on – three major tasks or components. Each of these is consistent with a renewed focus on the basics of market research, from methodology to report generation; and each has to be "lived" for a quality system to work.

1. Managing expectations

Please let us be clear about what we can deliver and confirm what our clients are expecting -- and that, hopefully, there is a fit. This is an old-fashioned approach that should never have gone out of style. Know what parameters are required and how you will measure them; what will be considered over- and under-delivery. Many programs will go wrong because of the specs. Let us not promise our clients that we can claim projectability if we do not have a statistically valid sample frame.

2. Impeccable execution and delivery

This is where the good housekeeping rules come into place. Proper training, managed work flows, clear hand-offs, intermediate control, systems – they are often a pain in the neck, but essential for quality and efficiency. This quality phase is well described in ISO 20252, the good housekeeping guide. This is where an independent audit could also fit in very well.

3. Timely and mutual feedback

You cannot improve quality if you do not get feedback; that is a basic fact of any system or business. At KN, we send out client satisfaction surveys after every significant project we field. Were the questions answered? Could we have done better? And, in the long term, how do our results play out in the marketplace? The reason that all of us are here today is because the market is giving us feedback.

Be cautious about your trade-offs

We have seen a change in the roles and attitudes of clients, both researchers and marketers. The research knowledge of many of the newer people in marketing is more limited than in the past. The average shelf life of the marketing executive is getting shorter and shorter, the demand for quick data to support short-term decisions higher and higher. Last week, I visited a CMO of a billion-dollar services marketer, to talk about next year's needs, and he confided to me that he spends about 2% of his time on market research issues. And this is a company spending more than $10 million a year, and one of the products sales success factors is market share measured by the researchers.

The research departments are more and more pushed in the direction of bean counting – keeping the score. The research profession is very much influenced by the procurement folks and financial people, who don't have a clue about what they're buying. And, in the process, we are using up valuable respondent time with the same moronic questions over and over. We are overstressing the respondent. So we are in a perfect storm, and even worse, it is spiraling.

To me, quality and value are deeply inter-related; the quality of a project is directly tied to whether it delivers the answers a client needs and helps guide actions, where appropriate. This requires an end-to-end commitment from both the research company and the client. It is another research basic. We work with clients to make sure that we elicit the questions that really need to be answered – which may or may not have been included in the questionnaire. We make sure the survey is well worded and does not create biased answers. We craft reports that point out the most pertinent information to the client's interests.

Now, the continuum of research and panel quality is not a black-and-white issue. You will not hear me say that the only solution is our KnowledgePanel®. There is an established purpose for different degrees of quality, even with satisficers, professional respondents, and cheaters. There is a lot of research that can be done on volunteer panels – maybe even with people who take surveys for a living. For a simple concept test or price evaluation, a promotion or copy test – the law of the large numbers and faster-cheaper-not-better is workable. But when high-level strategic decisions are at stake, opt-in panels really cannot fill the need for high-quality data. (Remember GIGO.)

Simon Kooyman is CEO of Knowledge Networks.

From 2001 to 2005, he served as Chairman and Chief Executive Officer of Ipsos North America and Chairman of Ipsos-ASI worldwide. While at Ipsos, he led the company's worldwide efforts to move market research to the Internet, including the development of a leading North-American Internet panel.