Accuracy: Missing in Action

By Simon Kooyman

Pity, disappointment, shame. Those three emotional forces best describe how I feel about the ARF FoQ, morphing into QeP v1.0. How is it possible that the most promising of all online research quality initiatives has floated away from its original goals – the examination and improvement of the trustworthiness of online research?

At Knowledge Networks, we have made our peace with the fact that there are market research applications that do not need a probability-based sample. Our participation in the ARF quality initiative therefore was and still is altruistic, driven by a genuine concern; the art and science behind rigorous marketing decision support is slowly replaced by quick and dirty dipstick methods. From that perspective, some may be right in saying that the industry is seriously ill; we were happy to contribute to ARF's initiative for a cure. And putting our money where our mouth is, we were the first to contribute with actual dollars, following through with hundreds of hours throughout the project.

So what do the deliverables of the Foundation's $1 million project look like? (It was not really that much in cash, but it sounds impressive.) The skeptics may have forecasted this two years ago, but it is disappointing to see the predictions realized – that this is just another "guidebook" or "process." The new Good Panel and Project Housekeeping Processes are as obvious (Capacity resource planning) and paternalistic (Evaluation meeting between supplier and client) as they are preachy (The Dedupe Declaration by the supplier).

Actually, the FoQ project is now leading us astray, by producing panel, sample/study and survey-based processes aimed at consistency over accuracy. Let's not mistake the two as synonymous. While the processes are useful, suddenly it is OK to be wrong, with error rates that are doubling, as long as you are consistently wrong. Suddenly it is OK to use shoddy respondent panels that provide results that are consistently far off from known benchmarks, as long as you mix it with other panel sources to gain consistent results. In other words, "The solution to pollution is dilution." We seem to not have learned from the environmental scientists, who know that you cannot avert disaster by lowering the PPM count with just flushing away cadmium, mercury, lead, asbestos or PCBs. Just like bad panelists, you have to get rid of contaminants at the source – not use them at all.

It gets worse. It seems acceptable that online panels are more than 20% off from reality of known benchmarks, as proven by the FoQ project with 100,000+ responses from 17 providers, and many other similar research-on-research projects. Samples from individual panel suppliers may report up to 70% deviation from uncomplicated behavior measurement, such as smoking. And this assumes that the sometimes self-selected samples that were contributed to the non-blinded projects were actually representative for the panel quality of each supplier.

Just like health care reform, we all know that we have a problem. We know that there is a solution, but we cannot get the powerful factions to align their interests and sacrifices. Many times during the ARF ORQC meetings, people voiced their support for a central respondent database (as done with credit cards) that would essentially resolve the industry's quality problem from the source. Now a consortium of the larger market research providers has, independently from ARF's ORQC, started developing such a database. The rest of the industry is blessed with guidelines, self-reporting, and copying from the Category Management text book.

Not knowing the accuracy level of panel sources used will result in bad decisions, which will result in financial losses for clients. We know this will happen again in the near future, and therefore we will be there, with the industry's cleanest and only probability-based online panel, most accurately reflecting the U.S. population. Our clients know that for important business decisions, you cannot afford to use questionable data, even if the CPI is only $5.

Simon Kooyman is Chief Executive Officer of Knowledge Networks