[ The Decision Maker's Guide to Online ResearchTM ]

2009-2010

Table of Contents

1. Gaining confidence in research
2. A multifaceted approach to coverage and representation
3. What you need to calculate a margin of error
4. The digital divide skews research
5. "Suspect" respondents: Defining the significance
6. Transparency allows meaningful assessment
7. Staying ahead of the sampling curve
8. Replicability: A key marker of quality
9. The special challenges of surveying minorities and young adults
10. A patented approach to counteracting biases
11. Why collaboration increases the value of research
12. Observations to decide by

Market research today poses a host of challenges to marketers and researchers: a contracting economy; fluid shopping, purchase and social behaviors; changing attitudes; and systemic uncertainty about businesses of almost every kind. It is becoming harder to know who we are, what we do and why we do it in a world in which marketers and all business people rely so heavily on information.

Such times demand that you get by with less in terms of research—or so some would say. Turning to 'good enough' research to inform marketing actions can introduce uncertainty, in the form of bias and inaccuracies, at a time when marketers can least afford to make mistakes. Now, more than ever, we need to be sure—about sample coverage and representation; the data, analysis, and resulting recommendations we use for decision making; and the research partners who are providing them.

As an example, would you want to find out after the fact that your late-stage new concept test was based on a sample in which over 40% "agree"/"strongly agree" with the statement: "I usually try new products before other people do"? In fact, that is what we found in two volunteer, opt-in web panels when respondents were asked that question.1 (See Table 1) Is that the concept test PI (Purchase Intent) you want to go to the Vice President of Marketing with? Likely not. Note that two probability-based panel sources showed that only about 25% had top two box agreement with this question—in agreement with classic norms for this question.

Gaining confidence in research

Once we have decided that the risk in a business decision is a priority we cannot afford to put aside, how do we come to terms with what research quality is? The word quality has become ubiquitous in market research—conferences, committees, guidelines, and advertising. The desire to strive for certainty is no doubt genuine in most cases; but we need guideposts to understand what quality is. Where are the facts about research accuracy and projectability that could truly guide a decision maker—especially in this down market—to data that serves better decisions?

What we are seeking, it seems, is not simply assertions, but confidence in our online survey research. Quality is about enabling marketers and advertisers to be sure when they are presenting information that will decide the course of a company or product line or ad campaign. Being wrong or inaccurate is even less of an option than it used to be.

To provide at least some of the facts and points of view essential to making good decisions is the goal of this guide. We will provide here a variety of facts and insights, often from Knowledge Networks' own research on research, as well as other sources, to help you establish what type of online research can best suit your needs. This means understanding not just issues of sampling and representativeness, but also the key role that survey design, analysis, and interpretation play in making insights not just precise, but worthwhile and relevant. In all of these areas, it requires rigor, expertise, and a commitment to collaboration to create research that is accurate within your risk tolerance as measured against business value.

A multifaceted approach to coverage and representation

There is a tendency in the research community to focus on one quality issue at a time; this at a cursory level makes it easier to decide which data sources to trust, but usually oversimplifies what a true assessment requires. Not too long ago, "professional respondents" were the primary concern—those respondents are focused on earning as much cash as possible through survey taking; more recently, this has evolved into a "full-court press" on people who belong to more than one online research panel. This is an important element of quality, and one we need to study and understand, but it is only one of many. Often these factors require industry collaboration to get at their effects on accuracy.

So, how sure do you want to be, taken in the context of the reason for your research? The accuracy and projectability of any research hinges on the choice of the study sample. Survey estimates that you want to be projectable must use a scientific basis with a known sample frame to calculate sampling margins of error.

The representativeness of a sample is more fundamentally important to the statistical validity of your research than duplicate respondents; eliminating multipanel respondents from an opt-in sample does not change the potentially biased nature of that sample. Opt-in results drawn from a nonprobability sample cannot be projected to any other population other than the small group that participates in the actual survey; no amount of after-the-fact cleaning can change that. Use of nonprobability samples is a projectability issue, and duplicate respondents are a data hygiene or accuracy issue within the sample. In addition, it is mathematically impossible to calculate sampling margins of error for survey data from the nonprobability web panels.

To be clear, we should define some of our key terms; according to Wikipedia (www.wikipedia.com), probability sampling is a technique "in which the probability of getting any particular sample may be calculated." Nonprobability sampling "does not meet this criterion and should be used with caution." Wikipedia goes on to say:

"Nonprobability sampling techniques cannot be used to infer from the sample to the general population. Any generalizations obtained from a nonprobability sample must be filtered through one's knowledge of the topic being studied. Performing nonprobability sampling is considerably less expensive than doing probability sampling, but the results are of limited value."

In plain English, whenever a sample is chosen without specifying the target population and without knowing the probabilities of selecting a given respondent, it is a nonprobability sample. The challenge with these types of samples is that there is absolutely no way to know how well (or how badly) such a sample represents the views of the total population.

Surveys from Internet river samples—"pop-up polls" or "tell us your opinion" boxes on websites—are a prevalent example. In these cases, the survey provider from which you are buying research has little certainty about who is responding to the question without asking further questions. The "sample" is polluted, consisting of people who happened to be on that website and decided to answer the questions. There is no way to know who answered and who skipped, and no way to determine what the "total population" would be. Mall intercept surveys, reality-show voting for a winner, and call-in polls are other examples.

Among other things, the opt-in approach by definition leaves out—in unpredictable ways—the 27% of the U.S. population that does not use the Internet or email2... not to mention the 68% of Spanish-dominant Hispanics and 41% of African Americans who do not use the Internet.3

The lure of cheaper data I n a time of economic troubles should not lead buyers to overlook the bigg er picture.

What you need to calculate a margin of error

To determine a margin of error, one must have a known sampling frame with a known and nonzero probability of respondent selection. Note that nonprobability samples, such as volunteer panels and river samples, do not meet these criteria, nor do samples for which people volunteer. The key advantage to a probability-based sample is that one can calculate how likely it is that the findings from the sample accurately represent the full population. That is, we can calculate the margin of sampling error, which is basically the "price we pay" for not interviewing every member of the population.

Thus, as AAPOR states,

"It is impossible to calculate the size of a poll's margin of sampling error with a nonprobability sample. Of note, this statistical fact does not stop some pollsters from calculating sampling error with a nonprobability sample—it just makes their calculations meaningless. Nonprobability samples are useful in the early stages of research or when a pollster needs to gain an ‘impression' of the preferences and attitudes of a target population but does not need to be very confident about how well the poll generalizes to the target population."4

Opt-in online panels are nonprobability based and have introduced new elements of uncertainty and bias into the research process; the lure of cheaper data sources in a time of economic troubles should not lead buyers to overlook the bigger picture of how much trust they should invest in opt-in data when the need is to be sure.

When BusinessWeek wrote an overview of the online research market last year5, it reported on the opt-in methodology as a significant problem. "Not all online polls are created equal," the article said. "The results aren't always accurate and sometimes miss by a mile."

When marketers find they are making decisions based on poorly conceived samples or badly executed online surveys, or if they believe that differences can be weighted away, then the marketplace needs to take notice. The research industry has raised doubts about the accuracy of nonprobability-based online surveys since their inception, even while their use has grown. A new study by Stanford University researchers Professor Jon Krosnick, David Yeager, and their co-authors5 provides an extensive comparison of seven nonprobability Internet survey panels with two other survey platforms based instead on probability sampling.

One is based on random-digit dial telephone methodology, and the other probability-based solution uses KnowledgePanel.

The authors concluded that the nonprobability Internet surveys were less accurate, and customary weighting adjustments did not uniformly improve them. They state in "Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples, August 2009" that "the foundations of statistical sampling theory are sustained by actual data in practice. Probability samples, even ones without especially high response rates, yield quite accurate results. In contrast, nonprobability samples are not as accurate, and are sometimes strikingly inaccurate."

The study also compared the answers from each sample type to benchmarks and found that "with post-stratification... the probability samples were more obviously superior: their estimates were significantly different from 31% and 38% of the benchmarks, respectively, whereas the non-probability sample Internet surveys were significantly different from between 69% and 77% of the benchmarks."

We have seen a variety of other evidence that the conventional online research demands scrutiny:

• Optimus from Peanut Labs, which captures duplicate and other "suspect" respondent behavior, found that, across 127 projects on eight industry panels, aggregate levels of "suspect" participants were anywhere from about 7% to more than 30% from panel to panel6
• A recent Knowledge Networks comparison7 showed that controlling for various demographic variables through weighting did not eradi- cate the differences between Internet and non-Internet households

The digital divide skews research

In an April 2009 Wall Street Journal article8, Reg Baker of the American Association of Public Opinion Researchers (AAPOR) said, "One of the things you consistently find is that online panels have lots of people who spend lots and lots of time online." A KN comparison of Internet/non-Internet households9 and opt-in/non-opt-in answers on a variety of topics in 2008 showed that:

• Members of two opt-in panels were more than twice as likely to say they spend 10+ hours a week on the Internet at home (see Table 2), compared to members of KnowledgePanel® and a custom

• Over 20% of opt-in online survey respondents say they use Facebook, when published usage within the U.S. adult population at the time was 13%. KN's KnowledgePanel®—the only online panel based on a representative sample of the U.S. population—showed a level of 13%, and the RDD-based ANES panel reported 15%
• Roughly 25% of the survey takers in opt-in samples said they had taken 20 or more surveys in the past four weeks. (KnowledgePanel reported 2% did; ANES, 0.3%)—and this excluded the survey from which we obtained this information
• 67% of Internet households report having recycled paper in the past 12 months, compared to 49% of non-Internet households. Yet, according to the EPA10, only about one-third of "MSW," a.k.a. trash, is recovered through recycling

Another paper11 drawing on the same data comparisons also found that levels for "first to try new products" were abnormally high in nonprobability panels, when compared to findings from Francis S. Bourne's seminal essay "The Adoption Process"12. The 16% share of early adopters in these volunteer panels is "higher than assumed in any known adoption-curve metrics," says the paper. The far lower levels in the probability panels, by contrast, were "more realistic" and "supportable" (See endnote 11.)

Overall, the comparisons showed that Internet/non-Internet differences are marked by unpredictability as often as predictability, with opt-in results frequently overstated. This tells us that attempting to correct for such differences with weighting or other efforts is impossible at worst and difficult at best.

The fact that a statistically valid approach does yield substantially better results has also been demonstrated by comparisons of KnowledgePanel® results to known benchmarks, as well as other methodologies. In the interest of transparency, KN has used General Social Survey (GSS) data13 to establish our respondents' likelihood to choose "Don't know" or "Too little" in a survey (see Tables 3 and 4).

Knowledge Networks research has also shown that, relative to responses from a probability-based recruited panel, opt-in panel respondents were more than twice as likely to report top 2 box agreement with "I like to tell others from new brands or technology".14

Further, in every case, the KnowledgePanel results were closest to the results obtained from the high-quality and rarely surveyed ANES panel (65% of the ANES panel had taken no surveys in the past 4 weeks, vs. 8% of the opt-in panel respondents and 17% of the KnowledgePanel respondents).

All of the above differences point to the diverse, unpredictable ways in which opt-in panelists are dramatically different from statistically valid samples of the general population, such as those represented by the ANES panel and KnowledgePanel. Decisions based on conventional opt-in data will likely not hold up in the "real" consumer or public policy world because it tends to unpredictably overstate some behaviors and attitudes. When the incidence of key personal characteristics becomes random, then the significance of survey data from those respondents is unknown and possibly quite low.

The fact is that, for all of the loudly expressed concern about quality, no opt-in research firm that we know of has fundamentally changed its sampling frame practices for the sake of accuracy. Perhaps the strongest evidence we have that results are being called into question is the simple fact that so many buyers and sellers have joined in the efforts to address research accuracy; one has to infer that they have either been burned by flawed online data or heard about someone who has.

"Suspect" respondents: Defining the significance

Recent research by Peanut Labs,15 through its Optimus technology, has shown that 15% of more than 21 million panel respondents are "suspect"—duplicates (1.5 million, or 7%—have taken the same survey before or been screened out of it), "speedsters" (268,000-plus, or 1.3% — completed the survey in less than the minimum estimated time), or "geographically questionable" (539,000-plus, or 2.6%—not connecting from the same region where they claimed to be located). In addition, an analysis by Peanut Labs of some 230 projects done for Penn, Schoen & Berland16 on eight different panels found an overall duplication rate of 4%, but with variations from survey to survey of 2% to 35%; KnowledgePanel's duplication rate is 1%.

In keeping with our transparency, here is how KN's KnowledgePanel looks on a comprehensive basis. Recent data on KnowledgePanel revealed 6% suspect respondents—including 1% duplicates as mentioned above and less than 1% speedsters and Geo-IP violators. In addition, a head-to-head comparison [see endnote 9] of KnowledgePanel and two "opt-in" panels found that fully 25% of those in both opt-ins had taken 20 or more surveys in the past month, compared to just 2% of KnowledgePanel. (See Table 5.)

In a promising effort, industry players of all sizes have created the initiative known as OpenSample, whose mission is to improve quality of online surveys by industry-wide sharing of information about respondents' historic survey participation, while protecting PII (Personally Identifiable Information). OpenSample is proposing the use of encrypted email addresses, machine digital fingerprinting, member/respondent ID, and other emerging methods to create a database of respondent addresses from panel and sample companies of all sizes; the data would be managed by an unbiased third party and allow identification of duplicate respondents in real time. These efforts at accountability, in cooperation with other industry initiatives, are indeed important and will help fill the information vacuum about not just the facts of duplication—which cannot be accurately arrived at otherwise—but also, through analysis, about its effect on survey outcome.

One important recent finding17, reported at the 2009 AAPOR conference, shows that KnowledgePanel tenure is not a major factor in survey answers. The study compared responses of longitudinal and cross-sectional samples to several survey questions—while controlling for other variables—in KnowledgePanel studies for AP-Yahoo! around the 2008 elections. We also tested for an increase in the stability of attitudes of the same respondents over time. Only three out of 14 logistic regression models showed a statistically significant effect of panel tenure on survey responses. This outcome is reassuring, because a fast-paced longitudinal study design, conducted on a Web panel, might be considered a "worst case scenario" for potential conditioning effects.

Transparency allows meaningful assessment

Perhaps the most important sign of worthwhile sample and data assessment is transparency. To be transparent, research companies need to provide information that

• can be directly compared, on essential metrics, between com- petitor research companies
• speaks to the true accuracy of research—such as response rates using a probabilistic sample, as opposed to "completion rates" among those who simply received a survey

One important (but not always essential) enhancement of transparency is the presence of a third party that can collect comparable information from competitors and be sure it is truthfully assessed. But transparency can also be achieved without a third party, through comparisons that do not cut corners. Providing a technical appendix on reports gives a starting point for transparent assessments and comparisons; the appendix should cover such key points as

1. What is the sampling frame and is it statistically valid?
2. How are respondents recruited?
3. What percentage of panelists who receive a survey actually complete it?
4. What percentage of panelists belong to 3+ other panels?
5. What is the sample design?
6. What are the weighting criteria?
7. Explain the analytic methodology used for the research
8. Describe steps taken to ensure data quality, with specific data-quality metrics
9. What are the response rates?
10. What are the sample sizes?
11. What incentives were used, if any?

Staying ahead of the sampling curve

Research from the Pew Internet and American Life Project indicates that about 20% of the U.S. population now has no landline telephone access at home18 — hence, RDD (random-digit dial) telephone sampling is becoming unworkable for statistically valid research on a general population basis. Researchers trying to maintain true statistical validity will likely have to migrate from RDD to alternatives for sampling, or use a mix of sample frames — as Knowledge Networks has in its pioneering use of a dual-frame sample to recruit new KnowledgePanel members. By incorporating Address-Based Sampling (ABS) along with the RDD telephone frame, Knowledge Networks can achieve a more representative sample of cell phone-only households, ethnic groups and young adults.19

It should also be remembered, however, that the biases created by volunteer (opt-in) online recruitment are dramatically more significant than those from the current exclusion (or underrepresentation) of cell phone-only households in gen pop samples (when samples are mostly based on RDD methods). Aside from the essential fact that it is not known who exactly the self-selected opt-in panel members represent, opt-in panel members do tend to be younger, more technology and Internet savvy, 100% with Internet access (mostly broadband), more likely to be members of other volunteer panels, and early adopters — among other differentiating characteristics. RDD gen pop samples are most importantly probability-based samples with recognized underrepresentation among young adults, lesser educated and minority individuals and by definition limited to landline-only households. All these underrepresented RDD demographics currently have solutions with standard weighting procedures that minimize the bias from that mostly missing portion of cell-only households. Yet this weighting does not eliminate the bias that is inherent in any nonprobabilistic sample.

Replicability: A key marker of quality

One result of biased samples is nonreplicable results for the same survey fielded with the same survey design from period to period, no more than several weeks apart. If the same sample criteria produce significantly different results for the same questions within a short timeframe, we need to wonder if our sample and/or the questionnaire is reliable. When P&G first began to talk openly about its concerns about online access panels20, it was responding in part to instances when it tried to replicate concept test results from the same panel, and instead wound up with divergent findings that would have led to different business decisions. No company should be left with uncertainty about answers and actions when decisions need to be made based upon the research.

Knowledge Networks applies replicability tests to KnowledgePanel; it ran the same multiple concept test surveys on KnowledgePanel two weeks apart. (See Table 6.) Across six concepts, with many questions including PI, the results replicated at a 95% confidence level for nearly all questions.21

The special challenges of surveying minorities and young adults

Few populations have been more in demand for online survey research in recent years than Hispanics, African Americans, and people under age 35 (Millennials and teenagers). But as their importance to public policy and marketing decisions has grown, the difficulty of reaching these groups through accurate, statistically representative methods has also increased. The likelihood that a young person or minority will drop out of a research panel is significantly higher (see Chart 2) than the general population.22

In general, these groups are either well ahead (young people) or somewhat behind (unacculturated Hispanics) the general population in terms of technology adoption. A recent study from the Pew Hispanic Center and the Pew Internet & American Life Project23 showed that 32% of Spanish-dominant Hispanic respondents use the Internet "at least occasionally"—less than half the percentages for Hispanic bilingual (76%) and Hispanic English-dominant (78%) respondents. This means that, when it comes to reaching Spanish-speaking Latino consumers, most online panels, which require that respondents already have Internet access, are missing somewhere between 68% (Spanish-dominant) and 22% (English-dominant) of the population from the get-go. The likelihood of these online panels accurately representing the entire Hispanic population, as well as the Spanish-speaking Latino community in the U.S., is low indeed.

Similarly, in recruiting for its KnowledgePanel LatinoSM, KN has found that roughly 55% of Spanish-dominant (at home) consumers we contact need a laptop and Internet connection to participate in any panel. (Giving Internet access along with a laptop to non-Internet persons in our sample allows KN to address the coverage bias resident in all other online samples.)

Pew also found that the main reason Hispanics cite for not using the Internet or email is much more likely to be lack of access than lack of interest.24 This means that offline Hispanics cannot be said to be technologically indifferent; in fact, they are far from it. (See Table 7.) And among young adults, their high levels of acceptance for technological activities like social networking and text messaging do not necessarily make them easier to reach through Internet surveys. Inured to spam and already overtargeted for marketing and advertising, those between 13 and 34 are often highly suspicious of the idea of survey research. Getting their attention on an ongoing basis is difficult; special efforts to recruit and maintain them are required.

A patented approach to counteracting biases

All considered, a top concern for panel companies is to minimize the respondent fatigue created by over-surveying these groups. In the case of KnowledgePanel, attention to this effort helps keep panel sample as representative as possible. Knowledge Networks has patented an approach to load balance all survey invitations, thereby providing a safeguard by scientifically correcting for biases created as a result of previous sample draws. But there are other factors crucial to conducting high-quality research among Hispanics. The interconnectedness of recruitment and retention factors does change your sample composition and, as a result, alter the effective minority sample that is available for your online research. Patterns and context factors exist; research companies need to find them and act on them to change the retention of minorities in online panels, as well as the initial attraction.

Why collaboration increases the value of research

Statistical accuracy, sample representation and survey design cannot be separated from research quality. It is impossible to have good insights and recommendations based on data that can be questioned; but scientific approaches alone do not guarantee that research will be valuable. To deliver survey results and analysis that make a difference to business and policy decisions requires a broader process that includes

• defining goals
• designing research that directly addresses key questions
• executing studies with highest accuracy
• interpreting findings so that they lead to real-world actions

All of these demand collaboration between research company and client—an art form that is being threatened by the trend toward cheaper, lower-quality online research. The low-quality variety is usually executed with little interaction and little attention to survey design or to analytic interpretation expertise post-survey. This approach in many ways can miss the bigger picture of why we do research in the first place—to capture opportunities accurately. Creating a well-designed questionnaire for the Web environment, for example, requires knowledge and experience often not available from today's research vendors. A recent experiment25 conducted by Knowledge Networks showed that simply changing the size of the text box available for open-ended responses can dramatically affect results. (Larger text boxes increased the number of characters offered by one-quarter to one-third.)

Similarly, Knowledge Networks recently conducted an experiment using items on a five-point satisfaction scale. Results presented at the 2008 conference of the Mid-Atlantic Association of Public Opinion researchers (MAPOR)26 show that reversing the order of answer choices makes a significant difference—also, that primacy effects for rating scales should be of concern for survey researchers.

Simply figuring out what questions to ask is another process that will contribute directly to the usefulness of any answers obtained. What is the value of an accurate survey that is not relevant to the client's true goal? When analytics can be added to provide clarity around data and decisions, it is up to the research firm to point out these opportunities. And, once all of the results have been obtained, a final marker of research "quality" is whether the findings lead to real-world actions—a step that researchers should take responsibility for, rather than leave as the client's job.

There should be no real difference between the rigor that produces representative samples and the diligence a researcher employs when he or she makes sure that a client gets the right answers for the right challenge. All work to a single end—heightening the value of research to its user.

Observations to decide by

At a time when the socio-economics of the world are in flux, and when every decision therefore carries greater weight, it is more important than ever to be sure you can trust your research results. If a certain data set leads you to spend marketing dollars unwisely and sets back important business goals and share price, then neither time nor money has been saved. It is important to make these kinds of assessments long before the first survey has been sent to a respondent.

One key element of marketing wisely is targeting precisely, and precision cannot be achieved when samples are biased from the start. Marketing to "the average" is not productive in most cases.

At the risk of oversimplifying, we would offer four observations that provide a measure of guidance, if not infallible answers, for decision makers:

• Think clearly about what decisions will be based on the data
  To shed light on the value of accuracy and quality in a particular project, think about what is at stake in the decisions you will be mak- ing. Using accurate data is one of the few ways we have to control the safety of key decisions; rather than assuming the best ("it seemed to work before; it will probably work again"), it is important to make a searching inventory of the potential risks, especially when there is less room for error in everyone's marketing and research efforts.
• Recognize the hierarchy of data quality issues
  Research quality and value are never about a single issue or set of issues; while "hyperactive respondents" are important to under-stand, they represent a small portion of the factors that might affect your research and decision making. Failing to begin with a statisti-cally valid research approach introduces dramatic biases that can not be corrected later on; while this problem is much more difficult to address than duplicate respondents, it cannot be ignored for that reason.
• Seek out transparency
  Transparency is the opposite of hype, and probably your best indicator of trustworthiness. Transparency happens when a company is willing to offer an even-handed account of its own performance, either in isolation or in a comparative set. The effect of transparency is assurance; the effect of hype is to inspire you to seek out transparency. You will know when you have found it; don't lose track of it when you do.
• Recognize expertise and collaboration as quality issues
  Even the best sample or respondent pool cannot assure that research results will be valuable and salient. Failing to identify the key issues at stake, asking the wrong questions, and interpreting data incorrectly all can cripple the value of an accurate survey. Research companies that can and will collaborate and bring expertise about your business issues to any project are becoming rare; but they lend quality, in the form of value, to research.

Endnotes

1. Dennis, J. Michael, Larry Osborn, and Karen Semans. "Comparison Study: Early Adopter Attitudes and Online Behavior in Probability and Non-Probability Web Panels." Accuracy's Impact on Research (online newsletter), Spring 2009.
2. Fox, Susanna, and Jessica Vitak (Pew Internet and American Life Project). "Degrees of Access." Presentation at Ovarian Cancer National Alliance, July 2008.
3. Campanelli, Melissa. "32% of Spanish-Dominant Latinos Go Online: Pew." Direct Marketing News, March 20, 2007.
4. American Association for Public Opinion Research, "Standard Definitions." Revised 2008.
5. Krosnick, Jon A., David S. Yeager, LinChiat Chang, Harold S. Javitz, Matthew S. Levindusky, Alberto Simpser, and Rui Wang. "Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples." August 2009. Unpublished paper.
6. Chadwick, Simon, and Ali Moiz. "Eliminating Suspect Respondents Using Digital Fingerprinting." Presentation at 2008 ARF Annual Convention.
7. Zhang, Chan, Mario Callegaro, and Melanie Thomas. "More Than the Digital Divide?: Investigating the Differences between Internet and Non-Internet Users on Attitudes and Behaviors." Presentation at the MAPOR 2008 Annual Conference.
8. Bialik, Carl. "Which Is Epidemic – Sexting or Worrying About It? Cyberpolls, Relying on Skewed Samples of Techno-Teens, Aren't Always Worth the Paper They're Not Printed On." The Wall Street Journal, April 8, 2009; page A9.
9. Callegaro, Mario, and Tom Wells. "Is the Digital Divide Still Closing?: New Evidence Points to Skewed Online Results Absent Non-Internet House holds." Accuracy's Impact on Research (online newsletter), Summer 2008.
10. United States Environmental Protection Agency. Municipal Solid Waste in the U.S.: 2007 Facts and Figures.
11. Dennis, J. Michael, Larry Osborn, and Karen Semans. "Comparison Study of Early Adopter Attitudes and Online Behavior in Probability and Non- Probability Web Panels." 2009.
12. Bourne, Francis S. "The Adoption Process," reprinted in Michael J. Baker (ed.), Marketing: Critical Perspectives on Business and Management. Routledge, 2001.
13. Dennis, J. Michael, Rick Li, and Joe Hadfield. "Results of a Within-Panel Survey Experiment of Data Collection Mode Effects Using the General Social Survey's National Priority Battery." Presentation at the AAPOR 62nd Annual Conference. 2007.
14. Same as 11.
15. Baker, Reg, Simon Chadwick, and Joel Rubinson. "Navigating the Research Quality Maze: A Complete Review of What Is Available Today." Peanut Labs Webinar, 2009.
16. Morgan, Alison. "Improving Online Data Quality: The Myth and the Reality." Peanut Labs Webinar, 2008.
17. Kruse, Yelena, Mario Callegaro, J. Michael Dennis, Stefan Subias, Michael Lawrence, Charles A. DiSogra, and Trevor Tompson. "Panel Conditioning and Attrition in the AP-Yahoo! News Election Panel Study." Presentation at the AAPOR 64th Annual Conference. 2009.
18. Pew Internet and American Life, "The Mobile Difference." (Based on data from Oct.—Dec. 2008; published March 2009).
19. DiSogra, Charles, J. Michael Dennis, and Patricia Graham. "Meeting the Challenge of Cell Phone-Only Households, Young Adults and Minorities: Introducing Address-Based Sampling to KnowledgePanel®." Accuracy's Impact on Research (online newsletter), Spring 2009.
20. Dedeker, Kim "Research Quality: The Next MR Industry Challenge." Research Business Report, Oct./Nov. 2006.
21. Knowledge Networks test/retest for a major manufacturer, 2006. Unpublished.
22. Graham, Patricia. "The Attraction & Retention Paradox of Ethnic Groups in Online Panels." Presentation at The Research Industry Summit on Data Quality, November 2008.
23. "Latinos Online," Pew Hispanic Center; answer cited based on data from August to October 2006.
24. Same as 23.
25. Callegaro, Mario. "Web Questionnaires: Tested Approaches from Knowledge Networks for the Online World." Accuracy's Impact on Research (online newsletter), Spring 2008.
26. Tang, Ge, Mario Callegaro, Tom Wells, and Yelena Kruse. "Response Options Order Effect and Category Number Association: An Experiment Using Items on a Five Point Satisfaction Scale in a KnowledgePanel® Survey." Presentation at the 2008 MAPOR Conference.