POV: Market Research in the First Person

Online Surveys for Federal Agencies: The Future Is Now

By Joe Garrett, VP

My survey research background started in the public sector, where I spent nearly half of my thirty-year career at the Census Bureau, eventually as Assistant Chief for Research and Methodology. In 1990, I left the Census Bureau to become VP and Chief Statistical Officer at Nielsen Marketing Research. These positions sharpened my thinking about the ever-present balancing act between survey quality and survey costs for large-scale research. The recent development of online survey technology has also produced considerable cost versus quality ramifications. For example, methods for collecting data via the Web emerged rapidly over the past decade; survey statisticians now consider online data collection when addressing some of the cost/quality tradeoffs – especially in the private sector. While the private sector has been quick to embrace online surveys, the Federal sector has been much slower. Now this appears to be about to change.

Many Federal surveys still cling to the traditional modes of data collection; interviewing in-person or by telephone. One reason for clinging to these is a belief that Web-based surveys are not "scientific." Although it's true that all other approaches to online data collection involve non-scientific methods that use non-random opt-in/volunteer panels, Knowledge Networks' KnowledgePanel® is based on scientific probability sampling principles and should be a useful fit for many Federal surveys. Other advantages of Web-based surveys include lower costs and faster data delivery. Looking to the future, these attractive advantages make online data collection difficult to ignore, especially given the fiscal pressures that are likely to be with the Federal government for a long period. I predict that the public sector will soon start to view Web-based data collection as both a valuable and a valid survey methodology.

Predicting change is not easy. Switching to online surveys, even in the private sector, did not occur overnight. I can remember predicting changes that would occur in survey research in a talk I gave to the American Association of Public Opinion Research's 1996 annual meeting. The topic was "The Future of Survey Research," and I was one of four invited plenary speakers challenged to predict where survey research would be twenty years hence. Click here to read Joe's 1996 predictions!

Some of these predictions are on track, especially in the private sector. But, what about the public sector's future use of online data collection methods? As stated previously, I firmly believe that Federal statistical agencies will move more data collection to online modes over the next several years. This is significant: the Federal statistical system will spend nearly 8 billion dollars in 2009 on survey-related procurements. Let me explain why the public sector will start using online surveys.

Economic conditions

First, the U.S. government debt is nearly ten trillion dollars. Moreover, this figure does not include unfunded obligations into the future for Medicare, Medicaid, and Social Security (nor the recent Federal bail-out packages). Economic conditions alone will force many agencies to explore more cost-effective ways of collecting data. The cost to conduct traditional surveys continues to escalate, but research budgets do not keep pace. Based on their unique importance for Federal policy and planning purposes, a few large-scale Federal surveys will continue to be conducted with traditional methods, but a large number of others will use new methods for data collection that are consistent with the new fiscal reality.

Data collection challenges

Second, traditional modes of data collection continue to experience cost and operational difficulties. The future of using the telephone as a primary mode for collecting survey data looks bleak. Not only are the costs high, but there are operational issues that will only get worse. These include declining use of landline phones in the United States; proliferation of cell-phone-only households (CPOHHs); enhanced use of call screening for ignoring unwanted calls; and a public that is tired of receiving telephone calls from survey organizations at dinner time. These difficulties likely will not be resolved, hastening the use of online data collection techniques in the public sector. Knowledge Networks is poised to take advantage of this situation by shifting its recruiting from a random-digit-dialing (RDD) approach to an address-based sample (ABS) methodology that eliminates the CPOHH problem.

The need for new "fitness for use" criteria

Third, to deal with the "survey cost versus survey quality" tradeoff, Federal agencies as well as the White House's Office for Management and Budget (OMB) – which reviews and approves all Federal data collections – should develop "fitness for use" criteria to aid discussion on when such trade-offs should be made. Such criteria will recognize the fact that all survey estimates are imperfect and that survey costs (think "cost per interview") vary widely. The "fitness for use" challenge is to determine when survey quality is "good enough" and at a certain price point to meet or surpass the research needs at hand. This clarity helps to resolve instances where high-quality online methods can be viewed as tenable replacements for more costly traditional approaches. For example, Knowledge Networks has had several of its data collection efforts approved by OMB, and this "fitness for use" approach worked well.

Flexibility intrinsic to Web-based surveys

Finally, the Federal sector can no longer afford to ignore the several advantages and statistical innovations inherent in scientific, online data collection. For example, the Internet offers many advances to survey instrumentation such as embedding video or other links in the survey instrument, or offering complex choice-type questions that are too difficult with telephone interviewing. Plus, unlike annoying telephone surveys that request "just 20 minutes of your time," online methods allow the respondent to choose when to respond within a window of time. Moreover, as more of the U.S. population has Internet connectivity in their home (now around 70%), the Internet may soon surpass landline telephones in statistical coverage of U.S. households (now around 82%, but still declining from a high of around 95% as people transition to cell-only telephone service to save money). To resolve the current coverage issue surrounding the fact that 30% of U.S. households do not have in-home Internet connectivity, Knowledge Networks supplies a computer and pays for the Internet service for recruited panelists needing such. These advantages and statistical features make a convincing case that the Internet will soon surpass the telephone as the best way to interview people within their homes.

In summary, as budgets become increasingly constrained due to economic pressure from almost everywhere, Federal statistical agencies need to focus on less-costly, but still scientifically respectable, modes of data collection. Online methodologies that use non-scientific opt-in/volunteer panels will not suffice for Federal data collection requirements; but high-quality, probability-based, online surveys should be expected to flourish due to their many advantages. The Federal sector is currently behind the private sector with respect to adopting online survey research, but I predict this will change markedly in the next few years.

Located in Washington, D.C., Joe Garrett is VP, Government & Academic Research. He works with federal agencies and Washington-based policy organizations and universities. Prior to joining KN, he was at Market Strategies International. Mr. Garrett's career spans 30 years; from 1997 - 2004, he was VP and Deputy Director of Surveys and Information Services at Mathematica Policy Research. Prior, Mr. Garrett served ACNielsen as VP and Chief Statistical Officer. He began his career at the Census Bureau in 1978. Mr. Garrett brings depth of experience in research; his expertise includes sample design, estimation, survey methods, and survey operations. He has published numerous papers and articles on statistical methods and survey research techniques.

Monument Photo: © Ptphotos I Dreamstime.com