For those of you interested in polling:
"The Pros and Cons of Auto-Dialed Surveys
It's no secret to regular readers of this blog that The Fix's interest in political polling borders on obsession. I love the idea of combing through data to find the messages hidden within the numbers.
Since I spend so much time parsing the polls, I have formed opinions about what makes a poll worthy of attention and what doesn't. Ever since my time at Roll Call newspaper, I have generally viewed telephone surveys that employ live interviewers as the most reliable. I'm much more skeptical about polling that uses automated interviewers -- Survey USA and Rasmussen Reports being the two firms best known for using this approach -- so I largely avoid citing these surveys on The Fix.
But polling -- like everything else in politics -- is constantly evolving, so it makes sense to take a closer look at the arguments for and against auto-dialed polls to see how much weight they should be afforded when handicapping a political race. For those with questions about the Zogby/Wall Street Journal online polling, we'll get to that in a future "Parsing the Polls."
In order to best tackle this immense subject, I am going to break it down into three more manageable pieces -- methodology (or how Survey USA and Rasmussen conduct their polls), advantages and disadvantages.
First, The Fix needs to acknowledge the yeoman's work that Mark Blumenthal (a.k.a. the "Mystery Pollster") has done on this subject. Mark's site is an invaluable resource to anyone with a deep interest in the science and art involved in polling.
Method Makes a Difference
How a polling firm goes about reaching respondents is often the most critical element of a successful survey. Survey USA and Rasmussen make their methodology available to the public -- albeit in varying forms of candor.
The only fundamental difference between standard telephone surveys and those conducted by Rasmussen and Survey USA is that instead of a live interviewer the voice walking the respondent through the questionnaire is automated. Survey USA has partnered with media outlets (typically television stations and/or newspapers) in all 50 states and uses a professional announcer familiar to most locals. Rasmussen employs a "single, digitally recorded, voice," according to its Web site.
Much like a traditional telephone poll, the raw results of the Survey USA and Rasmussen surveys are then weighted by the two firms to ensure the sample is reflective of the demographics of a particular congressional district or state.
The Pluses
The biggest benefit of these "robo" polls is cost. "Survey USA is the first research company to appreciate that opinion research can be made more affordable, more consistent and in some ways more accurate by eliminating the single largest cost of conducting research, and a possible source of bias: the human interviewer," reads a statement on the organization's Web site.
Most polling companies -- partisan and independent -- use phone banks to make the thousands of calls necessary to produce a scientific sample. The process is both time-consuming and extremely expensive. Using a recorded voice makes it easier to survey larger samples, poll more frequently and put surveys into the field closer to elections. That last element of flexibility could well explain why these sorts of surveys have done as well if not better than traditional polling in predicting the final outcome in races. Most live interview, telephone polls end on the Thursday before the election, meaning that they might well miss the late breaks in a campaign.
The second major argument for auto-dialed polls is that by eliminating the middle man (in the person of the human interviewer), the margin for so-called "measurement error" is reduced. Survey USA's Web site lists a variety of ways in which a live interviewer can compromise an interview, ranging from mispronouncing names to tiring toward the end of a call list to simply reading the questionnaire too fast or too slow.
One intriguing but more difficult to quantify element of this measurement error is that people tend to exaggerate their likelihood to vote when interviewed by a live person, while they answer more honestly when prompted by a recorded voice. The psychology behind this trend? Voting remains a private act for many people and they are not comfortable sharing their past voting history with a stranger over the phone. In fact, the anonymity of the recorded voice, the thinking goes, more accurately simulates the conditions of a voting booth.
The Minuses
Establishment pollsters take issue with several elements of auto-dialed polling.
The first deals with the response rates. A traditional live interview telephone poll has a response rate of roughly 30 percent -- meaning that three out of every ten households contacted participate in the survey. The polling establishment has long held that people are less likely to respond to an automated survey than a call from a real person, meaning that auto-dialed poll have even lower response rates and therefore a higher possibility of bias in the sample. Neither Rasmussen nor Survey USA makes their response rates public, although, in fairness, neither do most media outlets or major partisan pollsters.
The second -- and potentially more troublesome -- issue revolves around the randomness of auto-dialed polls. In a standard telephone poll, the interviewer may seek to add another layer of randomness by asking to speak for a specific person in a household, such as whoever most recently celebrated a birthday. Automated polls do not attempt to do that. Establishment pollsters argue that by stripping a level of randomness from the polling process, auto-dial pollsters must more heavily weight their samples to achieve demographic diversity --- rendering the results almost meaningless.
Drawing Conclusions
What to make of these varying viewpoints on the strengths and weaknesses of auto-dialed polls? Smart people will disagree. But we defer to two of the smartest we know to provide perspective on the debate.
Fred Yang, a well regarded Democratic pollster, says that he counsels his clients to avoid drawing broad conclusions from any single survey conducted by Survey USA or Rasmussen and rather to "look at [their polls] as a package in order to gauge the overall trends in the race."
Mystery Pollster Blumenthal echoed that sentiment. "Surveys are by their nature prone to error," he said. "The more you look at, the more confident you can be."
Want to read more about the debate over polling methods? Start with Blumenthal's discourse on the future of survey research in Public Opinion Quarterly.
You can also read University of Oregon professor Joel David Bloom's 2003 paper on the topic.
Then there's an article yours truly wrote for Roll Call that takes a deeper look at Survey USA.
As always, the comments section below is open for you to continue the discussion."
blog.washingtonpost.com |
