Mystery Pollster
Demystifying the Science and Art of Political Polling - By Mark Blumenthal
November 02, 2004
Exit Polls: What You Should Know
So sometime very soon, the traffic on certain web sites will hit astronomically high levels as the blogosphere goes in search of the latest leaked exit polls. The conventional wisdom on this is unshakable: The networks "know" who will win, but won't tell us. Lets take our own peek at results shared by those working at the networks today and get in on the secret.
Well, I hate to disappoint, but this site will not be a source of leaked "exits." However, I would like to take a moment and give you a bit of a reality check. Let me tell you a bit about what exit polls are and why you may want to take what you hear over the web with a giant grain of salt.
I have always been a fan of exit polls. Despite the occasional controversies, exit polls remain among the most sophisticated and reliable political surveys available. They will offer an unparalleled look at today's voters in a way that would be impossible without quality survey data. Having said that, they are still just random sample surveys, possessing the usual limitations plus some that are unique to exit polling (I also remain dubious about weighting telephone surveys to match them, but that is another story for another day).
A quick summary of how exit polls work: The exit pollster begins by drawing a random sampling of precincts within a state, selected so that the odds of any precinct being selected are proportionate to the number that typically vote in that precinct. The National Election Pool Exit Poll, which is conducting the exit polling for the six major networks today, will send exit pollsters to 1,495 precincts across the country.
One or sometimes two interviewers will report to each sampled precinct. They will stand outside and attempt to randomly select roughly 100 voters during the day as they exit from voting. The interviewer will accomplish this task by counting voters as they leave the polling place and selecting every voter at a specific interval (every 10th or 20th voter, for example). The interval is chosen so that approximately 100 interviews will be spread evenly over the course of the day.
When a voter refuses to participate, the interviewer records their gender, race and approximate age. This data allow the exit pollsters to do statistical corrections for any bias in gender, race and age that might result from refusals.
The interviewer will give respondents a 5 1/2 by 8 1/2 card to fill out that will include approximately 25 questions (see an example from the New Hampshire primary here). Respondents fill out the survey privately then put the completed survey in a clearly marked “ballot box” so they know their identities cannot be tracked and their answers remain confidential.
The biggest challenge to exit polls is logistical: How to transmit all the results to a central location quickly and accurately. In past elections, interviewers would take a 10 minute break every hour to tabulate responses. Interviewers would then call in tabulations at three approximate times during the day: 9:00 a.m., 3:00 p.m. and shortly before the polls close (disclaimer: I have no first hand knowledge of this year's procedures, which may be different).
Once the polls close, the interviewer will attempt to obtain actual turnout counts, and if possible, actual vote returns for their precinct. One of the unique aspects of the exit poll design is the way it gradually incorporates real turnout and vote data as it becomes available once the polls close. The exit poll designers have developed weighting schemes and algorithms to allow all sorts of comparisons to historical data that supports the networks as they decide whether to “call” a state for a particular candidate. When all of the votes have been counted, the exit poll is weighted by the vote to match the actual result.
So if this poll is so sophisticated, why can't we rely on the leaked mid-day “numbers” that will soon spread like wildfire across the web?
1) It is still just a survey – Even when complete, an exit poll still has the same random variation as any other survey. NEP says typical state exit polls will have a sampling error when complete of +/- 4% at a 95% confidence level, and +/- 3% for the national exit poll. Even if comparable to the final numbers – which they are decidedly not – the mid-day leaked numbers would have much greater error, perhaps +/- 7% or more.
2) The mid-day numbers do not reflect weighting by actual turnout – the end-of-day exit poll used to assist the networks in determining winners will be weighted by the actual turnout of voters at each selected precinct. The weighting will then be continuously updated to reflect turnout at comparable precincts. In the past, mid-day numbers have reflected a weighting based on past turnout, so the leaked mid-day numbers may tell us nothing about the impact of new registrants or the unique level of turnout this time.
One point needs emphasis here: even in past elections, networks never called an election based on raw exit poll numbers alone. They were first weighted by a tally of the full day's turnout at each sampled precinct. This end-of-day data is (obviously) not available at 12 noon.
3) Voting patterns may be different early in the day - People who work full time jobs typically vote more heavily before or after work. Even a perfect mid-day exit poll – and there is no such animal – may not be any better at picking a winner than the half-time scores in any given football game on Sunday. Also, despite what you may have heard on the West Wing, I know of no serious study showing a consistent Democratic or Republican tilt to the morning or evening hours (if anyone does, please email me).
4) Early or absentee voting - As of last night, the ABC News survey estimated that 15% of all registered voters nationally had already cast absentee or early ballots. Obviously, these voters will not be available to interviewers standing outside polling places. To incorporate early voting, the National Election Pool is doing telephone interviewing in 13 states to sample the votes of those who voted early. Will these early votes be included in the mid-day leaked numbers? Who knows? I wouldn't count on it. (Good question, Andrew).
5) They could be fictional - Both sides have huge armies of field workers sweating it out in the streets right now. Field workers have been known to find creative ways to boost the morale of their own troops or demoralize the other side. Might someone start a rumor by sending made up numbers to a blog? Ya think? After all, the guy most web surfers turn to for leaked exits likes to say that the information he provides is only 80% accurate. What are the chances he could be, excuse the technical term, making shit up?
6) The people who do exit polls would rather you ignored them - OK, admittedly, that is a pretty wimpy reason, but they have a point. Exit polls provide a valuable resource for all of us. The will help us better understand who the voters are, why they vote the way they do and what the answers are to some of the debates that have raged for months that will not be resolved by vote returns alone. When someone leaks or broadcasts results of an exit poll (or telegraphs it by winking the way certain news networks tend to do about about 4 or 5 o'clock), calls are made to ban exit pollsters from polling places. That would be a very bad thing.
Listen, I understand human nature, and I'm not going to try to change it. We are all intensely curious about what is going to happen tonight, and most of us will find a way to peek at leaked exit polls at some point today. I just want you to know that those leaked exit polls really don't tell us much more about the outcome of the race than the telephone polls we were obsessing over just a few hours ago. Even if we wanted to call a race on unweighted, unfinished, mid-day exit polls alone (something the networks will not do), we would need to see differences of 10-15 points separating the candidates to be 95% certain of a winner.
So look at them if you must, but please, don't go plugging the numbers into spreadsheets and assume that your Electoral College "projections" have any special magic or scientific validity. Then don't. You might be better off flipping a coin to determine the outcomes of states like Ohio, Florida, Wisconsin, Iowa, etc.
I'll be back with more later in the day. If you have questions about exit polls that don't involve today's numbers, please sent them my way. You might also want to check the Frequently Asked Questions at the National Election Pool web site. See the jump page for offline sources this post.
mysterypollster.com |
