Lies, damned lies, and statistics...
The Poll Watchers
Gallup, USA Today, CNN Polls Come Under Fire
Watchdog Group Issues Rebuke on Poll on Islamic Countries
By Richard Morin and Claudia Deane
Washington Post Staff Writers
Friday, March 22, 2002; 11:32 AM
The hugely influential Gallup Poll of Islamic Countries released last month continues to make news--not all of it good for the Gallup Organization, USA Today and CNN.
The National Council on Public Polls, a leading professional watchdog organization, recently posted on its Web site a stinging rebuke of CNN and USA Today for the way their reporters reported the overall results of the nine-nation survey project.
But now it appears that Gallup itself provided reporters with the sensational characterizations that were the primary target of NCPP criticism.
It's also apparent that the hype was unnecessary, tainting an otherwise remarkable and important polling project, survey experts said.
One thing is certain: The polls made big news around the world. President Bush publicly expressed dismay about the results. So did a small army of diplomats, policymakers and pundits.
And little wonder. As reported by CNN and USA Today, the poll found that 53 percent of those interviewed said they had an unfavorable view of the United States. Nine percent thought the U.S. military action in Afghanistan was justified. Fewer than one in five – 18 percent – in six of the nine countries believe Arabs carried out the September 11 terrorist attacks. (Three countries wouldn't let Gallup polltakers ask the question.)
Enron Arithmetic
You won't find those results on Gallup's Web site, however. That's because of one rather big problem: These eye-opening results were "actually the average for the countries surveyed regardless of the size of their populations," the NCPP noted. "Kuwait, with less than 2 million Muslims, was treated the same as Indonesia, which has over 200 million Muslims."
That's Enron arithmetic. It's as if California and South Dakota each were granted the same number of electoral votes in presidential elections.
The problem would be particularly worrisome if there were big differences in the results across the nine countries. And there were, at least on some key questions. For example, 36 percent of those interviewed in tiny Kuwait said the September 11 terrorist attacks were morally justifiable, compared to only 4 percent in Indonesia.
If the results of the two countries were averaged together, which is what NCPP said that USA Today and CNN did, the result suggests that about 20 percent of these Muslims seemed to view the attack as justified.
But if the results were properly adjusted to account for population, the result is very different: About 5 percent of all Muslims in Kuwait and Indonesia thought the terrorist attacks were morally justified.
So where did reporters get those bogus aggregate numbers?
From the Gallup Organization, said Andrea Stone, the reporter who wrote USA Today's page 1 story that showcased those skewed averages. The aggregates were clearly noted in the first fax sent by Gallup to USA Today and CNN, which also included the country-by-country results. "I didn't do the arithmetic," Stone said.
Frank Newport, editor-in-chief of the Gallup Poll, acknowledged that the aggregates were included in the material supplied to news organizations. So that means Gallup itself is partially to blame for the snafu, right?
"I don't know," Newport said.
Newport said Gallup analysts repeatedly cautioned against using the aggregated numbers when the results we re released at a seminar in Washington. But no such warning appeared on Gallup's first release to Stone or to reporters at CNN. And it was this release that reporters used to prepare their stories.
Gallup conspicuously omits references to the aggregate numbers in its own analyses of the poll data. Instead, it cites results from individual countries and strongly urges others to do the same. "Look at the [results from] individual countries, look at the differences in attitudes between countries," Newport said. "That's how we have chosen to look at it."
Newport is the vice president of NCPP, whose members include many of the nation's most respected and influential pollsters. He declined to say whether he endorsed NCPP's statement. Gallup is partners with CNN and USA Today on national surveys, but the news organizations were not partners in the Islamic poll.
None of this should suggest that the headlines from the Gallup Islamic poll would have been dramatically different if appropriate adjustments had been made or if only the country-by-country figures had been used. However you slice and dice the data, the fact remains that many Muslims in these nine countries don't like the United States.
Of course it's easier to report one overall average number than nine separate findings. But in this case, it was also wrong and unnecessary.
"That story could have been told without the overall averages," said Warren Mitofsky, an NCPP review board member and former head of Voter News Service. If anything, the story would have been improved "because the overall averages were meaningless."
It's also fair to say that Gallup Poll of Islamic Countries, despite its flaws, ranks as a singularly ambitious, thoughtful and thought-provoking project. The NCPP correctly called it an "important and fascinating study." (Humphrey Taylor, an NCPP review board member Humphrey Taylor and head of Harris Interactive, said the board was unaware that Gallup was the source of the aggregated numbers.)
In its release, the NCPP review board also faulted CNN and USA Today for repeatedly claiming the poll was a survey of "the Muslim world."
Turns out it's a small world – much too small. The nine countries in which the survey was conducted are home to only about 40 percent of all Muslims, the NCPP noted. Included in the study were Indonesia, Iran, Jordan, Kuwait, Lebanon, Morocco, Pakistan, Saudi Arabia and Turkey. Conspicuously absent were views of Muslims living in India, Egypt, Bangladesh and Nigeria. (Gallup never claimed its findings represented the views of all Muslims.)
While too few countries were sampled, too many interviews came from just two countries. "Two-thirds of the Muslims in the nine countries Gallup studied live in Indonesia and Pakistan," the NCPP reported.
One other problem: not everyone interviewed for the poll was Muslim. "The surveys were samples of all residents of the countries surveyed, not only Muslims," the NCPP statement read. (In hindsight, this probably was a minor problem: fewer than 500 of the 9,924 respondents were non-Muslim, according to Gallup.)
In fact, you didn't need to be a citizen of the country where the interviews were conducted. For example, fewer than half of the individuals in the Kuwait sample were Kuwaiti citizens.
Newport said his organization will see if changes need to be made to reduce the chances that its data are mischaracterized, misused or otherwise mangled by reporters. (One easy fix: Don't release the aggregate results when the aggregates are meaningless, as everybody acknowledges these were.)
But Newport stopped far short of criticizing USA Today or CNN for the way they used the results in their stories. "The thrust of what they suggested was an accurate indication of what the data showed," Newport said.
Meanwhile, those bogus aggregates continue to circulate freely in this country and around the world. They were cited approvingly by Tim Russert on last Sunday's "Meet the Press" and in recent days on the BBC and in stories appearing in the Agence France-Presse, the Manchester Guardian and other newspapers.
As it stands, the NCPP has no plans to amend its statement to criticize Gallup at least a little for releasing aggregate numbers.
"It's all water under the bridge," Taylor said.
E-mail Richard Morin and Claudia Deane at polls@washpost.com
© 2002 The Washington Post Company |
