White House Seeks
Peer-Review Standard
For Range of Studies
online.wsj.com
It is 2004. The Food and Drug Administration has received a slew of reports that an artificial heart valve fails when patients also take certain medications. The agency concludes that alerting the medical community will save lives. No go: A new federal rule requires that the FDA first assemble outside experts to meet, review the evidence and write a report. While the "peer review" grinds on, dozens more heart patients die.
Peer review has been the gold standard in science for more than 200 years. No reputable journal publishes a paper that has not first undergone scrutiny by qualified scientists in relevant fields; peer review is how the research community keeps bunk science out of the scientific record.
Who could possibly oppose this quality control? You'd be surprised. Some health officials, academics and consumer groups say peer review can impede regulations meant to protect public safety.
But in what John Graham of the White House Office of Management and Budget calls "a major priority for this administration," OMB is proposing that any study, risk assessment or other "significant regulatory information" used or produced by a U.S. agency not be released until it has undergone formal peer review of the kind OMB would prescribe.
Peer review, says Mr. Graham, "improves the technical quality of information products."
Outside scientists would do the review, and the agency would have to address any concerns they raise. There would be exceptions for emergencies, perhaps such as the hypothetical heart valves, but what qualifies as an emergency would be decided by OMB bureaucrats rather than FDA scientists, at least as the proposal now stands.
Some regulated industries, such as chemical makers, support the proposal. Depending on where you fall on the cynicism spectrum, you can attribute that to their commitment to "sound science," which they say peer review of federal studies will encourage, or to a desire to thwart health, safety and environmental regulation by insisting on a level of scientific certainty impossible to achieve in the real world.
Last month, the American Public Health Association announced its opposition to OMB's proposal, arguing that "public-health decisions must be made in the absence of scientific certainty, or in the absence of perfect information." And at a recent workshop at the National Academy of Sciences, it wasn't only the usual suspects, like Ralph Nader's Public Citizen, who voiced concern. So did pillars of the science establishment.
John Bailar, a leading biostatistician and professor emeritus at the University of Chicago, described himself as "greatly concerned about the potential for mischief" in the proposal. Climatologist Warren Washington, chair of the National Science Board, said "there could be opportunities for directing science in a way that is not in the public interest." Because science entails judgment, Sheila Jasanoff of Harvard University told me, she doubts peer review such as OMB envisions would produce better regulatory science.
In part, these concerns reflect a dirty little secret of peer review: Even at the best journals, it can't guarantee the "right" answer. Entire forests are felled to print papers whose conclusions turn out to be flat-out wrong (or fraudulent). Reviewers aren't always objective; some judge a study not on its merits but on whether they agree with its approach and conclusions. A recent analysis of peer review found scant evidence it improves published biomedical research.
To some, OMB's proposal looks like a solution in search of a crisis. "It offers peer review as a solution, but nowhere defines the problem," says Michael Taylor of Resources for the Future, a Washington think tank, and former deputy commissioner at the FDA.
Indeed, OMB hasn't publically given examples in which the lack of adequate peer review produced bunk science, but one official pointed me to a 1980 report by the National Highway Transportation Safety Administration. It concluded that the potential benefit of air bags "greatly outweighs the small possible risk of injury" to children. Industry studies disagreed. Since 1993, air bags have killed 144 kids.
"I see real value in [prescribed peer review] if it actually improves the analysis," says risk analyst Kimberly Thompson of Harvard. But, she adds, it isn't clear whether peer review would have brought out the need to protect kids from air bags. Besides, before an agency issues a regulation, it must by law consider findings contrary to its own, anyway.
At least peer review need not cause paralysis by analysis, as critics contend. Before decontaminating the Hart Senate Building after the anthrax mailings in 2001, notes Paul Gilman of the Environmental Protection Agency (which has required peer review of its science since 1995), the EPA gathered experts via conference call to speedily vet the plan. "You can do peer review quickly," he says. Adds Mr. Graham, it "enhances the credibility of government science."
For an agency focused on costs and benefits, OMB hasn't been very clear about what those of prescribed peer review would be. Nor has it laid out how that system would produce better regulatory science than, say, improving agencies' science advisory boards. The peer review proposal is open for public comment until Dec. 15. |
