PRINCIPLES OF FORECASTING
by Simon
Did you know there were principles of forecasting? I don't mean like the positions of the planets. Which for time spans of tens of thousands of years is fairly mechanical. The kind of forecasting I'm talking about involves events that are less deterministic than the motions of the planets. And yet there are principles.
The first is to classify the methodology. Are you starting with numbers or guesses? Which is to say how good is your data base? If you have numbers, what kind of precision is attached? Do you use the numbers directly? Or do you use statistical methods to tease out "useful" information?
OK. You have some data. Now you have to select a method of analysis that is both suitable to the data and the purpose for which it will be used. Is this an investment decision? Or just a report on something to keep an eye on? Do you have a business plan in hand or just a casual "this seems like a good idea"?
The above pages are full of annotated charts with little pop-up explanation boxes to help you understand the charts.
And if that isn't enough the authors of these pages and the accompanying book will give you free help if you describe your problem(s) to them.
We have come a ways and surely it can't be just to talk about forecasting methods. Well yes and no. I want to talk about climate. Climate forecasting.
J. Scott Armstrong, of the Wharton School, University of Pennsylvania, and Kesten C. Green, of the Business and Economic Forecasting Unit, Monash University have done a short audit of IPCC climate science [pdf] based on the forecasting principles outlined above.
nzclimatescience.net
I think it would be good to start with the title which really gets to the heart of the matter.
Global Warming: Forecasts by Scientists versus Scientific Forecasts
Naturally they have some points to make.
In 2007, a panel of experts established by the World Meteorological Organization and the United Nations Environment Programme issued its updated, Fourth Assessment Report, forecasts. The Intergovernmental Panel on Climate Change's Working Group One Report predicts dramatic and harmful increases in average world temperatures over the next 92 years. We asked, are these forecasts a good basis for developing public policy? Our answer is "no".
Much research on forecasting has shown that experts' predictions are not useful. Rather, policies should be based on forecasts from scientific forecasting methods. We assessed the extent to which long-term forecasts of global average temperatures have been derived using evidence-based forecasting methods. We asked scientists and others involved in forecasting climate change to tell us which scientific articles presented the most credible forecasts. Most of the responses we received (30 out of 51) listed the IPCC Report as the best source. Given that the Report was commissioned at an enormous cost in order to provide policy recommendations to governments, the response should be reassuring. It is not. The forecasts in the Report were not the outcome of scientific procedures. In effect, they present the opinions of scientists transformed by mathematics and obscured by complex writing. We found no references to the primary sources of information on forecasting despite the fact these are easily available in books, articles, and websites. We conducted an audit of Chapter 8 of the IPCC's WG1 Report. We found enough information to make judgments on 89 out of the total of 140 principles. We found that the forecasting procedures that were used violated 72 principles. Many of the violations were, by themselves, critical. We have been unable to identify any scientific forecasts to support global warming. Claims that the Earth will get warmer have no more credence than saying that it will get colder.
Then they have a devastating word about the "consensus".
• Agreement among experts is weakly related to accuracy. This is especially true when the experts communicate with one another and when they work together to solve problems. (As is the case with the IPCC process).
• Complex models (those involving nonlinearities and interactions) harm accuracy because their errors multiply. That is, they tend to magnify one another. Ascher (1978), refers to the Club of Rome's 1972 forecasts where, unaware of the research on forecasting, the developers proudly proclaimed, "in our model about 100,000 relationships are stored in the computer." (The first author was aghast not only at the poor methodology in that study, but also at how easy it was to mislead both politicians and the public.) Complex models are also less accurate because they tend to fit randomness, thereby also providing misleading conclusions about prediction intervals. Finally, there are more opportunities for errors to creep into complex models and the errors are difficult to find. Craig, Gadgil, and Koomey (2002) came to similar conclusions in their review of long-term energy forecasts for the US made between 1950 and 1980.
• Given even modest uncertainty, prediction intervals are enormous. For example, prediction intervals expand rapidly as time horizons increase so that one is faced with enormous intervals even when trying to forecast a straightforward thing such as automobile sales for General Motors over the next five years.
They have lots more where that came from. What it boils down to is a warning in the wash room. Keep your eye on this. It is not worth a meeting. Let alone a report to the investment committee.
In electronics we can work with very complex systems because the interactions are strictly limited. How is this done? A marvelous Bell Labs invention called the transistor. It isolates as well as performing other useful functions.
The electronics guys, with lots of knowledge and isolation plus simple models, are real happy when their predictions of what will happen next in a circuit comes within 5%. The climate guys say they can tell within better that 1%. What are the odds?
When you have lots of things or some very complex things interacting, prediction gets hard. As a very great Yogi is reputed to have said: "Prediction is very difficult, especially about the future."
Cross Posted at Power and Control
posted by Simon at 08:20 AM
classicalvalues.com
Here’s some thingsintersting and entertaining from the report:
………..
Although they may seem convincing at the time, expert forecasts make for humorous reading in retrospect. Cerf and Navasky’s (1998) 310 pages of examples, such as the following, are illustrative:
“[The nickel-iron battery will put] the gasoline buggies…out of existence in no time.”
Thomas Alva Edison 1910
“There is not the slightest indication that [nuclear] energy will ever be obtainable.”
Albert Einstein 1932
“I think there is a world market for about five computers.”
Thomas J. Watson, Chairman of IBM, 1943
“A few decades hence, energy may be free.”
John von Neumann, Fermi Award-winning scientist, 1956
Examples of faulty expert climate forecasts are easy to find, but are perhaps less humorous:
“If present trends continue, the world will be about four degrees colder in 1990, but eleven degrees colder in the year 2000. This is about twice what it would take to put us into an ice age.”
Kenneth Watt, UC Davis ecologist, Earth Day,
April 22, 1970 Swarthmore College speech
…………….
Experts’ forecasts of climate changes have long been popular. Anderson and Gainor (2006) found the following headlines in their search of the New York Times:
Sept. 18, 1924: “MacMillan Reports Signs of New Ice Age.”
March 27, 1933: “America in Longest Warm Spell Since 1776”
May 21, 1974: “Scientists Ponder Why World’s Climate is Changing: A Major Cooling Widely Considered to be Inevitable.”
Dec. 27, 2005: “Past Hot Times Hold Few Reasons to Relax About New Warming.”
In each case, the forecasts were made with a high degree of confidence.
……………..
Stewart and Glantz (1985) conducted an audit of the forecast by the NDU (1978) that was described above. They were critical of the report because it showed a lack of awareness of proper forecasting methodology. Their audit was hampered because the organizers of the study said that the raw data had been destroyed and a request to the Institute for the Future about the sensitivity of the forecasts to the weights went unanswered. Judging from a Google Scholar search, climate forecasters have paid little attention to this paper.
Carter, et al. (2006) examined the Stern Review (Stern 2007). They concluded that the Report authors made predictions without any reference to scientific forecasting.
Pilkey and Pilkey-Jarvis (2007) concluded that the long-term climate forecasts that they examined were based only on the opinions of the scientists. The opinions were expressed in complex mathematical terms. There was no validation of the methodologies. They referred to the following quote as a summary on their page 45: “Today’s scientists have substituted mathematics for experiments, and they wander off through equation after equation and eventually build a structure which has no relation to reality. (Nikola Telsa, inventor and electrical engineer, 1934.)” Thus, while it is sensible to be explicit about beliefs and to formulate these in a model, the forecaster must go beyond this to demonstrate that the relationships are valid and well-supported, especially when the models are complex.
Carter (2007) examined evidence on the predictive validity of the general circulation models (GCMs) used by the IPCC scientists. He found that while the models included some basic principles of physics, scientists had to make “educated guesses” about the values of many parameters because knowledge about the physical processes of the earth’s climate is incomplete. In practice, the GCMs failed to predict recent global average temperatures as accurately as simple curve-fitting approaches (Carter 2007, pp. 64 – 65) and also forecast greater warming at higher altitudes when the opposite has been the case (p. 64). Further, individual GCMs produce widely different forecasts from the same initial conditions and minor changes in parameters can result in forecasts of global cooling (Essex and McKitrick, 2002).
Interestingly, modeling results that project global cooling are often rejected as “outliers” or “obviously wrong” (e.g., Stainforth et al., 2005).
Taylor (2007) compared seasonal forecasts by New Zealand’s National Institute of Water and Atmospheric Research with outcomes for the period May 2002 to April 2007. He found NIWA’s forecasts of average regional temperatures for the season ahead were, at 48% correct, no more accurate than chance. That this is a general result was confirmed by New Zealand climatologist Dr Jim Renwick, who observed that NIWA’s low success rate was comparable to that of other forecasting groups worldwide. He added that “Climate prediction is hard, half of the variability in the climate system is not predictable, so we don't expect to do terrifically well.” Dr Renwick is an author on Working Group I of the IPCC 4th Assessment Report, and also serves on the World Meteorological Organisation Commission for Climatology Expert Team on Seasonal Forecasting; His expert view is that current GCM climate models are unable to predict future climate any better than chance (New Zealand Climate Science Coalition 2007).
In another example, the US National Hurricane Center’s report on hurricane forecast accuracy noted, “No routinely-available early dynamical model had skill at 5 days” (Franklin 2007). This comment probably refers to forecasts for the paths of known, individual storms, but seasonal storm ensemble forecasts are clearly no more accurate. For example, the NHC’s forecast for the 2006 season was widely off the mark. On June 7, Vice Admiral Conrad C. Lautenbacher, Jr. of the National Oceanic and Atmospheric Administration gave the following testimony before the Committee on Appropriations Subcommittee on Commerce, Justice and Science of the United States Senate (Lautenbacher 2006, p. 3):
NOAA's prediction for the 2006 Atlantic hurricane season is for 13-16 tropical storms, with eight to 10 becoming hurricanes, of which four to six could become major hurricanes. … We are predicting an 80 percent likelihood of an above average number of storms in the Atlantic Basin this season. This is the highest percentage we have ever issued.
By the beginning of December, Gresko (2006) was able to write “The mild 2006 Atlantic hurricane season draws to a close Thursday without a single hurricane striking the United States”.
……………
Chapter 8 was, in our judgment, poorly written. The writing showed little concern for the target readership, provided extensive detail on items that are of little interest in judging the merits of the forecasting process, provided references without describing what readers might find, and imposed an incredible burden on readers by providing 788 references. The readability of the chapter was low. For example, Section 8.2.1.3 “Parametrization,” a critical section for understanding the forecasting process, scored 23 for Flesch-Kinkaid Reading Ease (“plain English” is 60 and the Harvard Law Review scores 32), and a “grade level” Gunning-Fog Index of 21 (5 is “very readable” and 20 is “very difficult”). In addition, the Chapter reads in places like a sales brochure. In the three-page executive summary, the terms, “new,” and “improved,” and related derivatives appeared 17 times. Most significantly, the chapter omitted key details on the assumptions and the forecasting process that were used by its authors. Given these problems, we found it puzzling that so many of our respondents nominated the IPCC report as the most credible paper on long-term climate forecasts.
……………….
We each made a formal, independent audit of IPCC Chapter 8 in May 2007. To do so, we used the Forecasting Audit software on the forecastingprinciples.com site, which is based on material originally published in Armstrong (2001). To our knowledge, it is the only evidence-based tool for the evaluation of forecasting procedures.
While Chapter 8 required many hours to read, it took us each about one hour to rate the forecasting approach described in the Chapter using the Audit software. We have each been involved with developing the Forecasting Audit program, so other users would require much more time. Ratings are on a 5-point scale from -2 to +2. A rating of +2 indicates the forecasting procedures were consistent with a principle, and a rating of -2 indicates failure to comply with a principle. The Audit software also has options to indicate that there is insufficient information to rate the procedures or that the principle is not relevant to a particular forecasting problem.
Our overall average ratings were similar at -1.37 and -1.35.
………………
In our judgment, the uncertainty in forecasting global mean temperature is extremely high. For example, what, if any, is the current trend? Carter (2007, p. 67) wrote:
…the slope and magnitude of temperature trends inferred from time-series data depend upon the choice of data end points. Drawing trend lines through highly variable, cyclic temperature data or proxy data is therefore a dubious exercise. Accurate direct measurements of tropospheric global average temperature have only been available since 1979, and they show no evidence for greenhouse warming. Surface thermometer data, though flawed, also show temperature stasis since 1998.
Uncertainty over the trend in mean temperature is illustrated by Figure 1 (taken from Carter (2007) and originating in Davis and Bohling (2001)).
Figure 1
Climatic cycling over the last 16 000 years as indicated by averaged 20-year oxygen isotope ratios from the GISP2 Greenland ice core. Trend lines A-E all extend up to the end of the 20th century, fitted through the data for the last 16 000, 10 000, 2000, 700 and 100 years, respectively. The trends are indicative of both warming and cooling, depending upon the chosen starting point.
Global climate is complex. Scientific evidence on many key relationships is weak or absent; e.g., does CO2 cause high temperatures or do high temperatures cause CO2 (e.g. Jaworowski 2007)? And what effect do variations in solar activity have? (See for example Figure 2, reproduced from Soon 2005, and Svensmark’s 2007 paper on the sun’s influence on cloud seeding by cosmic rays). Measurements of key variables such as local temperatures and a representative global temperature are contentious in the case of modern measurements, because of possible artifacts such as the urban heat island effect, and often speculative in the case of ancient ones, such as those climate proxies derived from tree ring and ice-core data (Carter 2007).
Finally, it is difficult to forecast the causal variables.
The already high level of uncertainty rises rapidly as the forecast horizon increases.
While the authors of Chapter 8 claim that the forecasts of global mean temperature are well-founded, their language is imprecise and relies heavily on such words as “generally,” “reasonable well,” “widely,” and “relatively” [to what?]. The report makes many explicit references to uncertainty. For example, the phrases “. . . it is not yet possible to determine which estimates of the climate change cloud feedbacks are the most reliable” and “Despite advances since the TAR, substantial uncertainty remains in the magnitude of cryospheric feedbacks within AOGCMs” appear on p. 593. In discussing the modeling of temperature, the authors wrote, “The extent to which these systematic model errors affect a model’s response to external perturbations is unknown, but may be significant” (p. 608), and, “The diurnal temperature range… is generally too small in the models, in many regions by as much as 50%” (p. 609), and “It is not yet known why models generally underestimate the diurnal temperature range.” The following words and phrases appear at least once in the Chapter: unknown, uncertain, unclear, not clear, disagreement, uncertain, not fully understood, appears, not well observed, variability, variety, difference, unresolved, not resolved, and poorly understood.
Given the high uncertainty, the naïve method for this situation would be the “no-change” model. Remarkably, nowhere does the IPCC Report address the issue of forecastability. It should have been addressed prior to spending enormous sums on complex forecasting models.
In effect, given the current state of uncertainty regarding climate, prior evidence on forecasting methods suggests that attempts to improve upon the naïve model might increase forecast error. To reverse this conclusion, one would have to produce validated evidence in favor of certain methods. Such evidence is not provided in Chapter 8 of the IPCC report.
Figure 2
Annual-mean Arctic-wide air temperature anomaly time series (dotted lines) correlated with the estimated total solar irradiance (top panel; solid lines) and with the atmospheric carbon dioxide, CO2, mixing ratio (bottom panel; solid lines) from 1875 to 2000.
I encourage readers to go to the pdf file and view Figures 1 and 2. Very enlightening.
……………
Conclusions
The Stern Review concluded that, “The scientific evidence is now overwhelming; climate change presents very serious global risks, and it demands an urgent global response” (Stern 2007, p. xv). We have not been able to find any scientific evidence to support such a claim. We can only hope that before committing resources, decision makers will insist on scientific forecasts rather than accept the opinions of some scientists.
To provide forecasts that are useful for policy-making, one would need to prepare forecasts not only of global temperature, but also of the net effects of any temperature change; then on the effects of policy changes aimed at reducing temperature changes or the negative effects of it, the costs of such changes, and the likelihood of successful implementation. A failure at any stage would nullify any value to the forecasts.
We have shown that failure occurs at the first stage of analysis. Specifically, we have been unable to find a single scientific forecast to support the currently widespread belief in dangerous, human-caused “global warming”. Prior research on forecasting suggests that a naïve (no change) forecast would be superior to current predictions which are, in effect, experts’ judgments only.
Based on our Google searches, those forecasting long-term climate change have no apparent knowledge of evidence-based forecasting methods, so we expect that the same conclusions would apply to the other three necessary parts of the forecasting problem.
By relying on evidence-based forecasting methods, we conclude that policies founded on predictions of man-made global warming from models based on the opinions of scientists will be harmful.
Given the conditions involved in long-term global forecasts and the high uncertainty involved, prior research on forecasting suggests that even if the forecasting methods were properly applied, it may not be possible to improve upon the naïve, “no-change,” forecast. We do not even have evidence that it is possible to make useful medium term (e.g., one to five year) forecasts.
Our paper is concerned with rational assessments of public policy, not with public opinions. People will continue to believe that serious manmade global warming exists as they will continue to believe other things that have no scientific support (e.g., the biblical creation story, astrology, minimum wages as a way to help poor people, and so on). It is appropriate for concerned individuals to donate money or time for such perceived issues as long as they do not harm others in these efforts. It is not appropriate to impose on others policies that have not been shown to have scientific merit.
One might say that it is important to consider steps to prevent global warming, but we have the same level of confidence in saying that we should take steps to prevent global cooling. The more important question is “what is the best way to invest our resources for the benefit of mankind?” This would lead to such trade-offs as asking whether it is better to spend a dollar on reducing AIDS or air pollution or malaria or breast cancer, where we know what policies will work, or to spend it on controlling future climate control, where we have no scientific evidence.
Given the large uncertainties of climate change science, government polices on climate control are unwarranted and will reduce the well-being of people who are not the beneficiaries of the wealth redistribution that will occur as a result of such policies. For example, wealthy owners of beach-front properties, climate researchers, consultants, those involved in carbon trading markets, and manufacturers of some energy sources might expect to gain from global warming policies,
Those who advocate various positions on the climate owe it to the people who would be affected by the policies they recommend to base their advocacy on scientific forecasts that address all four of the key areas that are necessary for a rational analysis of the problem.
|
