Some years ago I made a "prediction" of sorts- that Nate Silver would at some point come out as a follower of Bayesian statistics, despite the nominally traditional "frequentist" language of his writings. The Bayesian approach to statistics is distinguished from the frequentist approach taught in most introductory classes by an open use of prior probabilities. These "priors" represent, in a flexible way, what we already informally know, or think we know, about the world. As new data come in, the Bayesian approach provides a systematic quantitative way to update our knowledge.
Silver has now written a wide-ranging book-length personal paean to Bayesian prediction methods, filled with more or less relevant anecdotes about poker, sports, chess, medicine, earthquakes, weather, climate, finance and politics. "The signal and the noise: why so many predictions fail but some don't" is most welcome as a readable popular introduction to real-world rational prediction methods and their limits. The anecdotes give a wonderful feel for how differently the same principles play out in different fields. Nevertheless, in key parts the book's signal is obscured by its own noise. Since, given its strengths, the book can speak for itself, I'm going to dwell here on those weaknesses, especially ones concerning climate predictions.
Silver is blistering in his description of how perverse incentives lead bloviating political pundits to repeatedly make wildly inaccurate predictions. In contrast, he shows too much delicacy in discussing how perverse incentives drive bad financial predictions. A casual reader will mainly notice the lengthy descriptions of technical errors in the risk-estimation models used. A more careful reader will pick up the brief, low-key descriptions of how strongly the entire financial sector is biased toward over-estimating the value of the products whose sales help them pocket a large slice of GDP as personal income. No reader would pick up what Cathy O’Neil points out in a devastating review: "The bankers packaging and selling these deals, which amongst themselves they called sacks of shit, did not blithely believe in their safety because of those ratings." There's not much to add to O'Neil on this issue, so I'll move on to others.
The chapter on global warming gets the generalities right but flounders on specifics. Silver starts with the right framework, that we know the basic physical processes involved in setting the earth's average temperature. In other words, we have some well-founded priors, so that as new data come in we are updating and refining our predictions, not starting de novo with every blizzard or drought. Unfortunately, it soon becomes pretty clear that Silver doesn't actually know the science, just believes that one should know the science.
What had started as a sophisticated discussion rapidly turns dicey when he gets to the actual physics: "...gases absorb solar energy that has been reflected from the earth's surface", an assertion that's wrong on several levels. The gases transmit most solar energy and absorb much of the energy from the earth because it is not reflected but radiated by the earth at much lower frequencies. The effect he describes wouldn't even cause warming. Then he writes that without this effect some 30% of the incoming solar energy would be reflected out into space. Again, the reflection is a different process caused by clouds, ice, etc. and it is not much reduced by greenhouse gases. As for the overall energy flow, essentially all of the energy ultimately gets back out into space, the gases only affecting how rapidly it escapes. He then writes "The [greenhouse] prediction relies on relatively simple chemical reactions ...." when the calculation he's describing relies on simple spectra for the absorption of infrared light, not a chemical reaction at all. He thinks that CO2 lasts about 30 years in the atmosphere. The actual concentration changes trail off on a wide range of time scales extending to more than a millenium. At this point one realizes that he's making just the sort of mistake he describes in other contexts, trying to play poker without knowing anything about cards. You might win, but it depends too much on luck, and you're vulnerable to hustlers.
Perhaps as a result, he treats a crackpot who ignores the basic science as being nearly on the same level as the people working on serious predictions. (Michael Mann has written on this problem here.) Silver's not quite like the pundits who couldn't follow the odds that he and the other rational aggregators were giving during the 2012 presidential campaign. He doesn't quite end up saying that the question of whether greenhouse gases are warming the earth is "too close to call", but he gets closer to that than his own methods should justify.
Silver's treatment of early predictions for the magnitude of global warming is superficial, in a way that tends to overestimate the uncertainty. In judging how urgent climate mitigation efforts are, we need for starters a good value for the climate sensitivity - how much the global mean temperature goes up for a doubling of CO2 concentration. Approximate physics-based estimates of this sensitivity have been known for over a century, but fancy modeling is now used to try to refine them, as well as to check if some complicated dynamical effect might throw the simple estimates way off. The role of the models is to try to tell us what output we'll get for a given input, where the input consists primarily of our greenhouse emissions, our reflective emissions, volcanic eruptions, and solar variations. The models cannot and do not need to predict what choices we'll make about emissions, or what accidental fluctuations will occur in volcanic or solar inputs. Ideally, models would be able to predict the variations due to El Nino/La Nina events, but for now those appear as just another noise obscuring the signal.
How well does the family of models work? The models have now been around long enough to allow some beginning tests of actual predictions, not just retrodictions. What we're looking for is how well the models do at cranking out the correct temperature record given the particular inputs that we've made or have happened in the last 20-25 years. Here Silver does a poor job of separating signal from noise, i.e. separating how well the models worked from:
1. how much the human inputs (e.g. sulfur from coal) differed from expected business-as-usual
2. accidental input noise (e.g. solar changes)
3. fairly short-term fluctuations (e.g. El Nino) not captured by current models.
Silver starts with a figure (12-3) schematically showing how different components of "uncertainty in global warming forecasts" depend on post-forecast time. Although not clearly labeled, it describes uncertainty in the slope of the temperature change, not of the temperature itself. That makes sense, since it's the slope that's most closely related to the sensitivity we're trying to estimate in order to figure out what the consequences of future CO2 emissions will be. The figure includes uncertainty from some medium-term noise, falling to low values after around 20 years. That period is described as a "sweet spot" to test the models, since on longer times there's increasing "scenario uncertainty" from differences in emissions from prior guesses. This is seriously misleading, however, since we know what the past emissions were, so scenario uncertainty contributes nothing to the uncertainty in judging how well the models work. Only the shorter-term noise does that. Thus there is no "sweet spot" for these comparisons. Twenty years is enough to just begin to get a handle on testing the models, and longer times will only improve the accuracy of the comparison.
Silver initially concludes that the warming has been below the range of predictions made around 1988. He then backtracks to remember that scenario uncertainty is not a problem with the models, slightly moderating that conclusion, but still concluding the forecasts "might deserve a low but not failing grade". He essentially ignores the non-negligible role of medium-term noise, whose influence is not nearly as unimportant after 20 years as he seems to suggest. Later, he does describe how a Bayesian updating procedure would take such noise into account, and how it's sensitive to realistic noise estimates, but doesn't go back to evaluate the models consistently. It would have been much simpler, at this point in the book, to just apply the Bayesian methods to the climate problem, as most practitioners do, rather than wander down unscientific detours.
So what's the bottom line, has the last 25 years of data shown big problems with the old models or not? How much have climate scientists had to change the estimates of the climate sensitivity? Dana Nuccitelli has written an informative blog on this issue. Estimates of one factor, the direct effect of CO2 on the radiation balance, have dropped about 15% and now seem quite definite. The net estimates of climate sensitivity remain significantly uncertain but the plausible range (~1.5°C-4.5°C) has scarcely changed over the last 25 years.
Estimates, outside the models, of the expected rate of CO2 emissions have also slightly dropped, in part because we have made modest efforts to reduce them. Reflective emissions, primarily from Chinese coal, have also risen, as Silver mentions. That raises the question of whether deliberately engineering such emissions might help reduce future warming. A quick look at the enormous risks involved in such geo-engineering proposals might have been more useful than the somewhat convoluted description of the climate predictions.
The chapter's general tone suggests that, given all the uncertainty, passivity isn't unreasonable. Only the most alert readers will catch a quick note mentioning that in a rational cost/benefit analysis the uncertainty here favors taking preventive action. The main reason is simply that the probable effects get worse much more rapidly if the temperature changes turn out to be bigger than expected than they are ameliorated for smaller changes. Another reason is that even for a given average change in temperature, the uncertainty in local climate effects leaves a tail of very unpleasant possibilities. This is exactly the sort of general point which a good Bayesian gambler like Silver would know how to explain well. It's a shame to see him downplay it and spend time on poor presentations of more specific science.
Worse than the overly agnostic predictions, however, is the advice to climate scientists. Some of the advice is good, e.g. to avoid false certainty in predictions, which can backfire when faced with a bit of noise. Silver takes the argument a lot farther, though. He says that, since scientific debates, in contrast to political ones, are somewhat rational and tend to make progress, scientists should steer entirely clear of the political arguments. He's missed the point altogether. Climate scientists are not trying to win bets on what will happen to the climate, they're trying to have an effect on what will happen to the climate. One might as well argue that a farmer should stay indoors, since all sorts of bad weather and horseshit lurk outdoors. It's good advice unless you actually need crops.
Perhaps aware that he hasn't exactly laid out a practical plan of action, at this point Silver slips out of the hard-boiled gambler persona and turns into a sort of Bayesian religious cultist. He writes, of science in general, "...if one believes in Bayes's theorem, scientific progress is inevitable as predictions are made and beliefs are tested and refined." This nonsensical claim has no basis in Bayes' theorem, P(A|B)P(B)=P(B|A)P(A), or in any other formal argument. Maybe information will accumulate, maybe the libraries will burn. You can make historical arguments for either prediction. A good Bayesian might lay some odds, not just assert the Bayesian deus ex machina will make us ever smarter and smarter. Silver goes on to another philosophical red herring: "Under Bayes's theorem, no theory is perfect. Rather, it is a work in progress, always subject to further refinement..." Whatever the merits of that reasonable attitude toward knowledge, it's not an implication of Bayes's theorem (see above), which is perfectly content to work with probabilities of 0 or 1. In fact, taken formally, the claim that we are certain to remain uncertain about everything is classically self-refuting.
[As a technical aside, I should mention that despite their enormous utility Bayesian methods are provably not useful for certain solvable real-world statistical problems. See this recent discussion to track down the papers. There is no sign that Silver is aware of this mathematical proof, much less of the real-world examples.]
Strictly speaking, the problems with the book should not, for the most part, be called noise, i.e. unpredictable fuzz. There's seems to be a predictable systematic slant to them, a hesitancy to speak truth to power. Silver describes getting interested in politics partly as an outgrowth of the effect of legislation on his career extracting money from gullible "fish" as a poker player. Very likely a poker background is good training for understanding what to expect of people in many circumstances. Environmental challenges are not, however, zero-sum games following fixed rules. They require a different approach.
Acknowledgement and disclosure: Gavin Schmidt made some helpful comments, but he has no responsibility for my conclusions. I should disclose that I worked a little with Silver on some forensic polling analysis, with mostly positive interactions. My priors in reading the book were not particularly different from my conclusions afterwards. In general, that pattern can either mean someone had insightful priors or failed to assimilate new evidence.