It’s hard not to be impressed by Nate Silver. He is a quant journalist who has mastered diverse areas such as poker playing, sports analysis and psephology. He showcased his election forecasting skills in his FiveThirtyEight blog that was picked up by the New York Times. That propelled him into the national limelight in the 2012 US presidential elections when his statistically based predictions outperformed those of many pundits and pollsters. Having shaken up the American news establishment with his data-driven reporting and comment, he recently announced he was quitting the New York Times for the ABC and ESPN TV networks.
In The Signal and the Noise, Silver combines an autobiographically tinged tour of his own specialties—sports, poker and elections—with a wide-ranging study of prediction in science, economics and finance. We learn about overconfidence among professional poker players, why Garry Kasparov lost his chess match with IBM’s Big Blue computer and the statistical power law that implies terrorists will one day kill millions of people. By structuring the book as a progression from failures to solutions, he maps out some pitfalls in statistical inference and reaches his preferred answer in the form of Bayes’ theorem.
As a writer who has been intermittently writing a book about Bayes and probability for more than a decade, Silver’s approach fascinates me. Â One reason for his success is that Silver focuses on areas that amplify his strengths. Predictions are at their most compelling when you have some kind of horse race. It could be a baseball game, a poker match or an election but these are all situations where a definitive, objective answer is provided in a specified time limit, and the prediction is delivered to the sound of a ticking clock.
The fact that most of the situations he specialises in have clearly defined winners or losers lets him reduce forecasts to probabilities of binary events—what is the chance that Obama will beat Romney?
Moreover, there are limited combinations of ways in which these events can happen. In poker, there are only 52 cards that can be dealt per game. In the US presidential election there are 538 Electoral College votes allocated at state level, and so on. Although other elements such as skill or sentiment play a role, the constraints help Silver deploy the machinery of Bayesian probability in a particularly straightforward way: you start out with an initial forecast based on the possibilities, and use a simple formula to update it using new evidence.
In the less autobiographical, more reporting-based parts of the book, Silver considers other areas where the horse race property is not so apparent. When forecasting earthquakes or global warming, we don’t have the time limit or the finite set of outcomes that poker or elections do. For example, you might ask whether a decade of non-increasing global temperatures makes a theory of global warming less likely. Silver shows how the Bayes’ theorem machinery allows you to use that evidence to revise your initial belief in your climate change theory.
However, as Silver points out in his climate change chapter, the real problem is how to come up with an initial likelihood of global warming before you even begin combining it with observations. The credibility of that prior estimate depends on having a compelling model of how the atmosphere works—but who is behind the model? Silver warns that policy biases can contaminate these prior estimates, and urges scientists to step away from the ‘street fight’ of climate change believers versus sceptics so that the unsullied machinery of Bayes can do its work in peace.
Maybe Silver is right, but it could be that these more difficult situations highlight a drawback in seeing the world through forecasting spectacles. This is particularly so where outcomes are produced not by some natural process or constrained set of possibilities as in sports or elections but by the actions of other people who may or may not be trustworthy.
Rather than treating the probability of events as being fundamental as Silver does, sometimes it makes more sense to start out with the way people make decisions. Here, the rankings or values given to uncertain outcomes are the fundamental quantities, with probabilities being derived from them.
This is not as bizarre as it sounds. In fact it goes back to the very birth of probability in the 17th century, when cheating in games of chance was rife, and gentlemen placed honour above all other priorities. Probability calculus allowed gentlemen to assess whether a result was sufficiently unlikely under conditions of a fair game that they should draw their swords and demand satisfaction.
This has an echo in the financial crisis, which happened after risk-averse investors bought complex securities labelled as safe by government-sanctioned ratings agencies paid by the investment bank issuers. The outraged investor or government, witnessing the statistical improbability of the triple-A meltdown, reaches for the modern equivalent of the 17th century gentleman’s sword, by filing a lawsuit.
The big rating agencies haven’t been able to shake off these lawsuits, either losing them (such as with the Australian Rembrandt CPDO case), settling on the eve of trial (as with the Cheyne SIV case) or are still fighting (see the United States vs. S&P). The banks have also fared badly, forking out tens of billions of dollars to settle claims. You don’t read about this in the Signal and the Noise.
Instead, Silver approaches the crisis as a problem of failed rating agency prediction, rather than an endogenous problem of corruption and fraud resulting from the incentives in place. By taking the approach he does, Silver appears to go easy on the rating agencies and investment banks. Someone with a financial background might have approached that subject differently.
The same trade-off between the outside perspective and lack of familiarity crops up in Silver’s chapter on stock market prediction and bubbles, which could have benefited from more recent research and insights from current practitioners. A practitioner might have challenged the idea that markets in S&P500 stocks can be readily compared to prediction markets such as Intrade. A couple of hours on a Bloomberg or Reuters terminal would show that there are better and faster ways to make money from value investing than Robert Shiller’s 10-year trailing index price-earnings ratios. And it’s not that convincing to argue that short-selling is difficult using thirteen-year old examples from the dotcom bubble.
Compare this with the chapters on political forecasting, poker or baseball statistics which animated by Silver’s personal or insider experience, and are superb as a result. However, these are minor quibbles in the scheme of what is an outstanding book. In his areas of specialisation, Silver is going to be a master for years to come and this book shows why.
28 July 2013
