Highly Recommended Podcast- Expert Prediction

The Folly of Prediction

TETLOCK: That experts thought they knew more than they knew.That there was a systematic gap between subjective probabilities that experts were assigning to possible futures and the objective likelihoods of those futures materializing.

DUBNER: Let me translate that for you. The experts were pretty awful. And you think: awful compared to what? Did they beat a monkey with a dartboard?

TETLOCK: Oh, the monkey with a dartboard comparison, that comes back to haunt me all the time. But with respect to how they did relative to, say, a baseline group of Berkeley undergraduates making predictions, they did somewhat better than that. Did they do better than an extrapolation algorithm? No, they did not. They did for the most part a little bit worse than that. How did they do relative to purely random guessing strategy? Well, they did a little bit better than that, but not as much as you might hope.

DUBNER: That “extrapolation algorithm” that Tetlock mentioned? That’s simply a computer programmed to predict “no change in current situation.” So it turned out these smart, experienced, confident experts predicted the political future about as well, if not slightly worse, than the average daily reader of The New York Times.

TETLOCK: I think the most important takeaway would be that the experts are, they think they know more than they do. They were systematically overconfident. Some experts were really massively overconfident. And we are able to identify those experts based on some of their characteristics of their belief system and their cognitive style, their thinking style

  ........

DUBNER: Hey, guess what, Sunshine? Al Gore didn’t win Florida. Didn’t become president either. Try walking that one back. So we are congenital predictors, but our predictions are often wrong. What then? How do you defend your bad predictions? I asked Philip Tetlock what all those political experts said when he showed them their results. He had already stashed their excuses in a neat taxonomy:

TETLOCK: So, if you thought that Gorbachev for example, was a fluke, you might argue, well my understanding of the Soviet political system is fundamentally right, and the Soviet Politburo, but for some quirky statistical aberration of the Soviet Politburo would have gone for a more conservative candidate. Another argument might be, well I predicted that Canada would disintegrate, that Quebec would secede from Canada, and it didn’t secede, but the secession almost did succeed because there was a fifty point one percentage vote against secession, and that’s well within the margin of sampling error.

DUBNER: Are there others you want to name?

TETLOCK: Well another popular prediction is “off on timing.” That comes up quite frequently in the financial world as well. Many very sophisticated students of finance have commented on how hard it is, saying the market can stay irrational longer than you can stay liquid, I think is George Soros’s expression. So, “off on timing” is a fairly popular belief-system defense as well. And I predicted that Canada would be gone. And you know what? It’s not gone yet. But just hold on.

DUBNER: You answered very economically when I asked you what are the characteristics of a bad predictor; you used one word, dogmatismm. What are the characteristics, then, of a good one?

TETLOCK: Capacity for constructive self-criticism.

DUBNER: How does that self-criticism come into play and actually change the course of the prediction?

TETLOCK: Well, one sign that you’re capable of constructive self-criticism is that you’re not dumbfounded by the question: What would it take to convince you you’re wrong? If you can’t answer that question you can take that as a warning sign.

DUBNER: In his study, Tetlock found that one factor was more important than any other in someone’s predictive ability: cognitive style. You know the story about the fox and the hedgehog?

TETLOCK: Isaiah Berlin tells us that the quotation comes from the Greek warrior poet Archilichus 2,500 years ago. And the rough translation was the fox knows many things but the hedgehog knows one big thing.

DUBNER: So, talk to me about what the foxes do as predictors and what the hedgehogs do as predictors.

TETLOCK: Sure. The foxes tend to have a rather eclectic, opportunistic approach to forecasting. They’re very pragmatic. A famous aphorism by Deng Xiaoping was he “didn’t care if the cat was white or black as long as it caught mice.” And I think the attitude of many foxes is they really didn’t care whether ideas came from the left or the right, they tended to deploy them rather flexibly in deriving predictions. So they often borrowed ideas across schools of thought that hedgehogs viewed as more sacrosanct. There are many subspecies of hedgehog. But what they have in common is a tendency to approach forecasting as a deductive, top-down exercise. They start off with some abstract principles, and they apply those abstract principles to messy, real-world situations, and the fit is often decidedly imperfect.

DUBNER: So foxes tend to be less dogmatic than hedgehogs, which makes them better predictors. But, if you had to guess, who do you think more likely to show up TV or in an op-ed column, the pragmatic, nuanced fox or the know-it-all hedgehog?

DUBNER: You got it!

TETLOCK: Hedgehogs, I think, are more likely to offer quotable sound bites, whereas foxes are more likely to offer rather complex, caveat-laden sound bites. They’re not sound bites anymore if they’re complex and caveat-laden.