Systems Playbook

Superforecasting (Philip Tetlock and Dan Gardner)

Main Argument

Forecasting the messy real world is a measurable skill, and a minority of ordinary people, "superforecasters," do it far better than experts and even intelligence analysts with classified data. Their edge is not raw intelligence or inside information but a way of working: they break hard questions into estimable parts, start from base rates, think in granular probabilities, actively seek out the case against their own view, update steadily as evidence arrives, and keep score so they can calibrate and improve. The conclusion that frames the playbook's use of it: judgment is bounded and biased, but it is improvable through deliberate procedure rather than willpower or talent.

Key Takeaways

  • Most expert prediction is barely better than chance ("the dart-throwing chimp"), and confidence is a poor guide to accuracy; only kept scores reveal who is actually any good.
  • Accuracy is measurable (e.g. the Brier score), and calibration, the match between stated confidence and real hit rate, can be improved with frequent feedback.
  • The "fox" outperforms the "hedgehog": drawing on many small ideas and aggregating perspectives beats applying one big theory.
  • Superforecasters work in probabilities, distinguishing 63% from 70% where others see only yes/no/maybe.
  • They start from the outside view (a reference class and its base rate) before adjusting for the specifics of the case.
  • They decompose intractable questions Fermi-style into sub-questions that can each be estimated.
  • They update incrementally and proportionally, avoiding both anchoring (under-reaction) and overreacting to vivid news.
  • Active open-mindedness, treating beliefs as testable hypotheses and hunting for disconfirming evidence, is the habit that most separates the best.
  • It is a practice, not a gift: the procedures generalize and can be taught, which is why debiasing is a discipline rather than a personality trait.

Concepts Extracted

Enriched