Abstract
You're well calibrated if your confidence matches how things turn out. Of everything you call 80 percent likely, about eight in ten should come true. Most people are overconfident at the top of the scale, and few ever find out because nobody keeps score.
This tool keeps the score for you. You log predictions with a confidence level and mark them right or wrong later, and a chart compares your hit rate at each level with perfect calibration. There's a short quiz if you want some data straight away. The Brier score sums up the result, and lower is better.
Method
- 01
Write a clear claim
Phrase it so anyone could later agree whether it came true, and add a date to check.
- 02
Pick a confidence
Choose a number between 50 percent, a coin flip, and 99 percent, nearly certain.
- 03
Resolve honestly
When the date arrives, mark it right or wrong. Don't reword the claim after the fact.
- 04
Read the chart
Compare your hit rate in each band with the diagonal line. Points below it mean you were overconfident.
- 05
Adjust
Nudge your future numbers toward what your record shows, and keep logging.
When to use it
- You make forecasts at work, like delivery dates or sales numbers, and want them to mean something.
- You want to practise Julia Galef's scout mindset by testing your beliefs against what happens.
- Your team says “probably” a lot and you'd rather use numbers.
Common pitfalls
- Making vague claims that can be argued either way when the date comes.
- Only logging the predictions you feel good about. You need the hard ones too.
- Drawing conclusions from a handful of results. Calibration shows up over dozens.
- Mixing up calibration and accuracy. You can be well calibrated and still unsure about lots of things.
Origin and references
Attributed to Glenn W. Brier; popularised by Philip Tetlock and Julia Galef (1950).
- Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1 to 3.
- Tetlock, P. E., & Gardner, D. (2015). Superforecasting: The Art and Science of Prediction. Crown.
- Galef, J. (2021). The Scout Mindset: Why Some People See Things Clearly and Others Don't. Portfolio.
- Lichtenstein, S., Fischhoff, B., & Phillips, L. D. (1982). Calibration of probabilities: The state of the art to 1980. In Kahneman, Slovic & Tversky (Eds.), Judgment under Uncertainty. Cambridge University Press.