Abstract
Evidence rarely proves or disproves an idea outright. It makes the idea more or less likely. Bayes' rule says by how much, and the answer depends on how likely you'd be to see this evidence if the idea were true compared with if it were false.
It protects you from two opposite mistakes. One is ignoring the base rate and jumping to a conclusion from a single striking clue. The other is refusing to change your mind at all. Thinking in counts, such as 'out of 1,000 cases like this one', makes the arithmetic much easier to follow.
Method
- 01
State the hypothesis
Write a claim that can be true or false, such as 'the outage was caused by yesterday's deploy'.
- 02
Set a prior
Before looking at this case, how often is a claim like this true? Start from a base rate if you can find one.
- 03
Add a piece of evidence
Estimate how likely you'd be to see it if the hypothesis were true, and how likely if it were false.
- 04
Update
The ratio of those two numbers shifts your odds. The result, called the posterior, becomes the prior for the next piece of evidence.
- 05
Act on the posterior
Decide how confident you need to be before you act, and note what evidence would change your mind.
When to use it
- You're trying to find the cause of something from clues, like a bug or a sudden change in the numbers.
- One vivid piece of evidence is pulling you toward a strong conclusion.
- You want to put a number on your confidence and revise it as you learn more.
Common pitfalls
- Ignoring the base rate. A rare condition is still fairly unlikely after one positive test.
- Counting the same evidence twice. Each clue should tell you something the others didn't.
- Treating 'consistent with the hypothesis' as strong evidence. It only counts if you'd be less likely to see it were the hypothesis false.
- Setting a prior of 0% or 100%. After that, no evidence can move you.
Origin and references
Attributed to Thomas Bayes (1763).
- Bayes, T. (1763). An essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 53.
- Gigerenzer, G., & Hoffrage, U. (1995). How to improve Bayesian reasoning without instruction: Frequency formats. Psychological Review, 102(4).
- Gigerenzer, G. (2002). Calculated Risks: How to Know When Numbers Deceive You. Simon & Schuster.