Learn
Calibration and the honest measure of confidence
You are well calibrated when the things you call 70 percent likely happen about 70 percent of the time. Calibration cannot be judged from a single decision, however memorable. It is measured across many forecasts, each written down before the outcome was known.
Where the measure comes from
Weather forecasting gave calibration its best-known tool. In 1950 the meteorologist Glenn Brier proposed scoring each forecast by the squared gap between the stated probability and what happened, counted as 1 if the event occurred and 0 if it did not. Lower is better: a perfect forecaster scores 0, and someone who always says 50 percent scores 0.25.
Weather forecasters, scored this way for decades, became some of the best calibrated professionals ever studied. Feedback that is fast, frequent, and written down is the reason.
Measuring your own
Keep a simple log. For each consequential call, write the option chosen, the outcome you expect, a probability, and the date you will check. When the date arrives, record what happened. After twenty or thirty entries, group them: of the calls you rated near 70 percent, how many came true?
If your 90 percent calls succeed only six times in ten, you are overconfident, and you now know by how much. That is far more useful than a general resolve to be humble.
Rewriting the forecast afterwards
This is the habit that quietly destroys calibration. Memory is generous: a call made at 60 percent becomes, in hindsight, something everyone saw coming. Psychologists call this hindsight bias, and once it sets in the record you learn from is no longer the one you made. Only a forecast written down and left untouched can be scored fairly.
Calibration needs a stream
One bold decision that worked proves little about the judgment behind it. Luck can carry a single call; it rarely carries forty. Teams that want to improve should log routine decisions too, so the sample grows quickly enough to teach something within a year.
How the record supports it
Argumont keeps the Verdict, its stated confidence, and the Human Decision in the Decision Passport, and the Outcome can be recorded beside them later. A saved Passport version cannot be overwritten, so the original bet stays exactly as it was made. Because the Judge's confidence describes the strength of the record rather than a probability, a team calibrates it differently: over time, it can see whether strongly supported Verdicts held up better than thin ones.
Five forecasts, scored
A manager logs five calls. A new office, rated 80 percent likely to open on time, does. A vendor renewal at 70 percent does. A launch date at 90 percent slips. A price rise at 60 percent holds. An office move at 50 percent goes badly. The Brier scores are 0.04, 0.09, 0.81, 0.16, and 0.25, an average of 0.27, slightly worse than saying 50 percent every time, which would have scored 0.25. The launch call carries most of the penalty, because 90 percent confidence on a missed date is expensive. Over thirty such entries, a pattern like this would point to one habit worth correcting: certainty about schedules.
Questions
- How many decisions do I need to judge my calibration?
- A rough picture appears after twenty or thirty logged forecasts. Reliable results for each confidence level take longer, which is why logging routine calls as well as large ones helps.
- Is a good outcome proof of good calibration?
- No. A single outcome says little about the judgment behind it. Calibration is a pattern across many forecasts, each scored against what it said in advance.
Further reading
Bring Argumont a Decision worth challenging.
Start with a paid 60 day pilot on your own real Decisions.