Confidence Is Not Probability: How to Calibrate Judgment
Turn verbal certainty into testable forecasts, score repeated predictions, inspect calibration by category, and improve without confusing confidence with truth.
Write forecasts as a probability attached to a precise event and resolution date. Record the base rate, reasons, and updates before the outcome. After many resolved questions, group forecasts into probability bands and compare stated probability with observed frequency. Improve category by category; do not infer calibration from a few memorable wins.
Certainty has no denominator
“I am highly confident” can describe evidence, emotional conviction, social status, or unwillingness to reconsider. Probability is different: it assigns a number to a defined event under specified conditions. Calibration then asks whether events forecast at a given probability occur at roughly that frequency across a suitable set.
A single prediction cannot prove a person calibrated. A 90 percent event can fail without the forecast being irrational. Calibration is visible only across repeated forecasts, and it is not the same as always choosing the most likely outcome.
A worked calibration review
After 60 project forecasts, a manager’s overall score looks respectable. The category view shows something else. Delivery-date forecasts are overconfident: events assigned 80 percent occur only about half the time. Compliance forecasts are cautious but discriminating. Market-demand forecasts are too few to judge.
The intervention should not be “be less confident.” For delivery, the manager adds an outside-view duration, tracks dependency approvals separately, and updates at fixed milestones. For compliance, the current process is retained. For demand, the organization collects more resolvable predictions before claiming improvement.
The calibration ledger: evidence and boundary
Forecasting-tournament research shows that probabilistic judgments can be recorded, scored, compared, and improved through structured practices and feedback. Studies of high-performing forecasters emphasize updating and aggregation rather than immutable talent. Research on natural-frequency formats shows that changing representation can improve reasoning about conditional probabilities in relevant tasks.
mellers-superforecasting, tetlock-tournaments, gigerenzer-frequencyClaim sources: mellers-superforecasting, tetlock-tournaments, gigerenzer-frequency
Build a calibration ledger
Each row needs:
| Field | Example | |---|---| | Question | Will the vendor deliver the signed export by 30 September? | | Resolution | Signed file received and passes documented checks | | Probability | 65% | | Base rate | 6 of 10 comparable deliveries were on time | | Up/down evidence | Dedicated owner / unresolved legal review | | Forecast time | Before observing the outcome | | Updates | Date, new evidence, old and new probability | | Result | 1 if resolved yes, 0 if no |
Freeze old entries. Updating is rational; rewriting the initial number is not. Keep canceled or ambiguously resolved questions visible and label why they could not be scored.
Measure two different qualities
Calibration asks whether predicted frequencies match observed frequencies. If events assigned 70 percent happen about 70 percent of the time, that band is calibrated.
Discrimination asks whether you assign higher probabilities to events that happen than to those that do not. Predicting the base rate for everything can be calibrated but uninformative. Good forecasting needs both.
A proper score such as the Brier score penalizes distance between probability and outcome:
(forecast probability − outcome)²
Lower is better for a binary event. Do not compare scores across question sets with radically different difficulty and base rates without adjustment.
Translate words into numbers
Teams often use “possible,” “likely,” and “almost certain” as if they had shared meanings. Ask each participant to assign a range before discussion. The disagreement is information.
For conditional risks, use natural frequencies:
Among 100 comparable cases, about 20 have the condition. Of those 20, 15 produce a positive test; among the other 80, 8 do.
This representation often makes denominators and false positives easier to inspect than abstract percentages. It does not remove the need to verify the rates or reference class.
The rival model: expertise should dominate
Domain expertise can identify mechanism, data quality, and cases that a generic base rate misses. A scoring ritual that rewards frequent easy questions can undervalue rare expert judgment. The right response is not to replace expertise with arithmetic. It is to make expert probabilities and reasons testable where possible.
Expert judgment should override a naive prior when the expert identifies diagnostic evidence and has a relevant track record. Prestige, eloquence, and confidence alone are not diagnostic.
The reversal test for the calibration ledger
Trust a forecaster’s numerical confidence more when the question is within the domain and horizon of their track record, criteria were fixed in advance, the reference class is defensible, updates are visible, and performance remains calibrated out of sample.
Reduce trust when:
- only successful forecasts are remembered;
- questions are reinterpreted after resolution;
- probability bands contain too few cases;
- accuracy comes from predicting the dominant base rate every time;
- a track record from one domain is exported to another;
- incentives favor dramatic certainty rather than honest scoring.
For one-off existential events, use model comparison, scenarios, and explicit uncertainty. There may never be enough repetitions to estimate personal calibration directly.
Calibration failures
Use a shadow forecast when organizational incentives punish uncertainty. The forecaster records an honest probability privately before the official discussion, then compares it with the public number and eventual outcome. A persistent gap reveals an incentive problem rather than a numerical-skill problem. The safeguard requires confidentiality and clear governance; it should not become a secret channel for unaccountable decisions.
- Confusing confidence in a source with event probability.
- Scoring a vague claim that cannot clearly resolve.
- Treating one surprising miss as proof of incompetence.
- Hiding forecasts that became embarrassing.
- Rewarding boldness without a proper score.
- Pooling unrelated domains and horizons.
- Reporting calibration without discrimination.
- Generating an AI probability with no reference class or evidence.
Where the calibration ledger stops guiding the decision
Calibration requires repeated, comparable, resolvable questions and can be gamed through question selection. A calibrated forecaster can still rely on a bad causal model, discriminate poorly, or recommend harmful action. Sparse, non-stationary, adversarial, and unique events demand wider uncertainty and methods beyond a personal forecast score.
Start with base rates, use probabilities carefully in expected value, and separate prediction from scenario planning.
Named sources
Evidence and further reading
Published July 29, 2026. No substantive revision has been recorded. Evidence last verified July 28, 2026.