Calibration and scoring

Evaluate probability forecasts across repeated cases without confusing one outcome with a verdict.

12 minutes · advanced · Original Poker4Quant lesson

What you will learn

Compute binary Brier score and distinguish calibration from forecast discrimination.

Prerequisites: Uncertain opponent frequencies · Variance and standard deviation

Probabilities need more than right/wrong labels

For binary outcomes y in {0,1}, the Brier score averages (p−y)² over forecasts p. Lower is better under this convention. A confident wrong forecast incurs a larger squared penalty than a cautious wrong forecast. A single outcome can be scored, but it cannot by itself establish calibration.

If the true event probability is r, expected score for forecast p is r(p−1)²+(1−r)p². This equals (p−r)²+r(1−r), minimized at p=r. The argument assumes evaluation cases are drawn from the modeled population and the forecaster aims to minimize expected score.

Calibration and informativeness are different

A calibrated group of 70% forecasts should have outcomes near 70% over enough relevant cases, allowing sampling variation. A constant forecast equal to an event’s base rate can be calibrated yet distinguish no high-risk cases from low-risk cases.

Compare forecasts against a meaningful baseline and inspect sample size, grouping and distribution changes. Do not claim skill solely because a handful of predictions were correct, or claim that an observed frequency must match a forecast exactly in a small batch.

Worked example

Two independent recorded cases were forecast at 80% and 30%. Their binary outcomes are 1 and 0. Compute the Brier score.

For the first case, squared error is (0.80−1)²=0.04.
For the second, squared error is (0.30−0)²=0.09.
Average the two errors: (0.04+0.09)/2.

The score is 0.065. This small record gives a score, not enough evidence to establish calibrated forecasting ability.

Common mistakes

Scoring only whether p is above 50% and discarding confidence.
Treating two scored cases as calibration proof.
Assuming calibration alone implies useful discrimination.

Keep these ideas

State the scoring convention and whether lower is better.
Calibration is a repeated-case property, not a single-result verdict.
Compare with a baseline and account for sampling and selection.