Decomposing forecast error across groups

What you will learn

Derive an exact grouped Brier decomposition, distinguish recalibration from report coarsening, and construct a pooled-calibration witness that preserves total loss while erasing resolution.

Prerequisites: Calibration and scoring · Conditional variance and the variance budget · Empirical bootstrap laws and support limits

Fix the evaluation ledger before interpreting its score

A probability forecast can earn the same average error for very different reasons. One system may separate high-frequency and low-frequency cases but exaggerate both probabilities. Another may predict a single middle value for everybody. An aggregate Brier score cannot show which defect should be repaired. We will construct an exact accounting identity for a finite ledger, then deliberately discard a group label and watch two diagnostic terms disappear together. This makes the distinction between improving a forecast and simplifying its report visible in arithmetic.

Assume each record has a binary outcome y, either zero or one, a probability forecast p in [0,1], and equal evaluation weight. Within each displayed group the issued forecast is constant. Groups are nonempty and collectively partition the ledger. These are algebraic conditions for the finite identity; independence is unnecessary for the identity itself. To interpret the ledger as evidence about future performance requires representative collection, forecasts recorded before outcomes, and a stable evaluation target. No population calibration or profitable trading claim follows from this small constructed ledger. In poker logs, selecting only shown hands could change the target before any score is calculated.

Cancel residuals within each constant-forecast group

For group g let n_g be its record count, w_g=n_g/N its weight, q_g its event fraction and p_g its common forecast. Let q be the event fraction across all records. A residual y−q_g sums to zero inside its group, because the group contains exactly n_g q_g successes. Write p_g−y=(p_g−q_g)+(q_g−y), square both sides and average within the group. The cross term vanishes by that zero residual sum. The group mean error therefore equals (p_g−q_g)² plus the mean of (y−q_g)².

Binary outcomes make the second part particularly simple. A fraction q_g contributes (1−q_g)² and the remaining fraction contributes q_g². Their sum simplifies to q_g(1−q_g). Weighting groups gives BS=Σw_g(p_g−q_g)²+Σw_g q_g(1−q_g). This proof depends on a constant forecast within each group. If you put varying probabilities into a broad bin and substitute their average, you have changed the scored forecasts. The three-term identity below applies to that changed ledger, not automatically to the original individual probabilities. Keep exact forecast values separate unless you explicitly rescore the coarsened forecast.

Separate reliability cost from resolution credit

Define reliability cost REL=Σw_g(p_g−q_g)². Define resolution credit RES=Σw_g(q_g−q)² and uncertainty UNC=q(1−q). Since Σw_g q_g=q and weights sum to one, expanding RES gives Σw_g q_g²−q². Meanwhile the weighted within-group variance is q−Σw_g q_g². Substitute the first expression into the second to obtain UNC−RES. Combining this with our residual calculation proves BS=REL−RES+UNC. Reliability and resolution are both nonnegative but enter the score with opposite signs. Lower Brier loss is better.

This is an accounting decomposition on the fixed ledger, not three independent controls. UNC is fixed when the outcome ledger stays fixed. REL measures squared disagreement with each displayed group frequency. RES measures how much those frequencies differ from the global frequency. Equal forecasts belong in a common forecast group when diagnosing a forecast system, even if another feature can split them further. A separate feature partition can reveal useful information that the forecast did not express. Do not award that unused information to the forecast. The workshop merges any groups whose issued probabilities coincide.

Work a ledger and then erase its informative label

Create eight cases in two groups of four. Group A has one event and forecast zero. Group B has three events and forecast one. Thus q_A=1/4, q_B=3/4, weights are one half, and q=1/2. Reliability is one half times (0−1/4)² plus one half times (1−3/4)², giving 1/16. Resolution is one half times (1/4−1/2)² plus one half times (3/4−1/2)², also 1/16. Uncertainty is 1/4, so the identity gives BS=1/4. Independently count mistakes: only the event in A and the non-event in B incur loss one; two errors among eight give 1/4.

Replace both forecasts by their pooled average 1/2 and merge the report. The single group has frequency 1/2, so REL and RES both become zero. Every case now incurs squared loss 1/4 and BS is unchanged. A perfect pooled reliability report has hidden two opposing original errors by changing the forecast and its information content. Alternatively recalibrate A to 1/4 and B to 3/4 while retaining the group distinction. Then REL becomes zero, RES stays 1/16 and BS falls to 3/16. These operations produce different forecasts: merging removes resolution, while this distinct-valued recalibration retains it.

Use the workshop as an information audit

Toggle individual outcomes to construct a new ledger. Cycle the group probabilities through zero, one quarter, one half, three quarters and one. Switch between retaining group forecasts and replacing them by their weighted pooled forecast. Observe the exact groups actually scored, then enter REL, RES, UNC and BS using fractions or decimals. Each edit invalidates the preceding audit so an old correct answer cannot certify a new ledger. First reproduce the worked example, then construct perfect discrimination with outcomes zero throughout A and one throughout B. Compare retaining correct endpoint forecasts with merging them.

Finally make all eight outcomes identical. Uncertainty and resolution become zero; only reliability cost remains. A zero uncertainty reference makes a normalized skill score undefined, though the raw loss remains well defined. Avoid dividing by that reference. In practical evaluation, empirical frequencies are noisy, and recalibrating on the same outcomes used to score creates optimistic evaluation. Fit any correction on separate training data and assess it on untouched cases. This workshop reports exact descriptive identities only. Its central diagnostic question is whether a better-looking reliability term came from better probabilities, discarded information, or a changed evaluation population.

Hands-on: Audit forecast grouping

Construct outcomes, edit group probabilities and merge reports to distinguish reliability repair from lost resolution.

Common mistakes

• Adding resolution as a penalty instead of subtracting its credit.
• Equating pooled reliability with correctness of the original subgroup forecasts.
• Replacing varying forecasts by a bin average without rescoring.
• Claiming exact population calibration from empirical group frequencies.
• Training a recalibration rule and evaluating it on the same outcomes.

Keep these ideas

• State the constant-forecast grouping and the evaluation population.
• Cancel within-group residuals to derive the identity.
• Reconcile BS=REL−RES+UNC with direct record losses.
• Distinguish changed probabilities from changed report granularity.
• Use separate data to evaluate a fitted correction.

Further reading

• ECMWF: Ensemble Verification I (2017), slides 18–23 ↗

  Read contingency-table grouping, Brier decomposition and reference-sample caution. Our eight-record ledger, residual derivation and merging experiments are original. No video review claimed. Checked 2026-10-25.

• Gneiting and Raftery (2007): Example 1, quadratic score ↗

  Read printed page 363, Example 1. The paper uses a multicategory reward; binary loss here is its negative divided by two. Only this section reviewed; original finite examples and no endorsement claim. Checked 2026-10-25.