Proper scoring rules and truthful reports

What you will learn

Prove strict propriety of finite categorical quadratic and logarithmic losses, construct an improper linear-reward witness, and distinguish expected regret from realized loss and support failure.

Prerequisites: Decomposing forecast error across groups · Likelihood from coarsened observations · Expected value as a weighted average

Specify what the reporting rule should reward

A forecasting evaluation rule changes incentives. If an evaluator only rewards the probability assigned to the category that happens, an assessor may gain by exaggerating the most likely category. We will examine this failure before constructing two losses that instead favor a complete truthful distribution in expectation. The question is about the reporting mechanism, not merely whether a particular batch of forecasts looks calibrated. A good-looking observed score can arise by chance, whereas propriety compares expected scores over every possible observation under a stated generating law.

Assume exactly one of three exhaustive, mutually exclusive categories A, B and C occurs. The model law is q=(1/2,1/4,1/4). An assessor reports p=(p_A,p_B,p_C), with nonnegative entries summing to one, before seeing the category. The report does not influence which category occurs. We use natural logarithms and minimize loss. This is a stipulated one-observation experiment; independence is unnecessary for the expected-loss comparisons. Treat the categories as three outcomes of a toy diagnostic, not empirical rates for poker hands. In practice q is unknown, so the activity grants knowledge that real forecasters do not possess.

Expose an improper incentive with a complete counterexample

Consider the seemingly attractive reward R(p,i)=p_i when category i occurs. Its expected reward is q dot p. Reporting the truth gives (1/2)^2+(1/4)^2+(1/4)^2=3/8. Reporting p=(1,0,0) instead gives 1/2. Thus an assessor who believes q can raise expected reward by claiming certainty in A. This violates propriety. Maximizing classification accuracy likewise cannot identify all three probabilities: many distributions with the same most likely category induce the same classification decision. A correct label and a truthful probability vector are different evaluation targets.

A loss is proper when its expected value is minimized by p=q for every generating law q in its declared domain. It is strictly proper when that minimum is unique. These are statements about expectation over the outcome, not promises that truth wins every realized observation. If C occurs, a report concentrating on C can beat the truthful report on that case. Allowing the assessor to select a report afterward would destroy the experiment. Report timing, the permitted report set and the sign convention belong in the definition, rather than being implementation details.

Derive quadratic regret by expanding the whole vector

For an observed category i define multicategory quadratic loss B(p,i)=sum_j (p_j−1{j=i})². This convention sums all three coordinates. It equals twice the familiar binary loss when there are just two categories and complementary probabilities. Expanding the squares gives sum_j p_j²−2p_i+1. Taking expectation under q yields E_q B(p,I)=sum_j p_j²−2sum_j q_j p_j+1. At the truthful report the result is 1−sum_j q_j². Subtracting that baseline leaves sum_j(p_j−q_j)².

The regret is a sum of squares, nonnegative and zero only when every coordinate matches. This proves strict propriety for the full probability simplex, including boundary laws. It also explains why scoring only the winning coordinate without the compensating sum of squares was defective. For q above, truthful expected quadratic loss is 1−3/8=5/8. A report (3/4,1/8,1/8) adds regret (1/4)²+2(−1/8)²=3/32, giving 23/32. Each coordinate matters even when its category did not occur. Do not compare this numerical scale directly with a binary Brier score or with a logarithmic loss.

Derive logarithmic regret and certify its sign

Logarithmic loss is L(p,i)=−ln p_i. Under q its expectation is −sum_i q_i ln p_i. Subtract the truthful expected loss H(q)=−sum_i q_i ln q_i to obtain D(q||p)=sum_i q_i ln(q_i/p_i). For positive q and p use ln t≤t−1 with t=p_i/q_i. Multiplying by q_i and summing proves sum_i q_i ln(p_i/q_i)≤sum_i(p_i−q_i)=0. Negating proves nonnegative regret. Equality requires each ratio to be one, so the minimum is unique. This is an independent finite proof, not an inference from a favorable numerical example.

For q=(1/2,1/4,1/4), the truthful loss is (1/2)ln 2+(1/2)ln 4=(3/2)ln 2, approximately 1.039721 nats. For p=(3/4,1/8,1/8), expected loss is (1/2)ln(4/3)+(1/2)ln 8=(1/2)ln(32/3), approximately 1.183562. Regret is (1/2)ln(4/3), approximately 0.143841. Independently calculate the regret directly: (1/2)ln(2/3)+(1/2)ln 2 gives the same expression. The order of q and p in the divergence matters. Reversing them computes a different expectation.

Probe zero support and distinguish realized from expected performance

If p_C=0 while q_C>0, observing C incurs infinite log loss and expected log loss is infinite. Quadratic loss remains finite because each coordinate lies in [0,1]. A tiny positive probability gives a large finite log penalty rather than an exception or a missing score. When q_i=0, its contribution to expected log loss is defined as zero, even if p_i=0; an impossible category is never observed under that law. Our proof above assumed all q entries positive. Boundary laws can be handled on their positive support; wasting forecast mass outside that support cannot improve the truthful expected loss.

The workshop supplies eight probability chips, each worth 1/8. Transfer a chip from one category to another, preserving normalization. Its ledger shows each possible outcome loss and its generating weight, so the experiment contains more than a single numerical quiz. Audit both expected losses, then find the unique truthful allocation (4,2,2). Try the certainty allocation and explain why the linear reward improves while both proper losses worsen. Switch to the second law q=(3/4,1/8,1/8), and rebuild the optimum. Every edit invalidates the preceding audit. An infinite log loss must be entered explicitly as Infinity; it must not silently become zero or a discarded observation.

Finally return to the first law and inspect the C row for the report (3/4,1/8,1/8). Its log loss ln 8 exceeds truthful ln 4. Yet another individual outcome could favor a different report. Strict propriety does not guarantee a finite test-set ranking, establish calibration in an unknown population, or imply that log loss is the best operational cost for every application. Scoring rules evaluate beliefs; a downstream action still needs its own payoff model. Clipping probabilities or omitting surprising cases changes the rule and needs a new incentive analysis. Keep the complete ledger and declare any numerical probability floor before evaluation.

Hands-on: Allocate a truthful report

Transfer eight conserved probability chips, inspect complete outcome loss ledgers and audit expectations under two generating laws, including zero-support reports.

Common mistakes

• Confusing loss minimization with reward maximization.
• Using only the observed coordinate in a multicategory quadratic loss.
• Reversing the generating and reporting laws in log regret.
• Omitting a zero-probability surprise instead of recording infinite log loss.
• Treating strict propriety as a finite-sample ranking guarantee.

Keep these ideas

• Declare the outcome law, valid reports and score orientation.
• Quadratic regret is squared coordinate distance.
• Logarithmic regret is a directed weighted log ratio.
• Truth uniquely minimizes expected proper loss; individual observations need not favor it.
• Positive true mass assigned zero forecast mass causes infinite expected log loss.

Further reading

• Gneiting and Raftery (2007), categorical scoring rules, section 3 ↗

  Read the introduction definition and section 3 examples 1, 3 and 4: quadratic and logarithmic scores, directed divergence and modal incentives. We negate reward to define loss. Three-category chip models, proof steps, diagnostics and all examples are original; no full-paper or video review claimed. Checked 2026-10-25.