Paired forecast comparisons

What you will learn

Derive paired forecast loss contrasts, construct an exact session-swap reference law under explicit exchangeability, and diagnose invalid row-level swaps and tail conventions.

Prerequisites: Validation splits and leakage · Proper scoring rules and truthful reports · Empirical bootstrap laws and support limits

Pair the losses before comparing models

Two forecasting rules should face the same evaluation cases when the question is which rule performed better on those cases. Freeze both rules, their preprocessing and the scoring convention before opening the evaluation ledger. A difficult session then affects both columns rather than entering only one model's score. Pairing does not make the cases representative, and an untouched holdout does not by itself supply a valid uncertainty calculation. We now keep the validation boundary fixed and study the reference experiment used to judge the observed difference.

Our original teaching ledger contains four sessions with two binary outcomes each. Rule A always reports probability 1/4; rule B always reports 3/4. The event might be a specified recorded action in a poker training simulation, not a recommendation about a real opponent. Each case receives equal weight. The loss is binary Brier loss, (p minus y) squared. Write d as loss A minus loss B: positive d favors B because smaller loss is better. Sessions have equal size, so averaging the four session means equals averaging all eight case differences. Unequal session sizes would require an explicit choice of weighting target.

Derive the signed contrast from the whole ledger

For y=1, A loses (1/4 minus 1) squared = 9/16, while B loses (3/4 minus 1) squared = 1/16. Their difference is 8/16 = 1/2. For y=0, their losses are 1/16 and 9/16, giving minus 1/2. Expanding the two squares also gives d = y minus 1/2, an independent algebraic check. A session containing two ones therefore has difference 1/2, a mixed session has zero, and two zeros give minus 1/2.

Let Dg be the mean difference in session g. Our observed statistic T is (D1+D2+D3+D4)/4. Retain both columns and their pairing even when the final summary is a single number. A positive T describes this evaluation ledger; it does not prove a positive population advantage. For example, four sessions of ones give T=1/2, but four sessions of zeros reverse its sign. Changing the recorded outcomes in the workshop is a counterfactual data exercise, not permission to edit a real test set until its result is favorable.

Specify exactly which swaps the null permits

Assume under the null that the joint distribution of each session's two loss columns is unchanged by exchanging the A and B labels throughout that session. Also assume that session label orientations are independent fair choices conditional on the unordered session pairs. This is a strong exchangeability assumption. Equality of expected losses alone does not imply it, and observing two model predictions on the same cases does not randomize their labels. For real model comparisons this symmetry must be justified; the activity adopts it as an explicit hypothetical reference model.

A session swap changes Dg to minus Dg while preserving both records together. Four sessions give 2 to the fourth power, or sixteen equally weighted sign assignments. For each assignment s, calculate Ts = (s1D1+s2D2+s3D3+s4D4)/4, with each sign either plus one or minus one. These are label assignments rather than sixteen necessarily different numerical statistics. If a session difference is zero, its two orientations still carry their probability mass. Collapsing identical values without keeping multiplicities changes the reference law. The workshop makes those repeated assignments visible.

Count an inclusive two-sided tail and expose a false unit

Choose the discrepancy absolute T before inspecting results. Its exact two-sided reference probability is the number of assignments satisfying absolute Ts greater than or equal to absolute T observed, divided by sixteen. Equality belongs in the tail, including the observed assignment. We enumerate the complete law, so there is no Monte Carlo error and no simulated-sample correction. This absolute-value convention is explicit; other software may define two-sided probabilities differently. A small result describes incompatibility with this null experiment, not the probability that B is the better model.

With eight ones, all four Dg equal 1/2. If k sessions retain their original labels, Ts=(k minus 2)/4. For k=0,1,2,3,4 the statistics are minus 1/2, minus 1/4, zero, 1/4, 1/2, with assignment counts 1,4,6,4,1. Only the two extremes reach the observed absolute 1/2, giving 2/16=1/8. Illegally swapping eight rows independently instead creates 256 orientations and only two extreme assignments, reporting 2/256=1/128. The observed mean has not changed. The apparent increase in evidence comes entirely from granting swaps the session null never permitted.

Audit ties, interpretation and the comparison protocol

The workshop lets you flip individual outcomes, inspect every session contrast, construct one sign assignment and mark all sixteen assignments belonging to the inclusive tail. Enter the observed mean, tail count and probability only after deciding which assignments qualify. When every session is mixed, all Dg and all Ts are zero. Every assignment ties the observed discrepancy, so the exact probability is one. Reporting zero because no statistic is strictly larger would invert the meaning of an entirely uninformative ledger. A zero difference also supplies no direction of improvement.

Report the effect size alongside the reference probability and the assumptions that support it. A large probability does not establish equal forecast quality; four sessions offer a coarse reference law. A small probability does not establish practical value, future performance or robustness to changed players. Preserve an untouched evaluation protocol for the selected pair. Trying many models and reporting only the smallest probability changes the selection procedure and requires separate multiplicity planning or new evaluation data. Finish with the signed ledger comparison, the session-level swap rule, the full weighted tail and the strongest conclusion those ingredients actually support.

Hands-on: Construct a session-swap tail

Edit paired outcomes, build session orientations and select every inclusive tail assignment before auditing the effect and exact probability.

Common mistakes

• Treating equality of expected losses as sufficient for sign exchangeability.
• Breaking a dependent session into independently swappable rows.
• Discarding repeated statistic values and their assignment multiplicities.
• Using a strict tail that excludes ties, or calling a reference probability a probability that a model is true.

Keep these ideas

• Freeze rules, evaluation cases, weights and discrepancy in advance.
• Positive loss A minus loss B favors B, but effect size and uncertainty differ.
• Generate only swaps allowed by the stated null; retain assignment weights.
• Count equality in the chosen absolute tail and report exchangeability and selection limits.

Further reading

• SciPy: paired sample permutations and exact enumeration ↗

  Read paired samples Notes, lines 150–164, and alternative conventions, lines 64–74. Used for label swaps, complete sign enumeration, inclusive ties and explicit two-sided conventions. Our four-session Brier ledger, block restriction, probability law, counterexample and activity are original. No SciPy dependency or video review. Checked 2026-10-25.

• UC Berkeley Statistics 240: paired comparisons and symmetry ↗

  Read paired assignment assumptions, lines 8–12, exchangeability and symmetric differences, lines 17–27, and equiprobable sign assignments, lines 34–39. Used only for the symmetry/assignment framework. We use a mean contrast, not the source signed-rank statistic; no source examples, code, quotes or video reused. Checked 2026-10-25.