Validation splits and leakage

What you will learn

Construct and audit task-specific validation splits, derive a memorization counterexample and separate player overlap from direct held-out-label contamination.

Prerequisites: Consistency and information growth · Decomposing forecast error across groups

Name the prediction task before splitting

An evaluation score answers a question about how a fitted rule behaves on specified cases. Before allocating records, state who will receive predictions, which information exists at that moment, and how cases are weighted. Predicting another hand from a familiar player and predicting the first hand from an unseen player are different tasks. A model may remember a player identifier without learning anything transferable. A low loss on familiar identifiers therefore cannot establish performance for new players. There is no universally correct split independent of the intended deployment question.

Our fictional ledger contains four players A, B, C and D, with two records each. Their binary outcomes are respectively 0, 0, 1 and 1, constant within a player. Every row has equal evaluation weight. These are constructed outcomes, not measurements of poker skill or a claim that real players never change. Training and holdout must both be nonempty. The learner controls every row assignment, while the rule and loss are fixed in advance. This complete small ledger makes the information paths inspectable rather than hiding them behind a random seed.

Derive what the fitted rule can remember

Define T as the training row set and H as its complement. The training rate is pT = sum of training outcomes divided by the number of training rows. For a holdout row belonging to player g, the memorizing rule returns that player's training outcome when at least one training row with identifier g exists. Otherwise it returns pT. Because each player's outcomes agree in this model, the stored value is unambiguous. There is no extra fitting step and no tuning after looking at the holdout result.

Use binary Brier loss (p minus y) squared and average over H. A remembered player contributes zero here, while an unseen player can contribute positive loss. Thus the complete loss is the sum of squared fallback errors on unseen-player rows, divided by all holdout rows, including remembered rows. This denominator matters: adding easy remembered rows can dilute the same difficult unseen-player errors. The activity displays the prediction and loss for each holdout row so that the reported average can be independently reconstructed.

Build a counterexample to row-level validation

Place the first record of each player in training and the second in holdout. Four training outcomes sum to two, so pT equals one half. Every holdout identifier is already represented in training. The memorizing rule predicts each outcome exactly and its measured mean loss is zero. Nothing in this calculation is arithmetically wrong. The mistake would be interpreting that result as evidence about players whose identifiers were unavailable during fitting. Disjoint row indices alone do not establish separation at the unit relevant to the task.

Now hold out both records of B and D, training on both records of A and C. The training rate remains one half, so the fallback prediction is unchanged. None of the holdout players is remembered. Two holdout outcomes are zero and two are one; each squared error is one quarter, giving mean loss one quarter. The difference between zero and one quarter is a witness to evaluation-task mismatch. It is not an unbiased estimate of a universal optimism correction, and a different ledger could produce a different sign or magnitude.

Separate group overlap from label contamination

A second information path crosses the boundary when fitting reads holdout outcomes themselves. The activity includes an intentionally unsafe all-labels mode: its memory table and fallback rate use the entire ledger, even though only designated training rows should be available. This gives zero loss for every nonempty holdout in this constructed model. Group separation cannot repair that violation because each held-out player's answer has entered the fitted table directly. Move whole players and restore training-only fitting before supporting an unseen-player claim.

The same boundary principle applies to learned preprocessing. A centering value, selected feature or chosen threshold can carry holdout information even without an obvious outcome column in the final prediction expression. Fit learned steps inside the training partition and apply their frozen transformations to held-out inputs. Keep model choice on a separate validation partition or within training-side cross-validation; reserve final test results for evaluating the chosen procedure. Repeatedly revising the procedure after seeing test performance makes the test part of the selection process.

Audit the claim and its remaining limits

For an unseen-player target, check three conditions in order: both partitions contain rows, fitting uses training labels only, and no player identifier appears on both sides. Count overlapping player identifiers once each, not once per matching row pair. For the returning-player target, every holdout identifier must appear in training; reading held-out labels remains forbidden. A structurally admissible split supports only its stated task. It does not establish representative sampling, stationarity through time, causal validity, or physical product readiness.

Do not optimize the split until its score looks impressive. Choose the evaluation design from the target question, then inspect whether there are enough independent players and appropriate time ordering. In actual longitudinal data, a group split may still train on future information, and a chronological split may still share players. Both restrictions may be needed together. This lesson isolates group identity and label access so their mechanisms can be proven exactly. Finish by recording the split, fitting boundary, row losses and claim, including why the strongest unsupported interpretation fails.

Hands-on: Build an evaluation boundary

Move individual records, compare whole-player holdout, change the permitted fitting labels and audit row losses and claim scope.

Common mistakes

• Calling disjoint rows independent players.
• Treating group separation as a cure for fitting on holdout labels.
• Dropping zero-loss rows from the mean denominator.
• Choosing the split after inspecting its score, then calling it an untouched test.

Keep these ideas

• Define the target before allocating observations.
• Audit both identity overlap and the fitting information boundary.
• Derive row losses and keep the complete holdout denominator.
• An admissible split is not a guarantee about every future population.

Further reading

• scikit-learn: Data leakage, sections 12.2–12.2.2 ↗

  Read fitting boundaries, training-only learned preprocessing and feature-selection leakage. Official project documentation. The eight-record memory model, arithmetic, activity and questions are original; no source code exercises or video review claimed. Checked 2026-10-25.

• scikit-learn: Validation workflow and grouped data, sections 3.1 and 3.1.2.4 ↗

  Read train/validation/test roles and grouped validation through GroupKFold and LeaveOneGroupOut. Used for task-specific group separation. The complete finite counterexample and claim audit are original. Checked 2026-10-25.