Shrinkage, bias and squared error

What you will learn

Derive estimator squared-error risk as variance plus squared bias, construct a fixed-anchor shrinkage family and distinguish pointwise risk improvement from uniform superiority.

Prerequisites: Exact laws of estimators · Variance and standard deviation

Specify a rule before judging its errors

An estimate can be right on average and still fluctuate enough to be a poor choice under squared error. Conversely, deliberately pulling estimates toward a fixed anchor can reduce their average squared distance from the truth. This lesson evaluates complete estimation rules before a dataset is observed. The target is a fixed population success probability p, not the outcome of the next trial. The loss of a reported estimate t is (t−p)². Its repeated-sample expectation R(p) is the mean squared error, also called squared-error risk. Smaller values are preferable under this explicitly chosen loss; other losses can rank rules differently.

Assume exactly four independent Bernoulli observations X1 through X4, each at the same p, accurately recorded as zero or one. Every repeat starts with four fresh observations. There is no selection, optional stopping, dependence, measurement error or changing population. Let K be their sum and U=K/4. Build Tλ=λU+(1−λ)/2 with a data weight λ between zero and one, selected before observing K. The anchor 1/2 is stipulated, not learned from these observations. This is a synthetic estimation experiment rather than a model validated on poker hands or returns. No prior distribution over p is assumed.

Separate random spread from systematic displacement

Write μ=E_p[T] and b=μ−p for an arbitrary rule T with a finite second moment. Then T−p=(T−μ)+b. Squaring gives (T−p)²=(T−μ)²+2b(T−μ)+b². Average over repeated datasets at the fixed p. The cross term becomes 2b(E_p[T]−μ)=0. Therefore R(p)=Var_p(T)+b². The cancellation uses centering at the rule’s own sampling mean; individual dataset cross terms need not be zero. Independence between a random error and a bias is unnecessary because b is constant in this expectation.

This identity explains why zero bias alone cannot identify the best rule. Variance measures spread around μ, while squared bias measures displacement of μ from p. Both enter in squared probability units. Bias may be negative, but its contribution to risk cannot be negative. Standard deviation and signed bias cannot simply be added to obtain risk. Taking the square root of risk gives root mean squared error, which returns to probability units. Nor is this the prediction error against a new binary outcome: the target here is p itself, so there is no additional irreducible binary-outcome noise term.

Derive the entire shrinkage risk family

Linearity gives E_p[U]=p. Independence supplies Var_p(U)=p(1−p)/4: the four indicator variances add, and dividing their sum by four divides variance by sixteen. Applying the affine rule gives E_p[Tλ]=λp+(1−λ)/2, signed bias bλ=(1−λ)(1/2−p), and variance vλ=λ²p(1−p)/4. Thus Rλ(p)=λ²p(1−p)/4+(1−λ)²(1/2−p)². Moving weight from the data to the anchor contracts the rule’s five possible outputs and reduces variance, but usually introduces bias.

Consider λ=1/2 and p=1/4. The rule outputs for K=0,1,2,3,4 are 1/4,3/8,1/2,5/8,3/4. Its sampling mean is 3/8, so bias is 1/8, squared bias is 1/64, and variance is (1/4)(3/16)/4=3/256. Risk is 3/256+4/256=7/256. The unbiased sample proportion has risk (1/4)(3/4)/4=3/64=12/256. Accepting bias has reduced risk by 5/256 at this particular population. The comparison concerns average losses across all possible datasets, rather than which estimate happens to look closer after one observation.

Audit the worked result without the decomposition

At p=1/4 the count probabilities for K=0 through 4 are (81,108,54,12,1)/256. These follow by expanding (3/4+z/4)^4; their numerators sum to 256. Subtract p from the five shrinkage outputs to get errors 0,1/8,1/4,3/8,1/2. Their squared values are 0,1/64,1/16,9/64,1/4. Weight and sum directly: R=(108+216+108+16)/(256×64)=448/16384=7/256. This independently reproduces the variance-plus-squared-bias calculation without relying on the cancellation identity.

The workshop displays this dataset-count ledger as you transfer four weight markers between the data and the fixed anchor. Each transfer changes all five outputs together, so the learner constructs a complete rule rather than editing a single favorable estimate. Inspect any of nine generating populations, compare row losses, and enter variance, squared bias, local risk and the largest risk across those nine populations. Fractions and decimals are accepted within an absolute tolerance of one millionth. Changing the rule clears all entries and population claims. Changing only the inspected population clears arithmetic entries but preserves the rule’s population comparison selections. Drafts last for this visit; hint and reset support another attempt.

Find the failure region before making a global claim

For half shrinkage let d=p−1/2. Its advantage over U requires d²/4<3p(1−p)/16, or 4d²<3(1/4−d²). Hence d²<3/28. Equality gives tied risk; outside this interval half shrinkage is worse. On the activity’s eighth-step grid it wins at p=2/8 through 6/8 and loses at the other four populations. At p=0 every sample proportion is exactly zero, with risk zero, while the half-shrinkage rule always reports 1/4 and has risk 1/16. The symmetric failure occurs at p=1. Reduced variance therefore does not establish uniformly lower risk.

The workshop also asks for the maximum risk on its displayed population grid, without assigning probabilities to those populations. That maximum is a stress comparison, not a prior-weighted average and not a certificate for unexamined populations or other estimator families. Selecting a weight after being shown the hypothetical true p would give an oracle choice unavailable to an analyst who does not know p. Keep the rule fixed while comparing populations. Likewise, a plug-in risk computed by replacing p with the observed U is an additional estimation procedure, not the exact risk derived here. State the loss, population assumptions and comparison region whenever presenting a shrinkage advantage.

Hands-on: Build a rule and map its risk

Transfer data weight to a fixed anchor, inspect complete count-loss ledgers, select populations with strictly lower risk and reconcile local decomposition with a grid stress maximum.

Common mistakes

• Adding signed bias to variance instead of squaring bias.
• Centering variance at the target instead of the estimator mean.
• Assuming every cross term vanishes separately.
• Using one favorable population to claim uniform improvement.
• Retuning the weight using an unknown true parameter.
• Confusing parameter-estimation risk with prediction error for a fresh binary outcome.

Keep these ideas

• Fix the estimator, target, loss and sampling model before comparing risks.
• Mean squared error equals variance plus squared bias.
• Shrinkage contracts random spread but can introduce systematic displacement.
• Compare risk functions across populations and include boundary counterexamples.

Further reading

• UC Berkeley STAT 135: Mean Squared Error and Consistency, definitions ↗

  Read the fixed-true-parameter convention and bias, variance and MSE definitions, lines 99–111. Used for the risk identity; our finite four-trial shrinkage family, proof expansion, rational audit and pointwise failure region are original. No asymptotic result or video review claimed. Checked 2026-10-25.

• Penn State STAT 415: Estimation, section 2.2.3 Mean Square Error ↗

  Read Definition 2.3 and the displayed variance-plus-squared-bias identity using the institutional HTML. The later Example 2.12 mistakenly prints subtraction in its MSE calculations; those lines are not adopted. Our centered proof, four-observation shrinkage family, direct loss ledger and failure region are original and independently checked. No video reviewed. Checked 2026-10-25.