Exact laws of estimators

What you will learn

Construct an exact finite sampling distribution for an inverse alarm estimator, evaluate its mean and bias at a fixed generating rate, and distinguish those summaries from an inclusive error-event probability.

Prerequisites: Likelihood from coarsened observations · Transforming distributions and losing information · Sampling error

Freeze the parameter and randomize whole datasets

An estimator is a rule that accepts a dataset and returns a number. An estimate is the particular number obtained after running that rule on the observed dataset. To study the rule before seeing data, hold the unknown population parameter fixed and imagine repeating the entire data collection. The resulting distribution of outputs is its sampling distribution. The population parameter does not acquire a probability distribution merely because the estimates vary. A posterior distribution would require a prior and a different conditioning calculation.

Use a synthetic alarm instrument with exactly two packets per dataset. Every packet contains two independent Bernoulli trials at a common fixed rate p. All four trials are independent, the instrument is accurate, and it reports Q for no successes or A for at least one success. Every repeat starts with fresh trials at the same p. There is no stopping after a favorable report, selective deletion, shared hidden rate, or replacement of the true rate by an estimate. These assumptions define a teaching experiment, not an empirical model of poker players or financial returns.

Construct the law on the space of possible datasets

One packet alarms with probability q=1−(1−p)². Let K count alarms among the two packets. The ordered recorded datasets are QQ, QA, AQ and AA. Independence gives masses (1−q)², (1−q)q, q(1−q) and q². QA and AQ are different datasets but both give K=1. Adding their masses yields P(K=0)=(1−q)², P(K=1)=2q(1−q), P(K=2)=q². These three masses sum to one by expanding ((1−q)+q)². Counting three possible values does not justify assigning each probability one third.

In the previous lesson the inverse alarm fit was T=1−√(1−K/n). Here n=2, so its possible outputs are t0=0, t1=1−1/√2 and t2=1. Attach the K probabilities to those outputs. This is an exact finite sampling law, including the endpoints, rather than a normal approximation or a simulation histogram. Its support is the same for every interior p, while its masses change with p. At p=0 or p=1 some nominally possible datasets have zero probability. Keep their labels in the editor, but do not treat them as positive-mass evidence.

Derive and independently audit the repeated-sample mean

For the worked generating rate p=1/2, a packet alarms with q=3/4. The K masses become 1/16, 6/16 and 9/16. Therefore E_p[T]=0×1/16+(1−1/√2)×6/16+1×9/16=(15−3√2)/16, approximately 0.672335. The signed bias is E_p[T]−p=(7−3√2)/16, approximately 0.172335. This positive bias describes the average of estimates across repeated complete datasets. It does not assert that every realized estimate exceeds the truth: QQ returns zero and a single-alarm dataset returns approximately 0.292893, both below one half.

Check the arithmetic with a second representation. At p=1/2 the sixteen four-bit elementary strings are equally likely. Exactly one string has both packets quiet. Six strings have exactly one alarm, since either packet can be quiet while the other has any of its three positive pairs. Nine strings have two alarms, since each packet has three positive pairs. Averaging the sixteen rule outputs directly gives the same expression (6t1+9)/16. At other rates the strings remain a useful enumeration, but their weights are p^s(1−p)^(4−s), where s counts elementary successes. Equal row counting would then be wrong.

An inverse fit need not be unbiased

The alarm fraction U=K/2 has expectation q, not p. Substituting its mean into the inverse function g(u)=1−√(1−u) gives g(q)=p. That identity concerns the mean input, whereas the sampling mean of the estimator is E_p[g(U)]. The explicit enumeration shows these are different quantities. A maximum likelihood estimate can maximize the likelihood for each observed dataset and still have finite-sample bias across datasets. Maximization and unbiasedness impose different conditions, so neither property should be inferred from the other.

The workshop lets you replace the inverse rule T with the raw alarm fraction U while preserving the observation model. This deliberately changes the estimator, not the data-generating mechanism. Its three outputs become 0, 1/2 and 1, and at p=1/2 its mean is 3/4, giving bias 1/4 when judged as an estimate of p. It is unbiased for q instead. Always name the target when discussing bias. Changing p in the controls changes the hypothetical population used to evaluate both rules; it does not estimate p from one dataset or create a posterior over competing populations.

Use the whole law for error events and model limits

Consider the inclusive event that an estimate overstates p by at least one quarter: T−p≥1/4. At p=1/2 this is T≥3/4. Only the two-alarm bucket qualifies, so its probability is 9/16. The mean bias 0.172335 cannot be used as the probability of this event. A signed average error and an event probability have different definitions and units. For the raw alarm fraction at p=1/4, the cutoff is exactly 1/2, so both K=1 and K=2 qualify. The boundary is included; using a strict comparison silently answers a different question.

In the activity, group all four ordered datasets by alarm count, then select the complete estimator buckets meeting the stated error cutoff. Inspect how much mass your grouping assigns before auditing the mean, signed bias and event probability. Rates run from zero through one in quarter steps; changing the rule or population clears stale entries and event selections. Fractions and decimals are accepted within an absolute tolerance of one millionth. Hints and reset support another attempt, and drafts last only for the current visit. Exact evaluation under this small model supplies neither a confidence interval from observed data nor evidence that real observations satisfy independence. More general bias and mean squared error comparisons belong to the following curriculum slice.

Hands-on: Assemble a sampling law

Group whole datasets, change the generating rate and estimator, and select exact overestimation buckets before auditing bias.

Common mistakes

• Assigning equal mass to the three possible estimates instead of summing dataset probabilities.
• Calling the sampling law a posterior over the fixed parameter.
• Applying the inverse function to the mean and assuming it equals the mean of inverse outputs.
• Inferring unbiasedness from maximum likelihood fitting.
• Dropping the inclusive boundary of the error event.
• Counting four-bit strings equally when p is not one half.

Keep these ideas

• An estimator varies across whole datasets while the generating parameter stays fixed.
• Group ordered datasets, map their counts to estimates, and conserve probability.
• Compute signed bias as the sampling mean minus the named target.
• An exact error probability requires selecting and weighting the appropriate support buckets.

Further reading

• UC Berkeley STAT 135: Computing Standard Errors ↗

  Read opening sampling-distribution and standard-error definitions and nonlinear-function discussion, lines 91–120. Used for conceptual distinctions only. Exact two-packet enumeration, error event and activity are original; no delta-method or video review claimed. Checked 2026-10-25.

• UC Berkeley STAT 135: Mean Squared Error and Consistency ↗

  Read introduction, fixed true-parameter convention and bias/variance/MSE definitions, lines 90–111. This lesson uses only fixed-parameter sampling and signed bias; it does not teach the later consistency proof or reproduce the source example. Checked 2026-10-25.