Exact confidence sets by test inversion

Construct acceptance regions, invert them into confidence sets and audit exact coverage against reporting size.

42 minutes · intermediate · Original Poker4Quant lesson

What you will learn

Construct a finite confidence procedure by inverting candidate-wise acceptance regions, prove its coverage identity and quantify the coverage versus cardinality cost of repairs.

Prerequisites: Exact laws of estimators · Combinations versus permutations · Sampling error

Replace a point answer with a reporting procedure

A point estimate supplies one candidate value, but does not describe how a reporting rule behaves under repeated collection. Here the output is a set of candidates. Before collecting data, specify which set you would report for every possible count. Coverage asks how often this random set includes the fixed generating value. It is a property of the complete procedure, including outputs that were not observed today. A plausible-looking set from one dataset cannot establish that property. Likewise, high likelihood at one candidate does not itself calibrate the probability of a reporting error.

Assume exactly three independent binary trials, each with the same fixed success probability p. The allowed parameter space is deliberately finite: Θ={0,1/4,1/2,3/4,1}. This is a stipulated training model, not a discretization that automatically protects an arbitrary real rate. Record K, the number of successes, without missingness, selection or early stopping. The sample size and complete reporting rule are fixed before collection. No prior distribution over Θ is supplied. Real poker or trading records can violate independence and constant-rate assumptions; this example does not validate those applications.

Build acceptance regions one candidate at a time

For each candidate p, construct an acceptance region A_p, a subset of counts {0,1,2,3}. This means that the point-null hypothesis claiming this particular p would not be rejected when K lies in A_p. Acceptance is an operational name for retaining a candidate, not proof that it is true. Under that candidate, P_p(K=k)=choose(3,k)p^k(1−p)^(3−k). Add the masses of accepted counts to get that row’s acceptance probability. To target coverage γ=3/4, require this probability to be at least 3/4 separately for every candidate. Equivalently, each point-null rejection probability is at most 1/4.

Use the following original belt: A_0={0}, A_(1/4)={0,1}, A_(1/2)={1,2}, A_(3/4)={2,3}, and A_1={3}. At p=1/4 the count numerators are (27,27,9,1) over 64; at p=1/2 they are (8,24,24,8) over 64. At p=3/4 reverse the first vector. The endpoint rows put all mass on their deterministic count. The five acceptance probabilities are therefore 1,27/32,3/4,27/32,1. Taking their minimum gives exactly 3/4. Averaging those five values would answer a different question requiring a weighting of candidate parameters.

Invert columns and prove the coverage identity

Once K=k is observed, report C(k)={p in Θ: k belongs to A_p}. Read down the observed-count column, retaining the labels of rows that accept it. The starting belt yields C(0)={0,1/4}, C(1)={1/4,1/2}, C(2)={1/2,3/4}, and C(3)={3/4,1}. These are sets of two discrete points, not filled real intervals. If K=1, the quarter and half candidates survive because each corresponding row accepts one success. A row that accepts some other count does not thereby survive the count actually seen.

For any fixed true p in Θ, the events “p belongs to C(K)” and “K belongs to A_p” are identical, count by count. Hence P_p(p in C(K))=sum over k of P_p(K=k) times 1{k in A_p}, which is the row acceptance probability already checked. This proves coverage for all allowed parameters without assigning probabilities to parameter labels or simulating many experiments. After observing K, the set is fixed and either contains the fixed truth or misses it. The 3/4 guarantee does not mean that a realized pair of candidates carries posterior probability 3/4; that would require a prior and a different calculation.

Price a repair in reported set size

Now add count zero to A_(1/2), leaving other cells unchanged. This repairs that row for a stricter target of 7/8: its coverage rises from 48/64 to 56/64. But the quarter and three-quarter rows still have 54/64, which is less than 56/64. Thus the entire belt still fails the stricter target. This is a useful diagnostic because inspecting only the repaired row suggests success. A valid repair adds count two to the quarter row and count one to the three-quarter row. Those row totals then become 63/64 each; the new minimum is 7/8.

Adding cells has a reporting cost even when it helps coverage. After just the first edit, C(0) contains three rates while the other columns still contain two. Under generating p=1/4, expected cardinality is [27×3+27×2+9×2+1×2]/64=155/64=2.421875. Independently, begin with the constant starting size two and add P_(1/4)(K=0)=27/64, giving the same answer. After the complete stricter repair, column sizes are (3,3,3,2), so the expected size at p=1/4 is 191/64. Coverage and cardinality measure different aspects of the rule; neither proves that this construction is optimal among all valid procedures.

Experiment without silently changing the guarantee

In the workshop, toggle any acceptance cell, choose a target, and inspect the exact probability ledger for all five rows. Then choose an observed count and manually reconstruct its reported set. Changing the generating row changes the repeated-sampling audit weights; changing the observed count only selects which set was realized. Neither operation changes the belt itself. Enter pointwise coverage, minimum coverage and expected cardinality as fractions or finite decimals within one millionth. Model edits clear old answers, and the audit can accept a correct diagnosis of a failing belt without calling that belt a valid confidence procedure.

Explore empty columns, disconnected candidate sets and endpoint rows. Removing count zero from the p=0 row makes its coverage zero, even if most other cells remain accepted. Adding a zero-probability cell can enlarge some reported sets without improving coverage at that row’s own parameter. An all-accepting belt has coverage one and always reports all five rates; validity alone does not ensure useful resolution. These are exact finite-model statements. Editing after inspecting a real dataset creates a selection procedure whose coverage must be recalculated as a whole. The interactive design exercise is preparation for a fixed rule, not permission to choose a favorable belt after seeing results.

Worked example

For three independent trials and Θ={0,1/4,1/2,3/4,1}, audit the starting acceptance belt, invert K=1, and assess adding count zero to the half row.

At p=1/4, enumerate the eight ordered binary strings: count groups have masses 27,27,9,1 over 64. The starting accepted counts zero and one total 54/64=27/32.
At p=1/2 the accepted counts one and two total 6/8=3/4. Endpoint coverages are one; the three-quarter row mirrors the quarter row. The minimum is 3/4.
Read column one: C(1)={1/4,1/2}. For every candidate p, membership p in C(K) is exactly K in A_p, proving row coverage equals set coverage.
Add count zero to the half row. Its coverage becomes 7/8, but the minimum across all rows becomes 27/32, so a 7/8 claim still fails.
The column sizes become (3,2,2,2). At generating p=1/4, average size is (81+54+18+2)/64=155/64. Independently compute 2+27/64.
Add count two to the quarter row and count one to the three-quarter row. Coverages are now (1,63/64,7/8,63/64,1); sizes are (3,3,3,2), with quarter-rate mean 191/64.

Rowwise acceptance mass proves exact coverage on the stated parameter space. Repairing coverage can enlarge reports, and one successful row cannot establish a uniform guarantee.

Hands-on: Build and invert an acceptance belt

Toggle candidate-wise acceptance cells, invert an observed-count column, repair failing coverage rows and price the change in expected reported cardinality.

Build and invert an acceptance belt

Three independent binary trials; fixed p belongs to Θ = {0,1/4,1/2,3/4,1}. A row specifies which counts retain its candidate. Design the whole rule before collecting real data. No off-grid coverage is claimed.

Edit acceptance rows

Each row lists count probabilities as numerators over 64. Press a count to accept or reject it. Highlight and pressed state mean accepted.

Candidate p = 0

Count masses 0,1,2,3: (64, 0, 0, 0) / 64.

Candidate p = 1/4

Count masses 0,1,2,3: (27, 27, 9, 1) / 64.

Candidate p = 1/2

Count masses 0,1,2,3: (8, 24, 24, 8) / 64.

Candidate p = 3/4

Count masses 0,1,2,3: (1, 9, 27, 27) / 64.

Candidate p = 1

Count masses 0,1,2,3: (0, 0, 0, 64) / 64.

Choose the audit experiment

The generating rate weights repeated datasets. The observed count selects a realized report; it does not change the belt.

Invert the observed column yourself

Select every candidate whose row accepts count 1. Gaps stay gaps; this is a finite set, not a filled interval. An empty selection represents the empty set.

Audit the complete procedure

Use the ledger to compute three totals. Fractions and finite decimals are accepted within 0.000001. A correct failure diagnosis does not certify the belt.

Build a fixed reporting rule, then audit its coverage. Draft lasts for this visit.

Common mistakes

Reading an acceptance row as the reported parameter set instead of inverting a count column.
Averaging coverage across candidates without a stated parameter distribution.
Interpreting a confidence guarantee as a posterior on the realized set.
Extending a five-rate guarantee to an arbitrary rate in [0,1].
Choosing a reporting rule after seeing real data while keeping its preselected-rule guarantee.
Treating a correctly diagnosed failing belt as a valid confidence procedure.

Keep these ideas

Specify the experiment, parameter space and complete rule before collection.
Construct acceptance regions using each candidate’s own sampling law.
Invert observed-count columns into sets of candidate parameters.
The membership identity transfers rowwise acceptance to set coverage.
Audit the worst allowed parameter and the expected reported cardinality separately.