Recovering base rates under label shift

What you will learn

Invert an invariant binary reporting channel to identify an unknown deployment base rate, transport its positive-report forecast, and distinguish unique, nonidentified and incompatible mixtures with explicit sensitivity.

Prerequisites: Likelihood from coarsened observations · Likelihood ratios · Combining conditional branches

Separate an unknown population from an observed signal

A stored forecast can be correct for its original population and wrong for a new one. Suppose a fixed inspection rule emits either positive or negative for a hidden state S or its complement N. Deployment reveals the signal, but state labels arrive too late to count directly. We want the deployment fraction t=P(S), using an already characterized inspection mechanism. This is an inverse population problem: the unknown is the mixture producing many reports, rather than the hidden state of one reported case. Only after identifying that mixture will we update an individual positive report.

Assume the states are exhaustive and mutually exclusive, the reporting rule stays fixed, and its conditional positive probabilities a=P(+|S) and b=P(+|N) remain unchanged between source and deployment populations. Both states were available when the source mechanism was characterized. The source fraction is one half in our example, but deployment need not share it. All activity rates are exact stipulated population fractions. They are not sample estimates, and no independence assumption is needed for the population identities. Estimation from collected records would additionally require a justified sampling design and uncertainty analysis. These are synthetic inspection states, not clinical or poker performance measurements.

Invert the mixture and check the probability constraints

Partition positive reports by their hidden state. Their disjoint masses are at and b(1−t), so the observable positive fraction is r=at+b(1−t)=b+(a−b)t. If a differs from b, subtract b and divide by a−b to obtain t=(r−b)/(a−b). This rearrangement supplies one candidate, but a legal probability must still lie between zero and one. A candidate outside that interval is a contradiction of the exact assumed mechanism and observed marginal, not a negative number of state cases to retain in a table.

As t ranges over all probabilities, r traces the interval between a and b. Therefore r inside that closed interval is necessary and sufficient for a compatible mixture when the columns differ. Endpoints count: r=b gives t=0 and r=a gives t=1. A reversed detector with a<b remains invertible. Positive then favors N, but the algebra and feasibility interval still work. Discrimination here concerns unequal conditional columns, rather than whether a report has the intuitively correct name. Clamping an impossible answer to zero or one would conceal an exact mismatch; it would not solve the original equation.

Build a complete table and transport the forecast

Take a=3/4, b=1/4 and deployment positive fraction r=3/8. The inverse equation gives t=(3/8−1/4)/(3/4−1/4)=(1/8)/(1/2)=1/4. Reconstruct all four cells: S positive is 3/16, S negative is 1/16, N positive is 3/16 and N negative is 9/16. The state row totals are 1/4 and 3/4. The positive column totals 6/16=3/8, the negative column totals 10/16=5/8, and all cells sum to one. These checks certify both the inferred state composition and the forward reporting calculation.

Now P(S|+)=at/r=(3/16)/(6/16)=1/2. Under the source fraction one half, the corresponding probability was (3/8)/(3/8+1/8)=3/4. A source posterior therefore cannot simply be copied into deployment. Independently count a population of 32 tickets: eight S tickets split into six positive and two negative; twenty-four N tickets split into six positive and eighteen negative. Twelve positives contain six S tickets, recovering one half without the inverse formula. The observed positive fraction 3/8 is neither the hidden base rate 1/4 nor the positive-report posterior 1/2. Each has its own denominator.

Distinguish nonidentification, instability and incompatibility

If a=b=1/2, every t gives r=1/2. Observing that marginal cannot identify the state composition. A table built at t=0 and another at t=1 both match it, yet their state fractions disagree completely. Division by zero is a symptom of this missing information. If the same blind mechanism is paired with exact r=3/8, there is instead no matching table. Many solutions and no solution require different reports. For a silent mechanism a=b=0 and r=0, every state mixture matches, while P(S|+) is undefined because a positive report has zero mass.

Unequal columns can still be fragile. Keeping a and b fixed gives a change Δt=Δr/(a−b). For the separated mechanism, a positive-fraction perturbation of 1/8 moves the inferred fraction by 1/4. With a=1/2 and b=1/4, that same perturbation moves it by 1/2. This sensitivity is not an uncertainty interval, since we have specified no sampling law for the measured rates. In real data an out-of-range empirical fraction may reflect sampling fluctuation or a changed mechanism. Conversely, a feasible exact marginal does not prove the conditional mechanism stayed invariant: many changed channels could reproduce it. Obtain labeled deployment checks before trusting that assumption operationally.

Construct and challenge a mixture witness

The workshop gives eight state markers and 32 mass tickets. Move a marker between S and N; the channel splits its four tickets into positive and negative cells. Change the observed positive count separately, then classify the inverse problem and construct a matching table when one exists. Enter the identified deployment fraction and positive-report posterior as fractions or decimals. For ambiguity enter not identified; for impossible exact totals enter incompatible. Use undefined only when conditioning on a zero-mass positive event. Every mechanism, marginal, candidate or answer edit invalidates the preceding audit.

First reproduce the worked example with two state markers. Next switch to the reversed channel and find a different matching composition for the same observed fraction. Then use the blind channel with four positive markers and compare state allocations zero and eight: both are witnesses, so neither proves uniqueness. Decrease the observed positives to three to construct a no-solution case. Finally inspect the silent boundary. A matching forward total certifies compatibility only; identification also requires unequal columns. Keep the source characterization, deployment marginal and inferred state mixture separate in any written conclusion, and describe the invariance assumption explicitly rather than treating an algebraic fit as its empirical validation.

Hands-on: Construct a deployment mixture witness

Move hidden-state mass through fixed conditional channels, match observed signal totals and distinguish unique, ambiguous and impossible inverse problems.

Common mistakes

• Using the positive-report fraction as the hidden state fraction.
• Copying a source posterior after the state composition changes.
• Treating equal channel columns as evidence for a unique half mixture.
• Clipping an incompatible exact inverse without reporting its residual.
• Treating inverse sensitivity as a statistical confidence interval.

Keep these ideas

• Write the forward mixture before attempting inversion.
• Unequal conditional columns identify a feasible state fraction.
• Equal columns produce either many mixtures or none.
• Normalize the reconstructed positive column to transport the forecast.
• Declare invariance and separate exact compatibility from empirical validation.

Further reading

• Lipton, Wang and Smola: Detecting and Correcting for Label Shift with Black Box Predictors ↗

  Read sections 1 and 3, assumptions A.1–A.3, and section 4 through equation (1) and the matrix reformulation on PDF pages 1–3. Used for conditional invariance, source support and inverse-mixture identification. The binary inspection table, ticket witnesses, sensitivity derivation and diagnostic questions are original. No empirical-results, consistency-proof, video or professional-credential review claimed. Checked 2026-10-25.