From a churn probability
to a decision
A churn model outputs a number per customer. Acting on it needs a threshold, and 0.5 is a convention with nothing behind it. The threshold follows from what a retention offer costs and what it saves — and it only works if the probabilities mean what they say, which is the part that usually goes unchecked.
The probabilities have to be true
Class weighting is the standard reflex for imbalanced labels. It barely changes how well the model ranks customers, and it wrecks what the numbers mean — it was told the minority class is more common than it is, so it says so.
SAYS ON AVERAGE
19.4%
against 20.4% who actually left
AUC barely moves between these models. It measures ranking, and ranking survives any monotonic distortion of the probabilities — so it cannot see this failure at all.
WHAT THAT COSTS
At the same break-even threshold, the class-weighted model sends an offer to 947 customers and the calibrated one to 431. More than double the spend, from a model whose AUC is 0.8503 against 0.8567 — a difference no ranking metric would flag in review.
Then the threshold is arithmetic
Treating a customer pays when the chance they leave, times the chance the offer works, times what they are worth, beats what the offer costs. Rearranged, that is a break-even probability — p* = C / (e · V) — and the classifier appears nowhere in it.
BREAK-EVEN THRESHOLD
0.333
50 / (0.3 × 500)
CUSTOMERS TREATED
475
of 2,500
VALUE AT THAT THRESHOLD
21,400
on held-out customers
Nothing is retrained as this moves. The threshold is a property of the offer, not of the classifier — which is also why a model whose probabilities are inflated applies it to the wrong customers.
What it adds up to
DERIVED THRESHOLD
0.333
50 / (0.3 × 500)
BEST ON THIS SAMPLE
0.300
the gap from derived is sampling noise
LOST BY USING 0.5
2,650
12% of the value on the table
The derived threshold and the empirical best land close together, which is the check that matters: the arithmetic was not fitted to this sample, so agreeing with it is evidence the derivation holds rather than a coincidence engineered after the fact.
Limits
- This is bank data, not game data. Public player telemetry with real per-user features barely exists — studios keep it — and a synthetic stand-in would mean the numbers measured nothing. The method is the part that transfers.
- Cost, value and effectiveness are inputs, not findings. I do not know a bank’s real figures. The claim is that the threshold follows from whatever they are, which is why they are on a slider rather than asserted.
- Effectiveness is assumed constant. In reality an offer works differently on different customers, and estimating that is uplift modelling — a harder problem than this one, and the honest next step rather than something quietly folded in.
- One split, one seed. Held-out figures come from a single stratified split. Repeated splits would put an interval around every number here.