A model that says "ninety percent chance of rain" is making a promise: across every day it scores at ninety, it should be right about nine times in ten. Keeping that promise is calibration — whether the predicted probability matches the real observed frequency. It is a different question from accuracy or ranking. A model can separate rainy days from dry ones beautifully, yet still be systematically overconfident, claiming ninety when the truth is sixty. When you act on a probability — set a reserve price, decide to biopsy, raise an alert — calibration is what makes the number safe to trust. This lesson builds the reliability check that exposes a model whose confidences quietly lie.
flowchart LR A["bin predictions<br/>by confidence"] --> B["per bin compare:<br/>confidence vs frequency"] B --> C["on the diagonal:<br/>well calibrated"] B --> D["off the diagonal:<br/>over or under confident"] style C fill:#166534,color:#fff style D fill:#b91c1c,color:#fff
The reliability check
Split the predictions into bins by their confidence: all the 0.0 to 0.25 predictions together, then 0.25 to 0.5, and so on. In each bin compute two numbers: the average predicted probability (the model's confidence for that group) and the observed frequency of the positive class (how often it was actually right). If the model is calibrated, those two numbers agree in every bin and the points sit on the diagonal. Where confidence outruns frequency the model is overconfident there; where frequency outruns confidence it is underconfident. This table of bin-by-bin agreement is called a reliability diagram, and it is the simplest honest x-ray of a classifier's probabilities.
data = [
(0.9, 1), (0.9, 1), (0.9, 1), (0.9, 0), (0.9, 1),
(0.6, 1), (0.6, 0), (0.6, 0), (0.6, 1), (0.6, 0),
(0.4, 0), (0.4, 1), (0.4, 0), (0.4, 0), (0.4, 1),
(0.1, 0), (0.1, 0), (0.1, 1), (0.1, 0), (0.1, 0),
]
def bin_of(prob, width=0.25):
return min(int(prob / width), int(1 / width) - 1)
bins = {}
for prob, lab in data:
bins.setdefault(bin_of(prob), []).append((prob, lab))
for b in sorted(bins):
rows = bins[b]
conf = sum(p for p, _ in rows) / len(rows)
freq = sum(l for _, l in rows) / len(rows)
print("bin", b, "conf", round(conf, 2), "freq", round(freq, 2))
One number for the whole curve
Reading a table of bins is slow, so we summarise it with Expected Calibration Error (ECE). For each bin take the absolute gap between confidence and frequency, weight it by that bin's share of all predictions, and sum. The result is one number: zero means perfectly calibrated, larger means the probabilities drift further from reality on average. The absolute value matters — without it, a model overconfident on positives and underconfident on negatives would have its gaps cancel and look flawless. ECE is not the only calibration score, but it is one you can compute in a dozen lines, and it answers the question a stakeholder actually asks: when this model says eighty, how far off is eighty?
data = [
(0.9, 1), (0.9, 1), (0.9, 1), (0.9, 0), (0.9, 1),
(0.6, 1), (0.6, 0), (0.6, 0), (0.6, 1), (0.6, 0),
(0.4, 0), (0.4, 1), (0.4, 0), (0.4, 0), (0.4, 1),
(0.1, 0), (0.1, 0), (0.1, 1), (0.1, 0), (0.1, 0),
]
def bin_of(prob, width=0.25):
return min(int(prob / width), int(1 / width) - 1)
def expected_calibration_error(data, width=0.25):
n = len(data)
bins = {}
for prob, lab in data:
bins.setdefault(bin_of(prob, width), []).append((prob, lab))
ece = 0.0
for rows in bins.values():
conf = sum(p for p, _ in rows) / len(rows)
freq = sum(l for _, l in rows) / len(rows)
ece += (len(rows) / n) * abs(freq - conf)
return ece
print("ECE", round(expected_calibration_error(data), 3))
data = [
(0.9, 1), (0.9, 1), (0.9, 1), (0.9, 0), (0.9, 1),
(0.6, 1), (0.6, 0), (0.6, 0), (0.6, 1), (0.6, 0),
(0.4, 0), (0.4, 1), (0.4, 0), (0.4, 0), (0.4, 1),
(0.1, 0), (0.1, 0), (0.1, 1), (0.1, 0), (0.1, 0),
]
def accuracy(data, thresh=0.5):
correct = 0
for prob, lab in data:
pred = 1 if prob >= thresh else 0
if pred == lab:
correct += 1
return correct / len(data)
print("accuracy", round(accuracy(data), 2))
flowchart TD M["a trained model"] --> R["ranking quality<br/>does it separate the classes"] M --> CA["calibration<br/>are the probabilities real"] R --> R1["measured by AUC, ROC"] CA --> CA1["measured by reliability, ECE"] R1 -.-> N["independent:<br/>good at one, bad at the other"] CA1 -.-> N style N fill:#2563eb,color:#fff
Why this matters for decisions
Notice the asymmetry in the cost of being wrong about confidence. A doctor who acts on a model's "ninety percent malignant" when the true rate is sixty will order too many biopsies; one who trusts a deflated "thirty percent" will miss cancers. Accuracy at a single threshold hides both errors, because a cut-off throws away everything except above-or-below. Calibration preserves the full probability, which is what a decision under uncertainty actually needs: not a verdict, but an honest degree of belief to weigh against the cost of each action. That is why the next lesson trades the threshold for an explicit cost model and picks the cut-off that minimises expected loss.
What does this print? The observed frequency is the fraction of labels that are positive.
labels = [1, 1, 1, 0] print(round(sum(labels) / len(labels), 2))
Three of the four labels are positive.
Frequency is 3 / 4 = 0.75.
Write bin_of(prob, width) returning the index of the fixed-width bin prob falls into. Bins run [0, width), [width, 2*width), ... up to 1.0; clamp a prob of exactly 1.0 into the last bin.
def bin_of(prob, width):
# index of the bin [k*width, (k+1)*width) containing prob
pass
The bin index is int(prob / width).
Use min(..., number_of_bins - 1) so 1.0 lands in the final bin.
def bin_of(prob, width):
return min(int(prob / width), int(1 / width) - 1)
This gap function returns conf - freq without an absolute value, so an overconfident bin and an underconfident bin cancel each other in ECE and the model looks flawless. Calibration error must measure MAGNITUDE. Fix it.
def ece_gap(conf, freq):
return conf - freq
An underconfident bin has a negative raw gap, which must not cancel a positive one.
Wrap the subtraction in abs().
def ece_gap(conf, freq):
return abs(conf - freq)
Write expected_calibration_error(data, width) where data is a list of (probability, label) pairs. Bin by predicted probability, and return the sum over bins of (bin_size / total) * abs(frequency - avg_confidence).
def expected_calibration_error(data, width):
# bin, then weight each bin's |freq - confidence| by its share of the data
pass
Group pairs into bins with a dict keyed by bin_of(prob).
In each bin, confidence is the mean probability and frequency is the mean label.
def expected_calibration_error(data, width):
n = len(data)
def bin_of(prob):
return min(int(prob / width), int(1 / width) - 1)
bins = {}
for prob, lab in data:
bins.setdefault(bin_of(prob), []).append((prob, lab))
ece = 0.0
for rows in bins.values():
conf = sum(p for p, _ in rows) / len(rows)
freq = sum(l for _, l in rows) / len(rows)
ece += (len(rows) / n) * abs(freq - conf)
return ece
A weather model prints "90% chance of rain" on the days it later rained only 60% of the time. What is true?
Confidence (0.9) outruns observed frequency (0.6), the definition of overconfidence. On a reliability diagram that bin plots below the diagonal, and it contributes a 0.3 gap to ECE.
Recap
- Calibration means a predicted probability matches the observed frequency: days scored 0.8 are positive about 80% of the time.
- A reliability diagram bins predictions by confidence and compares each bin's average confidence to its actual frequency; the diagonal is perfect.
- Expected Calibration Error weights each bin's absolute confidence-minus-frequency gap by the bin's size, so zero is ideal.
- Ranking and calibration are independent: a model can separate classes well yet be over- or under-confident, so measure both before trusting a probability.
Next you will stop treating the threshold as fixed and instead choose it to minimise an explicit, asymmetric cost.