A classifier does not really output a label — it outputs a score, a number expressing how positive each example looks. Turning that score into a yes-or-no prediction means picking a threshold: a score at or above it is positive, below it is negative. A high threshold predicts positive rarely (few false alarms, but you miss real cases); a low one catches more real cases at the cost of more false alarms. Every threshold is a different operating point on the same model, and there is no single right one — only trade-offs. A curve traces all of them at once, so instead of arguing over one cut-off you see the whole landscape of choices and then pick the point that fits your cost. This lesson builds those curves by hand.
flowchart LR A["each example<br/>has a score"] --> B["pick threshold t"] B --> C["score at or above t?<br/>predict positive"] C --> D["lower t:<br/>more positives predicted"] D --> E["one point per t<br/>traces a curve"] style B fill:#2563eb,color:#fff style E fill:#166534,color:#fff
From one threshold to two rates
To draw a curve you need two numbers per threshold. The true positive rate (also called recall) is the share of actual positives you caught: TP divided by all real positives. The false positive rate is the share of actual negatives you wrongly flagged: FP divided by all real negatives. Walk the threshold from high to low and each step pulls more examples above the line, so both rates climb — but at different speeds depending on how cleanly the score separates the classes. Plot FPR on the x-axis against TPR on the y-axis and each threshold becomes a point; joining them gives the curve. A model that ranks perfectly hugs the top-left corner; one that merely guesses lands on the diagonal.
data = [
(0.9, 1), (0.8, 1), (0.7, 0), (0.6, 1), (0.5, 1),
(0.4, 0), (0.3, 0), (0.2, 1), (0.1, 0), (0.0, 0),
]
def counts(data, t):
tp = fp = tn = fn = 0
for score, label in data:
pred = 1 if score >= t else 0
if pred == 1 and label == 1:
tp += 1
elif pred == 1 and label == 0:
fp += 1
elif pred == 0 and label == 0:
tn += 1
else:
fn += 1
return tp, fp, tn, fn
def tpr(tp, fn):
return tp / (tp + fn)
def fpr(fp, tn):
return fp / (fp + tn)
tp, fp, tn, fn = counts(data, 0.5)
print("TP", tp, "FP", fp, "TN", tn, "FN", fn)
print("TPR", round(tpr(tp, fn), 2), "FPR", round(fpr(fp, tn), 2))
The ROC curve and its area
Sweep the threshold across every distinct score and you get a point for each, forming the ROC curve. The area under it, AUC-ROC, collapses the whole curve to one number between zero and one. It has a clean meaning: the probability that the model ranks a randomly chosen positive above a randomly chosen negative. An AUC of 1.0 means perfect ranking; 0.5 means the scores carry no signal, which is the diagonal line. Notice what AUC measures — ranking quality, not calibration and not accuracy at any one cut-off. Two models can share an AUC while making very different mistakes, so a single summary is a map, not the territory. Still, it is a fair first read on whether the score separates the classes at all.
data = [
(0.9, 1), (0.8, 1), (0.7, 0), (0.6, 1), (0.5, 1),
(0.4, 0), (0.3, 0), (0.2, 1), (0.1, 0), (0.0, 0),
]
P = sum(1 for _, lab in data if lab == 1)
N = len(data) - P
def roc_points(data):
thresholds = sorted({s for s, _ in data})
pts = [(0.0, 0.0)]
for t in thresholds:
tp = sum(1 for s, lab in data if s >= t and lab == 1)
fp = sum(1 for s, lab in data if s >= t and lab == 0)
pts.append((fp / N, tp / P))
return pts
def auc(points):
pts = sorted(points)
area = 0.0
for i in range(1, len(pts)):
x0, y0 = pts[i - 1]
x1, y1 = pts[i]
area += (x1 - x0) * (y0 + y1) / 2
return area
print("AUC-ROC", round(auc(roc_points(data)), 2))
flowchart TD X["same model,<br/>rare positives"] --> Y["ROC curve"] X --> Z["PR curve"] Y --> Y1["looks strong:<br/>huge negative pool hides false positives"] Z --> Z1["looks honest:<br/>false positives hit precision head-on"] style Y1 fill:#b45309,color:#fff style Z1 fill:#166534,color:#fff
Precision-recall: the curve that cannot hide
The precision-recall curve plots precision against recall as the threshold sweeps. Precision asks: of everything I flagged positive, how much was real? It divides true positives by all predicted positives, so every false positive taxes it directly. Recall is the true positive rate again — the fraction of real positives the model retrieves. Because precision's denominator is predicted positives rather than the enormous negative pool, it cannot camouflage a flood of false alarms the way FPR can. When positives are rare, the PR curve tells a far more truthful story: a mediocre model sags well below the top-right corner, exposing exactly the false-positive cost ROC buried. Average precision, the area under it, is the headline number practitioners watch under imbalance.
data = [
(0.9, 1), (0.8, 1), (0.7, 0), (0.6, 1), (0.5, 1),
(0.4, 0), (0.3, 0), (0.2, 1), (0.1, 0), (0.0, 0),
]
P = sum(1 for _, lab in data if lab == 1)
def average_precision(data):
thresholds = sorted({s for s, _ in data}, reverse=True)
prev_recall = 0.0
ap = 0.0
for t in thresholds:
tp = sum(1 for s, lab in data if s >= t and lab == 1)
fp = sum(1 for s, lab in data if s >= t and lab == 0)
recall = tp / P
precision = tp / (tp + fp) if (tp + fp) else 1.0
if recall > prev_recall:
ap += (recall - prev_recall) * precision
prev_recall = recall
return ap
print("average precision", round(average_precision(data), 3))
What does this print? Precision is true positives divided by predicted positives.
def precision(tp, fp):
return tp / (tp + fp)
print(round(precision(3, 1), 2))
Predicted positives are tp + fp, here 3 + 1 = 4.
Precision is 3 / 4 = 0.75.
Write tpr(tp, fn) returning the true positive rate (TP over all actual positives) and fpr(fp, tn) returning the false positive rate (FP over all actual negatives). Return 0.0 when the denominator is zero.
def tpr(tp, fn):
# TP / (TP + FN), or 0.0 if there are no positives
pass
def fpr(fp, tn):
# FP / (FP + TN), or 0.0 if there are no negatives
pass
tpr divides tp by the total actual positives (tp + fn).
Use a conditional expression to avoid dividing by zero.
def tpr(tp, fn):
return tp / (tp + fn) if (tp + fn) else 0.0
def fpr(fp, tn):
return fp / (fp + tn) if (fp + tn) else 0.0
This function is NAMED precision, but it divides by (tp + fn) — the actual positives — so it secretly computes RECALL. Precision must divide by predicted positives. Fix the denominator so the tests pass.
def precision(tp, fp, fn):
return tp / (tp + fn)
Precision cares about predicted positives, which is tp + fp — fn is a distractor.
Guard the division so an all-zero case returns 0.0 instead of raising.
def precision(tp, fp, fn):
return tp / (tp + fp) if (tp + fp) else 0.0
Write auc(points) that takes a list of (x, y) pairs and returns the area under them with the trapezoid rule: sort by x, then sum (x1 - x0) * (y0 + y1) / 2 over each consecutive pair.
def auc(points):
# sort by x, then trapezoid over consecutive pairs
pass
Sort first so the x values run left to right.
Each trapezoid's area is its width times the average of its two heights.
def auc(points):
pts = sorted(points)
area = 0.0
for i in range(1, len(pts)):
x0, y0 = pts[i - 1]
x1, y1 = pts[i]
area += (x1 - x0) * (y0 + y1) / 2
return area
A fraud dataset is 99% legitimate. Your model scores AUC-ROC = 0.99 but average precision = 0.40. Which conclusion is correct?
Under heavy imbalance the vast negative pool keeps the false positive rate tiny, so ROC and AUC look strong. Average precision is taxed by every false positive, so 0.40 reveals that many flagged transactions are not fraud — the honest read.
Recap
- A classifier emits a score; the threshold converts it to a label, and every threshold is a different trade-off.
- ROC plots false positive rate against true positive rate; AUC-ROC is the probability the model ranks a random positive above a random negative.
- Precision-recall plots precision against recall; precision is taxed by every false positive, so PR is honest when classes are imbalanced.
- Under imbalance a high AUC-ROC can hide a model that floods you with false alarms — read the PR curve and average precision before trusting it.
Next you will see that even a beautifully ranked model can be miscalibrated: a predicted 0.9 need not mean ninety percent.