Every classifier that outputs a score faces the same question: where do you draw the line? The reflex is to cut at 0.5, but that default hides an assumption — that every wrong answer costs the same. In reality the two kinds of mistake rarely weigh equally. In cancer screening a false negative (a tumour missed) is catastrophic, while a false positive (an extra biopsy) is merely unpleasant, so the line should sit low and catch every plausible case. In spam filtering the balance flips, because burying a real email angers users more than letting junk through. The threshold is not a property of the model — it is a decision, and the right value depends on what each mistake costs. This lesson picks it deliberately.
flowchart LR T["the threshold t"] --> L["lower t:<br/>fewer FN, more FP"] T --> H["raise t:<br/>more FN, fewer FP"] L --> D["costly FN<br/>e.g. cancer screening"] H --> E["costly FP<br/>e.g. spam filtering"] style D fill:#166534,color:#fff style E fill:#b45309,color:#fff
Putting a price on each mistake
Call the cost of a false negative cost_fn and the cost of a false positive cost_fp. At any threshold you can count how many of each you make, so the expected cost of that threshold is cost_fn * FN + cost_fp * FP summed over the data. Walk the threshold from high to low and the two counts move against each other: lower the line and false negatives fall while false positives rise. The total cost usually dips and then climbs again, forming a curve with a single minimum. The threshold at that minimum does the least harm given your costs — not the one with the best accuracy, and almost never 0.5.
data = [
(0.9, 1), (0.8, 1), (0.7, 1), (0.6, 0), (0.5, 1),
(0.4, 0), (0.3, 1), (0.2, 0), (0.1, 0), (0.0, 0),
]
def errors_at(data, t):
fn = fp = 0
for score, lab in data:
pred = 1 if score >= t else 0
if pred == 0 and lab == 1:
fn += 1
elif pred == 1 and lab == 0:
fp += 1
return fn, fp
def expected_cost(data, t, cost_fn, cost_fp):
fn, fp = errors_at(data, t)
return cost_fn * fn + cost_fp * fp
fn, fp = errors_at(data, 0.5)
print("at 0.5: FN", fn, "FP", fp)
print("cost (FN=10, FP=1):", expected_cost(data, 0.5, 10, 1))
Finding the minimum
Because the cost is flat between consecutive scores, the minimum always lands at one of the data's score values — so you sweep those candidate thresholds, compute the cost at each, and keep the smallest. This is the same curve you drew last lesson, turned into a decision: every point on the ROC curve is a threshold, and cost is the ruler that picks one. Notice what changes when you swap the cost ratio: with false negatives expensive the chosen threshold drops toward zero (flag more, accept false alarms); with false positives expensive it climbs toward one (flag less, accept misses). The model has not changed — only the consequences have, and the threshold follows them.
data = [
(0.9, 1), (0.8, 1), (0.7, 1), (0.6, 0), (0.5, 1),
(0.4, 0), (0.3, 1), (0.2, 0), (0.1, 0), (0.0, 0),
]
def errors_at(data, t):
fn = fp = 0
for score, lab in data:
pred = 1 if score >= t else 0
if pred == 0 and lab == 1:
fn += 1
elif pred == 1 and lab == 0:
fp += 1
return fn, fp
def expected_cost(data, t, cost_fn, cost_fp):
fn, fp = errors_at(data, t)
return cost_fn * fn + cost_fp * fp
def best_threshold(data, cost_fn, cost_fp):
thresholds = sorted({s for s, _ in data})
best_t, best_c = None, None
for t in thresholds:
c = expected_cost(data, t, cost_fn, cost_fp)
if best_c is None or c < best_c:
best_c, best_t = c, t
return best_t, best_c
t, c = best_threshold(data, 10, 1)
print("FN costly -> threshold", t, "cost", c)
t, c = best_threshold(data, 1, 10)
print("FP costly -> threshold", t, "cost", c)
data = [
(0.9, 1), (0.8, 1), (0.7, 1), (0.6, 0), (0.5, 1),
(0.4, 0), (0.3, 1), (0.2, 0), (0.1, 0), (0.0, 0),
]
def errors_at(data, t):
fn = fp = 0
for score, lab in data:
pred = 1 if score >= t else 0
if pred == 0 and lab == 1:
fn += 1
elif pred == 1 and lab == 0:
fp += 1
return fn, fp
def expected_cost(data, t, cost_fn, cost_fp):
fn, fp = errors_at(data, t)
return cost_fn * fn + cost_fp * fp
naive = expected_cost(data, 0.5, 10, 1)
optimal = min(expected_cost(data, t, 10, 1) for t in {s for s, _ in data})
print("naive 0.5 cost", naive)
print("optimal cost", optimal)
flowchart TD S["sweep every threshold"] --> C["compute expected cost<br/>at each one"] C --> M["keep the threshold<br/>with minimum cost"] M --> O["that point is the<br/>operating point on the curve"] style M fill:#2563eb,color:#fff
From one number to a toolkit
You began this module suspicious of a single accuracy figure, and you leave with a toolkit for the questions accuracy cannot answer. The confusion matrix names the four outcomes; precision and recall weigh them by what you can afford to miss; ROC and PR curves trace every threshold at once, with PR honest under imbalance; calibration checks whether the probabilities themselves are real; and now expected cost turns the curve into a defensible decision. None of these replaces the others — each lights a different face of the same model. The discipline is to report several, choose metrics that match the real cost of being wrong, and never let a headline number silence the rest.
What does this print? Expected cost weights each error by its price.
cost_fn = 10 cost_fp = 1 fn = 2 fp = 3 print(cost_fn * fn + cost_fp * fp)
False negatives cost 10 each: 10 * 2 = 20.
False positives cost 1 each: 1 * 3 = 3. Total 23.
Write errors_at(data, t) returning a (false_negatives, false_positives) pair. A row is (score, label) with label 1 for positive; predict positive when score >= t.
def errors_at(data, t):
# FN: predicted negative but label is 1
# FP: predicted positive but label is 0
pass
Predict positive with
score >= t, then compare to the label.A false negative is predicted-negative with label 1; a false positive is predicted-positive with label 0.
def errors_at(data, t):
fn = fp = 0
for score, lab in data:
pred = 1 if score >= t else 0
if pred == 0 and lab == 1:
fn += 1
elif pred == 1 and lab == 0:
fp += 1
return fn, fp
This loss function adds raw counts, so a missed cancer counts the same as a wasted biopsy — it cannot tell an asymmetric cost from a symmetric one. Fix it to weight each error by its price.
def loss(fn, fp, cost_fn, cost_fp):
return fn + fp
Each false negative costs cost_fn and each false positive costs cost_fp.
Sum cost_fn * fn + cost_fp * fp instead of fn + fp.
def loss(fn, fp, cost_fn, cost_fp):
return cost_fn * fn + cost_fp * fp
Write best_threshold(data, cost_fn, cost_fp) returning the score value whose expected cost is smallest. data is a list of (score, label) pairs; sweep every distinct score as a candidate threshold.
def best_threshold(data, cost_fn, cost_fp):
# try each distinct score as a threshold, return the one with least cost
pass
Sweep sorted(set of scores); at each, compute cost_fn * fn + cost_fp * fp.
Keep the threshold with the smallest cost seen so far.
def best_threshold(data, cost_fn, cost_fp):
def errors_at(t):
fn = fp = 0
for score, lab in data:
pred = 1 if score >= t else 0
if pred == 0 and lab == 1:
fn += 1
elif pred == 1 and lab == 0:
fp += 1
return fn, fp
thresholds = sorted({s for s, _ in data})
best_t, best_c = None, None
for t in thresholds:
fn, fp = errors_at(t)
c = cost_fn * fn + cost_fp * fp
if best_c is None or c < best_c:
best_c, best_t = c, t
return best_t
In cancer screening a false negative costs about ten times a false positive. How should the threshold compare to the default 0.5?
When false negatives are expensive the cost-minimising threshold falls, predicting positive more readily so fewer cancers slip through, even at the price of more biopsies.
Recap
- The threshold is a decision, not a model property; the default 0.5 assumes every mistake costs the same.
- Expected cost is
cost_fn * FN + cost_fp * FP; sweeping the threshold traces a cost curve with a minimum. - The best threshold is the score value where expected cost is smallest — low when false negatives are expensive, high when false positives are.
- Optimising accuracy minimises raw error count and silently equalises the two mistakes, so let expected cost choose the cut-off instead.
That completes the evaluation module. Next you turn from judging models to training them, starting with how a rule is extracted from labelled examples.