So far every number you have predicted has lived on a sliding scale: a house price, an exam score, tomorrow's temperature. That job is called regression — predicting a continuous value, a number where a little more and a little less both make sense. But a great many questions are not like that at all. Is this email spam? Is this transaction fraudulent? Is this cell malignant? The answer is not a number on a scale; it is a category, a bucket the example belongs in. Predicting a category is called classification, and it is the single most common job in applied machine learning.
Two jobs, two shapes of answer
The cleanest way to hold the two apart is to look at the answer's shape. Regression answers how much? with a number that can land anywhere on a line. Classification answers which one? with one of a fixed set of labels. The labels might be two — spam or not spam — or many — cat, dog, bird. What they share is that there is no in-between label, and no notion that dog is more than cat. They are buckets, not points on a ruler.
flowchart LR A["a labeled example"] --> B["what do we predict?"] B --> C["REGRESSION<br/>a number: price, score, temperature"] B --> D["CLASSIFICATION<br/>a category plus probability:<br/>spam / not spam"] style C fill:#2563eb,color:#fff style D fill:#10b981,color:#fff
Classification is really probability estimation
Here is the idea that makes the rest of this module click: a good classifier does not just shout a label. It estimates how confident it is in each label, as a probability, and then picks the most likely one. A spam filter that outputs a 0.93 probability of spam is far more useful than one that outputs a bare spam verdict, because the probability lets you set your own cut-off. Anxious about losing real mail? Raise the threshold to 0.99. The model gives you the odds; you decide the bet. So classification has two layers: estimate the probability of each label given the features, then apply a threshold to turn that probability into a decision.
def predict_score(hours, w, b):
return w * hours + b
w, b = 8.0, 30.0
for h in [0, 2, 5, 10]:
print('hours =', h, '-> predicted score =', predict_score(h, w, b))
Turning a probability into a decision
Once you have a probability, the decision is usually one comparison: if the probability of the positive class meets or exceeds a threshold, predict that class; otherwise predict the other. The default threshold is 0.5, a coin flip's worth of confidence, but it is a choice you own. Move it down and you catch more positive cases at the cost of more false alarms; move it up and you become more conservative. There is no universally correct threshold — it depends on how badly a miss hurts in each direction.
flowchart LR P["model outputs<br/>probability p"] --> Q["p >= 0.5 ?"] Q -->|"yes"| A["predict class A"] Q -->|"no"| B["predict class B"] style A fill:#10b981,color:#fff style B fill:#b91c1c,color:#fff
def classify(score, threshold):
if score >= threshold:
return 'pass'
return 'fail'
for s in [40, 50, 60]:
print('score =', s, '->', classify(s, 50))
Why we cannot just fit a line to 0 and 1
A tempting shortcut is to treat the two classes as the numbers 0 and 1, fit the same straight line you used for regression, and read its output as a probability. It almost works, and it almost works in a way that hides the failure. The trouble is that a line of the form w * x + b is unbounded: feed it a large input and it sails right past 1; feed it a small one and it dives below 0. Neither negative nor above-one values can be a probability. A probability must live between 0 and 1, with nothing allowed outside.
def line(hours, w, b):
return w * hours + b
w, b = 0.07, -0.1
for h in [0, 5, 10, 20]:
p = line(h, w, b)
print('hours =', h, '-> raw line output =', round(p, 3))
flowchart TD L["raw line output: w*x + b"] --> N1["can be negative<br/>(e.g. -0.1)"] L --> N2["can exceed 1<br/>(e.g. 1.3)"] L --> N3["not a valid probability"] N1 --> X["need a squashing function<br/>into 0..1"] N2 --> X N3 --> X X --> S["the sigmoid (next lesson)"] style X fill:#b45309,color:#fff style S fill:#166534,color:#fff
Not all mistakes cost the same
A spam filter can be wrong in two directions, and they are not equally painful. Mark a real message as junk and your reader misses a job offer; let junk through and they merely sigh. A medical screen flips the calculus: a false alarm sends a healthy person to extra tests, but a missed disease can be fatal. The point is that the 0.5 cut-off is a starting point, not a verdict — you move it to trade one kind of error for the other, weighting the side that hurts more in your problem. No algorithm can set that trade-off for you; it is a judgement about the world, owned by whoever lives with the consequences. Probabilities make the trade-off tunable; bare labels hide it.
More than two classes
Everything here generalises past a yes-or-no answer. When there are several categories — cat, dog, bird — a classifier outputs a probability for each one and you simply choose the largest, an operation called argmax. The probabilities across the classes are designed to add up to one, because the example must belong to something. The two-class case you are building is just the simplest version: one probability fully determines the other, since the second class must take whatever share is left over. Later in this module you will extend the same idea to many classes at once.
What does this print? The boundary is inclusive: a score of exactly the threshold passes.
def classify(score, threshold):
if score >= threshold:
return 'pass'
return 'fail'
print(classify(49, 50))
print(classify(50, 50))
print(classify(51, 50))
49 is below 50, so it fails.
50 meets the threshold exactly, so it passes.
51 is above, so it also passes.
Write classify(score, threshold) that returns 'pass' when score meets or exceeds threshold, and 'fail' otherwise.
def classify(score, threshold):
# return 'pass' if score >= threshold, else 'fail'
pass
Use a single if with the >= operator.
Return 'fail' in the fall-through case.
def classify(score, threshold):
if score >= threshold:
return 'pass'
return 'fail'
A classifier estimates a probability for each label. Write most_likely(probs) that takes a dict like {'spam': 0.8, 'ham': 0.2} and returns the label with the highest probability.
def most_likely(probs):
# return the label (key) whose value is largest
pass
Track the best label and best probability seen so far while you loop.
Start best_prob below zero, since every probability is >= 0.
def most_likely(probs):
best_label = None
best_prob = -1.0
for label, prob in probs.items():
if prob > best_prob:
best_prob = prob
best_label = label
return best_label
The pass mark is 50, so a score of exactly 50 should pass — but this function sends it to fail because it uses strict greater-than. Fix the single operator so the boundary case passes.
def classify(score, threshold):
if score > threshold:
return 'pass'
return 'fail'
Strict > excludes the boundary value itself.
Use >= so a score equal to the threshold is included.
def classify(score, threshold):
if score >= threshold:
return 'pass'
return 'fail'
Why is fitting a plain straight line to 0/1 labels a poor way to predict a class probability?
A probability is bounded between 0 and 1, but an unconstrained line w*x + b can produce any real number — negative, or greater than one. The sigmoid (next lesson) squashes that line into the 0..1 range so its output can be read as a probability.
Recap
- Regression predicts a continuous number (how much?); classification predicts a category (which one?).
- A trustworthy classifier estimates the probability of each label, then applies a threshold (default 0.5) to make a decision.
- The threshold is yours to move: lower it to catch more cases, raise it to be cautious.
- Fitting a raw line to 0/1 labels cannot produce valid probabilities, because a line is unbounded and a probability must stay between 0 and 1.
- Accuracy alone flatters a lazy classifier that always picks the majority class.
Next you will build the function that fixes the unbounded-line problem — the sigmoid — which squashes any number into a clean probability between 0 and 1.