Module 4 · Classification from Scratch ⏱ 16 min

From Two Classes to Many

By the end of this lesson you will be able to:
  • Explain one-vs-rest: recasting a K-class problem as K binary classifiers by relabelling
  • Implement multiclass prediction as the argmax over per-class classifier scores
  • Describe when one-vs-rest grows expensive and what generalises it

Logistic regression, as you built it, answers a yes-or-no question: spam or not spam, sick or healthy. The real world is rarely that binary. A digit recogniser picks among ten numerals; a fruit sorter separates apples, oranges, and pears. Two classes will not stretch that far.

The cleanest way to widen a binary classifier — and the one that reuses everything you have already built — is one-vs-rest. You do not discard your logistic model; you train a small squad of copies and let them vote. This lesson shows how a handful of binary classifiers combine into one multiclass decision, and how to build that vote by hand.

flowchart LR
  B["binary: one classifier<br/>class A vs class B"] --> M["multiclass: K classes"]
  M --> O["one-vs-rest: one classifier per class<br/>class k vs the rest"]
  style O fill:#10b981,color:#fff
One binary classifier handles two classes; one-vs-rest extends the idea to many by giving each class its own classifier.

One classifier per class

Pick a single class — say, class 1 — and temporarily pretend the problem is binary: is this point class 1, or is it something else? You relabel the data so every class-1 example becomes a 1 and every other example becomes a 0, then fit an ordinary logistic classifier on those fresh labels.

Repeat that for every class. With K classes you end up with K binary classifiers, each a specialist for one category trained against all the rest. None of them knows the others exist; each answers only its own yes-or-no question, as confidently as it can. Building those K relabelled datasets is most of the labour, and it is a one-line transformation once you see it.

One-vs-rest relabelling: turn a 3-class dataset into a separate binary problem per class.
labels = [0, 1, 2, 1, 0, 2]

def one_vs_rest(labels, target):
    return [1 if y == target else 0 for y in labels]

print('class 1 vs rest:', one_vs_rest(labels, 1))
print('class 2 vs rest:', one_vs_rest(labels, 2))

Predicting by highest score

When a new point arrives, you run it through all K classifiers. Each returns a score — its conviction that the point belongs to its own class. A point that strongly resembles a class-1 example earns a high score from the class-1 classifier and low scores from the rest.

The multiclass prediction is then the argmax: whichever classifier shouts loudest wins. You are not averaging opinions or holding a majority vote; you are trusting the single specialist that is most persuaded. This works because each classifier was trained to be confident about exactly one class, so the loudest voice reliably names the right category.

flowchart TD
  P["a new point"] --> C0["classifier 0<br/>class 0 vs rest"]
  P --> C1["classifier 1<br/>class 1 vs rest"]
  P --> C2["classifier 2<br/>class 2 vs rest"]
  C0 --> S["score each, keep the highest"]
  C1 --> S
  C2 --> S
  S --> W["predicted class = argmax"]
  style W fill:#166534,color:#fff
At prediction time every classifier scores the point; the class with the highest score wins.
The multiclass decision is an argmax over the per-class scores.
def predict_class(scores):
    best = 0
    for i in range(1, len(scores)):
        if scores[i] > scores[best]:
            best = i
    return best

print('predicted class:', predict_class([0.2, 0.7, 0.5]))
print('predicted class:', predict_class([0.1, 0.3, 0.9]))

What you are really comparing

You may take the argmax over raw scores or over the sigmoid probabilities — the ranking comes out the same, because the sigmoid is a monotonically rising function that preserves order. What you must not do is compare classifiers that were trained on the original multiclass labels as if those were binary. Each specialist had to be retrained on its own one-vs-rest relabelling; without that, its score carries no meaning.

Ties are the one wrinkle. If two classifiers tie for the top score, argmax keeps the first one it saw — an arbitrary but fixed choice. Dead heats are rare with real weights, and when they genuinely matter you move to the proper generalisation rather than inventing a tie-breaker.

flowchart LR
  subgraph R["one-vs-rest"]
    R1["K classifiers<br/>each class vs all others"] --> R2["predict the highest score"]
  end
  subgraph E["the alternative"]
    E1["one multinomial model"] --> E2["K outputs that sum to 1"]
  end
  style R2 fill:#166534,color:#fff
  style E2 fill:#3776ab,color:#fff
One-vs-rest is the simple generalisation; a single multinomial model is the compact one for many classes.
Exercise

What does this print? The loop tracks the index of the largest value seen so far.

scores = [0.2, 0.7, 0.5]
best = 0
for i in range(1, len(scores)):
    if scores[i] > scores[best]:
        best = i
print(best)
Exercise

Write predict_class(scores) that returns the index of the largest score — the one-vs-rest prediction (the class whose classifier is most confident).

def predict_class(scores):
    # return the index of the largest score
    pass
Exercise

This predict_class picks the class with the LOWEST score, but one-vs-rest predicts the class that is MOST like itself — the HIGHEST score. Fix the comparison so it returns the argmax.

def predict_class(scores):
    best = 0
    for i in range(1, len(scores)):
        if scores[i] < scores[best]:
            best = i
    return best
Exercise

Complete multiclass_predict(point, classifiers) for one-vs-rest. Each entry in classifiers is a weight list w = (w0, w1, ...) for one class; that class's score is w0 + w1*x1 + w2*x2 + ... over the point's coordinates. Return the class (its index) with the highest score. The score helper is provided — focus on the prediction.

def score(w, point):
    total = w[0]
    for i in range(len(point)):
        total += w[i + 1] * point[i]
    return total

def multiclass_predict(point, classifiers):
    # score every classifier, then return the index of the highest score
    pass

Recap

  • One-vs-rest recasts a K-class problem as K binary ones: for each class, relabel its examples as 1 and all others as 0, then train a logistic classifier.
  • To predict, score the new point with every classifier and take the argmax — the class whose specialist is most confident.
  • Argmaxing raw scores and argmaxing sigmoid probabilities give the same answer, because the sigmoid preserves order.
  • Each binary classifier must be trained on its own relabelling; a model fit on the raw multiclass labels has no usable one-vs-rest score.
  • Mind the class imbalance in each sub-problem and the K-fold training cost — both push very large problems toward a single multinomial model.

The binary logistic model you built now handles as many classes as you care to train classifiers for. Next you will learn to measure whether those predictions are actually any good.

Checkpoint quiz

In one-vs-rest with K classes, how many binary classifiers do you train, and how are their outputs combined?

You argmax over the raw scores of your one-vs-rest classifiers. Would converting each to a sigmoid probability ever change the predicted class?

Go deeper — technical resources